Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning

Authors: jonathan  cook
chris  lu
edward  hughes
joel z leibo
google  deepmind
jakob  foerster
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:51:21

Abstract: Cultural accumulation drives the open-ended and diverse progress in capabilities spanning human history. It builds an expanding body of knowledge and skills by combining individual exploration with inter-generational information transmission. Despite its widespread success among humans, the capacity for artificial learning agents to accumulate culture remains under-explored. In particular, approaches to reinforcement learning typically strive for improvements over only a single lifetime. Generational algorithms that do exist fail to capture the open-ended, emergent nature of cultural accumulation, which allows individuals to trade-off innovation and imitation. Building on the previously demonstrated ability for reinforcement learning agents to perform social learning, we find that training setups which balance this with independent learning give rise to cultural accumulation. These accumulating agents outperform those trained for a single lifetime with the same cumulative experience. We explore this accumulation by constructing two models under two distinct notions of a generation: episodic generations, in which accumulation occurs via in-context learning and train-time generations, in which accumulation occurs via in-weights learning. In-context and in-weights cultural accumulation can be interpreted as analogous to knowledge and skill accumulation, respectively. To the best of our knowledge, this work is the first to present general models that achieve emergent cultural accumulation in reinforcement learning, opening up new avenues towards more open-ended learning systems, as well as presenting new opportunities for modelling human culture.

Main paper: 
 introduction: 
 The capacity to learn skills and accumulate knowledge over timescales that far outstrip a single lifetime (i.e., across generations) is commonly referred to as cultural accumulation [Hofstede et al., 1994 , Tennie et al., 2009] . Cultural accumulation has been considered the key to human success [Henrich, 2019] , continuously aggregating new skills, knowledge and technology. It also sustains generational improvements in the behaviour of other species, such as in the homing efficiency of birds [Sasaki and Biro, 2017] . The two core mechanisms underpinning cultural accumulation are social learning [Bandura and Walters, 1977, Hoppitt and Laland, 2013] and independent discovery [Mesoudi, 2011] . Its effectiveness can be attributed to the flexibility with which participants engage in these two mechanisms [Enquist et al., 2008 , Rendell et al., 2010 , Mesoudi and Thornton, 2018] . Cumulative culture thus executes an evolutionary search in the space of behaviours, with higher rates of independent discovery corresponding to a higher mutation rate. Ultimately, how individuals should balance social vs. independent learning depends on the validity and fidelity of social learning signals, i.e., how successful other agents in the environment are at the same task.Given the profound success of cultural accumulation in nature, it is natural to explore its applicability to artificial learning systems, which remains an under-explored research direction. In deep reinforcement learning (RL), the learning problem is usually framed as taking place over a single "lifetime". Generational methods that do exist, such as iterated policy distillation [Schmitt et al., 2018 , Stooke et al., 2022] and expert iteration [Anthony et al., 2017] , explicitly optimise new generations with hand-crafted and explicit imitation learning techniques, as opposed to learning accumulation implicitly, which should ultimately enable greater flexibility. Prior work has successfully demonstrated emergent social learning in RL [Borsa et al., 2017 , Ndousse et al., 2021 , Bhoopchand et al., 2023] , showing that it helps agents solve hard exploration problems and adapt online to new tasks. However, these works have only considered one iteration of information transmission from an expert to a student. In this work, we therefore investigate how social learning and exploration can be balanced to achieve cultural accumulation in RL. This would lay the foundations for an open-ended, population-based self-improvement loop among artificial learning agents. It would also establish new tools for modelling cultural accumulation.In humans, cultural accumulation takes place over a number of timescales due to the different rates at which knowledge, skills and technology are acquired [Perreault, 2012 , Cochrane et al., 2023] . We therefore present two corresponding formulations of cultural accumulation in RL: in-context accumulation, which operates over fast adaptation to new environments, and in-weights accumulation, which operates over the slower process of updating weights (i.e., training). The in-context setting can be interpreted as analogous to short-term knowledge accumulation and the in-weights setting as analogous to long-term, skills-based accumulation.We demonstrate the effectiveness of both the in-context and in-weights models by showing sustained generational performance gains on several tasks requiring exploration under partial observability2 .On each task, we find that accumulating agents outperform those that learn for a single lifetime of the same total experience budget. This cultural accumulation emerges purely from individual agents maximising their independent rewards, without any additional losses.2 BackgroundQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The abstract and introduction motivate the study of cultural accumulation in artificial learning agents and claim that, under our models, cultural accumulation can be achieved over RL training and in-context RL in the settings we consider, both of which are backed up by our results.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
partially-observable stochastic games: 
 A Partially-Observable Stochastic Game (POSG) [Kuhn, 1953] where O is the observation space. γ ∈ (0, 1] refers to the discount factor of returns.At each timestep t, each agent i samples an action from its policy a i t ∼ π θ i (o i t ), where θ i t corresponds to the policy parameters for agent i. These actions are aggregated to form the joint action a t = (a 1 t , . . . , a N t ). The environment then calculates the next state s t+1 ∼ T (s t , a t ). The agents then receive the next observation o t+1 ∼ O(s t+1 ) and reward r t ∼ R(s t+1 ) from the environment. The objective for each agent i in a POSG is to maximise its expected discounted sum of returnsJ(π θ i ) = E s0∼ S,a0:∞∼π(s i 1:∞ )∼T [ ∞ t=0 γ t R(s t , a t )].A Partially-Observable Markov Decision Processes (POMDP) [Kaelbling et al., 1998 ] is a special case of the POSG with only one agent. For a given agent i in a POSG, if the policy parameters of all other agents θ -i are fixed, the POSG can be reduced to a POMDP for agent i. Informally, this can be done by assuming the other agents are part of the environment. In other words, sampling from other agent policies would be part of the transition function.To solve POMDPs, we make use of memory-based policy architectures that represent the prior episode interactions at timestep t using ϕ t . Memory is needed to solve POMDPs because of the state aliasing introduced by partial-observability. For recurrent neural networks (RNNs) ϕ t refers to the hidden state at timestep t. We now condition the policy on this state a i t ∼ π θ i (o i |ϕ i t ) and also update the hidden state with recurrence function h θ i , such that ϕ t+1 = h θ i (o i |ϕ i t ).In this paper, we consider meta-RL settings where an agent is tasked with optimising its return over a distribution of POMDPs M. At each episode3 , a POMDP is sampled M ∼ M and the agent is given K trials to optimise its cumulative return. This setting has also been called in-context RL [Laskin et al., 2022] . We adopt the setting of Ni et al. [2021] , where meta-RL is formulated as another POMDP M, in which the selection of M is now a part of the state s. The agent's last action a t-1 , reward r t-1 and trial termination are appended to the observation o t .One common approach to meta-RL uses gradient-based optimisation to train a meta-policy with respect to M using recurrent network architectures [Duan et al., 2016 , Wang et al., 2016] . Within each episode, updates to the agent's internal state ϕ enable static agent parameters to implement an in-context RL procedure, allowing the agent to adapt to a specific M . During adaptation to a new POMDP, this procedure aggregates and stores information from sequential observations. Over long enough episodes, agents can therefore face an in-context exploration-exploitation tradeoff, where within-episode exploration of the environment could yield higher returns.Generational training refers to training populations of agents in sequence, thus chaining together multiple learning processes [Stooke et al., 2022] . We define a generation g m = {g 1 m , . . . , g Npop m } as a population of N pop agents. Each g n m can therefore be model weights, or an internal state in the case of in-context RL. We distinguish N pop from the N used in Section 2.1, because each member of a generation of size N pop may be within its own POSG of N agents. We use n to index a specific member of a generation. A new generation is generated by a process that conditions on the old generation g m+1 = f (g m ). By modelling cultural accumulation over both in-context RL and RL training, f could capture generational accumulation in both knowledge and skills.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: In our conclusion we outline the limitations of this work and highlight directions for future research.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem statement: 
 In this work, we investigate how to achieve cultural accumulation in RL agents. We formalise a measure of success in Equation 1, where we compare the returns achieved in a held-out set of environments when learning is spread across G total generations to learning in a single lifetime with the same total experience budget. This comparison evaluates whether our models capture the benefits of cultural accumulation over singe lifetime learning.EQUATIONPolicies are parameterised according to π θ (•|ϕ) where θ are parameters of the trained network and ϕ is the agent's internal state. We define in-context accumulation as cultural accumulation during online adaptation to new environments. In this setting, θ are frozen and T is the length of an episode. We assume that a meta-RL training process has produced θ and that the internal state ϕ is used to distinguish between generations, which we elaborate on in Section 4.1.2. We define in-weights accumulation as cultural accumulation over training runs. Here, T is the number of environment steps used for training each generation and each successive generation is trained from randomly initialised parameters θ, meaning that θ instead are the substrate for accumulation.Memory Sequence: To study processes of cultural accumulation among humans, Cornish et al.[2017] use a random sequence memorisation task in which sequences recalled by one participant become training data for the next in an iterated learning process. In doing so, they show a cumulative increase in recalled string length, thus simulating cultural accumulation. We thus present Memory Sequence as a simple environment for investigating basic cultural accumulation in RL agents. In Memory Sequence, there is an arbitrary sequence made up of digits that agents must infer over the course of in-context learning or training. Each action corresponds to predicting a digit. Episodes are fixed length and agents receive +1 reward for predicting the correct next digit and -1 reward for predicting an incorrect next digit. The sequence is randomly sampled and fixed during in-context adaptation for in-context accumulation, whilst it is fixed across training for in-weights accumulation. Goal Sequence: Previous work on social learning in RL uses a gridworld Goal Cycle environment [Ndousse et al., 2021] . In this environment, there are n identical goals and agents must figure out the correct cyclic ordering in which to navigate through them. Agents receive an egocentric partial observation of the environment state on each timestep. Other prior work uses a 3-dimensional Goal Cycle environment [Bhoopchand et al., 2023] . Goal Cycle is limited in its ability to evaluate generational improvement across learned agent behaviours, because once an agent has identified the correct cyclic order, it should simply repeat the same action sequence of cycling through the goals for the duration of an episode. We therefore introduce Goal Sequence (Figure 2 ) as a simple adaptation of Goal Cycle, but with more open-ended properties. Specifically, we replace the cycle with a non-repeating sequence of goals that agents must discover by navigating based on egocentric, partial observations. We use a 7 × 7 grid with 3 goal types and agents observe a 3 × 3 grid of cells directly in front of them. For in-context accumulation, the goal sequence and positions are randomly generated and fixed across in-context adaptation. For in-weights accumulation, the goal sequence is fixed across training, but goal positions are still randomised between episodes.Travelling Salesperson: Finally, we use the Travelling Salesperson Problem (TSP). Given that cultural accumulation involves the transmission of information not otherwise immediately accessible to new generations, we specifically focus on the partially-observable variant of TSP [Buck and Keller, 2008, Noormohammadi-Asl and Taghirad, 2019] in which some or all of the cities' coordinates are not observed. The agent's objective is to visit every city, minimising the total length of the route travelled. The positions of cities are randomly generated and fixed across in-context adaptation for in-context accumulation. City positions are fixed across all of training for in-weights accumulation. 
cultural accumulation in rl: 
 We model cultural accumulation as taking place within POSGs. Each member of a given generation g m+1 learns in parallel on an independent instance of the environment, which also contains static members of the previous generation, g m . Therefore, from the perspective of a given agent n in g m+1 this is a POMDP, since the prior, static generation can be considered part of the environment. In this work, we focus on generational improvement and have thus introduced a separation between the development phase of an agent, where they are learning from and improving upon prior generations, and the transmission phase, where they are simply acting and being observed by the next generation.Under this formulation, we propose and investigate two distinct mechanisms for cultural accumulation in RL agents based on our in-context and in-weights dichotomy introduced in Section 3.We first consider in-context accumulation, where we aim to achieve cultural accumulation via fast in-context adaptation following a separate phase of meta-RL training. Meta-RL training can be viewed as imbuing agents with multiple memory systems [Squire et al., 1993] . By updating ϕ, in-context RL executes a processes of knowledge acquisition, or declarative memory [Squire, 2004] updating. In-context accumulation extends this knowledge acquisition to operate across multiple generations. The corresponding results are in Section 5.1. ϕ n ← φ 9:B := trajectory buffer 10:for each trial k ∈ [0, K -1] do 11: p obs = 1 -k/(K -1) 12:while trial not done do 13: θ n ← update(θ n , B) 24: end for To train agents capable of in-context accumulation, we require three attributes: (1) agents must learn to learn from the behaviour of other agents (i.e., social learning) within the context of a single episode,o n ∼ O(s) 14: if IsVisible ∼ Bernoulli(p obs ) then 15: Set oracle visible in o n 16: end if 17: a O ∼ π θ O ,ϵ (s), a n ∼ π θ n (o n |ϕ n ) 18: ϕ n ← f θ n (o n |ϕ n ) 19: s, r ∼ T (s, a) 20: B ← (o n , a n , r n ) // Append(2) agents must be able to act independently by their last trial, so that their behaviour can provide useful demonstrations for the next generation, (3) agents must use sufficient independent exploration to improve on the the previous generation, or each new generation will only be as good as the last.To facilitate (1) during training, but not at test time, we assume access to an oracle O with fixed parameters θO . O is able to condition on the full state s, which we refer to as privileged information.Agent n can observe the behaviours of O as they jointly act in an environment. To achieve (2), we include this representation of O in o n with probability p obs at each timestep and linearly anneal p obs across the K trials from 1 → 0. This ensures that the agent learns to act independently by the final trial. Perez et al. [2023] show that increasing opportunities to learn socially leads to less diversity in homogeneous populations, which could be seen as limiting independent discovery in our context, further motivating this constraint. For (3), we add some random noise to the oracle policy according to a tunable parameter ϵ4 . This encourages the agent to learn that the behaviour of other agents O in the environment may be sub-optimal, in which case agent n should engage in selective social learning [Poulin-Dubois and Brosseau-Liard, 2016] . This approach could be seen as imposing an information rate limit [Prystawski et al., 2023] between the oracle and learning agent. We present this process altogether as Algorithm 1, with changes to standard RL 2 in red. for each population member n ∈ [1, N pop ] do 7: ϕ n m ← φ 8: s n ← s 0 9: for each trial k ∈ [0, K -1] do 10: if k = K -1 then 11: φn m ← ϕ n m // Storep obs = 1 -k/(K -1)14: ϕ n m-1 ← φn * m-1 // Seta n m-1 ∼ π θ (o n m-1 |ϕ n m-1 ), a n m ∼ π θ (o n m |ϕ n m ) 21: ϕ n m-1 ← f θ (o n m-1 |ϕ n m-1 ), ϕ n m ← f θ (o n m |ϕ n m ) 22: s n , r n ∼ T (s n , a n ) 23:end whileend for 25: end for 27: end for Having trained agents via Algorithm 1, we evaluate their in-context accumulation. During this evaluation phase, O is replaced by the best5 of the previous generation n * , as we do not assume For each generation g m , we store the internal state of each agent on the first step of its final trial and call this the "reset state" φn m . After the final trial, we then keep the previously stored reset state of the best performing agent φn * m in terms of rewards received in its last trial. When introducing the next generation g m+1 , generation g m transitions into transmission phase and we represent the adapted behaviour of the best performing agent by resetting the internal states ϕ 1:N m ← φn * m at the beginning of each trial. Thus, the next generation g m+1 can in essence observe the best of g m repeating its final trial. In practice, we mask g m+1 from the observations of g m in order to reflect the fact that agents do not expect to see other agents in their final trial, as guaranteed by training attribute (2) above. Since the observations of g m+1 can include information about the learned behaviours of g m , we implicitly have g m+1 = f (g m ), establishing a purely in-context equivalent to generational training (Section 2.4). This process is presented in Algorithm 2.n * = argmax n∈[1,Npop] t I[k = K -1]r n m // SelectFinally, we present an algorithm for cultural accumulation to take place over the course of training successive generations of policies (i.e., in-weights) rather than through solely updating the internal state (i.e., in-context). This is akin to generational training (Section 2.4), but without modifying the policy-gradient loss, by simply including the previous generation within environments on which the new generation is training, so that their actions can be observed and learned from. This alternative model of cultural accumulation assumes an agent's lifetime to be an entire training run, rather than a single episode. Here, we anneal the probability of observing the previous generation p obs on a given environment timestep linearly over training. The corresponding approach is detailed in Algorithm 3, presented in Appendix A. In-weights RL (i.e., RL training) can be interpreted as updating procedural memory [Johnson, 2003] via skill acquisition. We thus interpret in-weights accumulation as gradual, skills-based accumulation. Note that there is no separation between "training" and "evaluation" in this case, because the in-weights algorithm views cultural accumulation as part of the training. This is in contrast to in-context accumulation, which is an evaluation-time algorithm executed by appropriately trained meta-policies. The results that correspond to this setting are in Section 5.2. 
results: 
 For each of our experiments, we use a Simplified Structured State Space Model (S5) [Smith et al., 2022] modified for RL [Lu et al., 2024] to encode memory, building on the PureJaxRL codebase [Lu et al., 2022a] . This model architecture runs asymptotically faster than Transformers in sequence length and outperforms RNNs in memory tasks [Morad et al., 2024] . PPO [Schulman et al., 2017] is As single-lifetime baselines, we use RL 2 trained on: (a) episodes of equivalent length to one generation and (b) episodes of equivalent length to all cumulative generations. All in-context evaluations are performed on held-out environment instances for the same number of trials.Memory Sequence: In Figure 3 , we show that in-context learners trained according to Algorithm 1 are capable of accumulating beyond single-lifetime RL 2 baselines and even beyond the performance of the noisy oracles with which they were trained when evaluated on a new sequence. Interestingly, when evaluating the accumulation performance of agents trained with oracles of different noise levels, we see that it degrades when oracles are too accurate. This is directly inverse to results in imitation learning [Sasaki and Yamashina, 2021] , where the quality of expert trajectories positively impacts the performance achieved when training on these trajectories. We attribute this result to an over-reliance on social learning when oracles are too accurate, which therefore impedes the progress of independent in-context learning during training. Conversely, if oracles are too random, agents do not acquire the necessary social learning skills to effectively make use of prior generations. Goal Sequence: Figure 4 (left) shows that in-context accumulation also significantly outperforms single-lifetime RL 2 when evaluated on a new goal sequence. On this more challenging, partiallyobservable navigation task, we found that higher, but still imperfect oracle accuracies during training produced the most effective accumulating agents. This is likely due to the fact that learning to learn socially in this environment is a much harder problem that involves finding and actively following another agent. We additionally observe that the performance of each generation typically drops slightly on the last trial or two, where agents are entirely or mostly independent. This indicates that although agents are able to independently recall and navigate most of the sequence they have learned, improving on the previous generation, they are still somewhat over-reliant on demonstrations.TSP: We present these results in Figure 4 (right). Again, we see that cultural accumulation enables sustained improvements considerably beyond RL 2 over a single continuous context (i.e., lifetime). We only show the RL 2 baseline trained on shorter episodes, as the baseline trained on longer episodes failed to make meaningful learning progress. In Figure 2 , we show routes traversed by different generations for an example set of city locations. Notably, these routes become more optimised across generations, with later generations exploiting a decreasing subset of edges.Figure 6 : In-weights accumulation on Goal Sequence.Memory Sequence: Figure 5 (left) shows that agents can accumulate over the course of training via Algorithm 3. For in-weights accumulation, the sequence is kept fixed over training, meaning that agents are trained to learn as much of a single sequence as possible. After one generation of accumulation, these agents outperform single-lifetime training for a duration equivalent to 5 complete generations. This demonstrates that single-lifetime learners succumb to primacy bias [Nikishin et al., 2022] and converge prematurely. In light of this, we investigate whether in-weights accumulation can be paired with other methods to overcome primacy bias. In particular, we use the simple approach of resetting some policy network layers [Nikishin et al., 2022] , opting for resetting the final two feed-forward layers, and present these results in Figure 5 (right). We observe that resetting alone helps mitigate premature convergence, but that when we apply resetting to accumulating agents6 , the improvements are compounded and agents reach higher returns than via either method independently.Goal Sequence: Figure 6 shows that in-weights accumulation exceeds the performance of training a single agent for a continuous run equivalent to the total duration of 4 accumulation steps.We show results for in-weights accumulation in TSP in Figure 7 , within Appendix B.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provide an anonymous open-sourced repository containing all code used to run experiments and instructions needed to reproduce results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
related work: 
 Social Learning: Considerable focus has been placed on using social learning to overcome hard exploration problems, imitate human behaviour, and generalise to unseen environments [Ndousse et al., 2021 , Bhoopchand et al., 2023 , Ha and Jeong, 2023] . Ndousse et al. [2021] trains agents that achieve reasonable performance without a demonstrator by increasing the proportion of episodes with no demonstrator in handpicked increments over training. We employ a similar approach for training agents to eventually act independently by annealing the probability of observing the previous generation across training. To obtain independently capable agents in-context, we anneal the probability of observing the previous generation between trials, which is similar to the expert dropout used in Bhoopchand et al. [2023] .Machine Learning Models of Cumulative Culture: A growing body of work uses advances in machine learning to model cultural accumulation in-silico [Prystawski et al., 2023 , Perez et al., 2023, Explicit Imitation Learning Implicit Third-Person Social LearningBehavioral Cloning [Pomerleau, 1988] Inverse RL [Russell, 1998] Observational Learning by RL [Borsa et al., 2017] Emergent Social Learning [Ndousse et al., 2021] Learning Few-Shot Imitation [Bhoopchand et al., 2023] Generational Training Kickstarting [Schmitt et al., 2018] X-Land [Stooke et al., 2022] Our Work [Stooke et al., 2022] . Implicit third-person social learning is more general than hand-crafted imitation learning algorithms. Our work combines both to achieve cultural accumulation. 2024]. Perez et al. [2024] use Bayesian RL with constrained inter-generational communication to reproduce social learning processes documented in human populations. This communication is represented in a domain specific language. Rather than requiring an explicit communication channel between agents, we use social learning to facilitate accumulation in a more domain agnostic manner, demonstrating performance gains in multiple distinct settings. Prystawski et al. [2023] demonstrate cultural evolution in populations of Large Language Models, where language-based communication is used as the mechanism for knowledge transfer between generations.The generational process of cultural accumulation can be seen as iterations of implicit policy distillation and improvement. Iterative policy distillation [Schmitt et al., 2018 , Stooke et al., 2022] is therefore related to our work. Unlike these methods, we do not assume that the new learner has access to the policies of experts or past generations, but only observations of their behaviour as they interact with a shared environment. We also do not explicitly modify the learning objective, leaving the agent free to learn how much or how little to imitate past generations.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Key training and test details needed to interpret results are in the main text any further details are included in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 We take inspiration from the widespread success of cumulative culture in nature and investigate cultural accumulation in reinforcement learning. We construct an in-context model of accumulation, which operates on episodic timescales, and an in-weights model, which operates over entire training runs. We define successful cultural accumulation as a generational process that exceeds the performance of independent learning with the same total experience budget. We then present in-context and in-weights algorithms that give rise to successful cultural accumulation on several tasks requiring exploration under partial observability. We find that in-context accumulation can be impeded by training agents with oracles that are either too reliable or too unreliable, highlighting the need to balance social learning and independent discovery. We also show that in-weights accumulation effectively mitigates primacy bias and is further improved by network resets.Limitations and Future Work: An interesting future direction would be to use learned autocurricula [Dennis et al., 2020 , Jiang et al., 2021] for deciding when agents should learn socially or independently, instead of linearly annealing the observation probability. Another extension would be to investigate cumulative culture in settings with different incentive structures, such as heterogeneous or cooperative rewards [Rutherford et al., 2023] . Future work could also study ways we can evolve the process of cultural accumulation itself [Lu et al., 2023a] or learn to influence the learning process of other agents [Lu et al., 2022b [Lu et al., , 2023b] ] for cultural transmission. Finally, we note that whilst understanding cumulative culture has benefits for both machine learning and modelling human society, the consequences of self-improving systems should warrant consideration. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: All plots include shaded regions indicating standard error or error bars representing 95% confidence intervals.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: In the paper's conclusion, we discuss the potential positive and negative societal impacts of this work. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper does not include the risk of any data or models that have risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: No existing assets were used in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The open-sourcing of our code includes the release of the environments used with supporting documentation. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: p obs = 1 -t/(T -1) 9:s ∼ Ŝ 10:while not done do 11: o n m-1 , o n m ∼ O(s)a n m-1 ∼ π θn * m-1 (o n m-1 )16:a n ∼ π θ n m (o n m ) 17:s, r ∼ T (s, a) In this setting, after two generations, performance exceeds agents trained for a single lifetime equivalent to 5 generations. However, we do not observe as much improvement on subsequent generations. In the Goal Sequence experiments, we select the best performing agent of the last generation for the current generation of agents to observe during training, automating the selection process. In human and animal cultural accumulation, this selection is instead learned through prestige cues Barkow et al. [1975] , Horner et al. [2010] . Thus, in the Memory Sequence experiments, we do not automatically select the best of the past generation for the agents to observe. Instead, the agents can observe the entire last generation. For In-Context Accumulation, we sort the observed oracles and agents by their performance during pre-training and evaluation. Thus, the agents ideally learn to weight the last generation by their relative performance. For In-Weights Accumulation, sorting does not make a difference, as random network initialization is agnostic to the observation ordering. Thus, the agent learns implicitly through observation, which of the past generation is the best to imitate.For in-context accumulation in Memory Sequence, we use sequences comprised of three digits, with a maximum length of 24, and give the agents four trials per episode. Each generation has a population size of three. We perform initial meta-training for 8e6 timesteps. Agents get a reward of 1.0 for a correct response and a reward for -1.0 for an incorect one. The trial ends when an incorrect response is given or the maximum length is reached.For in-weight accumulation in Memory Sequence we use sequences comprising of ten digits, with a maximum length of 24. Agents train for 1e5 timesteps. Each generation has a population size of five. Agents get a reward of 1.0 for a correct response and a reward for -1.0 for an incorect one. The episode ends when the maximum length is achieved.We use 7 × 7 grids with 3 goal types. Agents observe a 3 × 3 grid of cells directly in front of them. Observations are symbolic with 4 channels (one for each goal type and one for agents). On each timestep, an agent can take one of three actions: move forward, turn left, turn right. Agents receive +1 reward for hitting the correct next goal and -1 reward for hitting the incorrect next goal. For in-context experiments, trials are 30 steps long and there are 4 trials in an episode. For in-weights experiments, episodes are 50 steps long.For in-context accumulation in TSP, we use six cities and provide eight trials per episode. The city positions are uniformly sampled in the unit square. Each generation has a population size of three. We perform initial meta-training for 8e6 timesteps. Agents get a reward of (√ 2-dist(cur city,next city)) √ 2if the selected next city is valid. Agents get a reward of -1.0 and the trial ends if they select an already-visited city.For in-weight accumulation in TSP, we use twenty-four cities. Each generation has a population size of eight. Agents train for 3e5 timesteps. The reward setup is equivalent to above.The policy and value networks share two fully layers and a GRU layer. Values and policies are computed with two fully connected layers. All layers use leaky ReLU activations. Here, we take the example of a 4-dimensional input for the 4-dimensional embedding of an action observation and assume the in-weights setting, so no trials. The final output is 10-dimensional in the case of a random sequence made up of 10 distinct digits. The policy and value networks share three convolutional layers, a fully connected layer, and an S5 layer. Values and policies are computed with two fully connected layers. Convolutions use leaky ReLU activation functions and all other layers use tanh activation functions. Inputs are 5-dimensional for one-hot encodings of 3 goal types, walls and agents. A 3-dimensional vector for one-hot encodings of reward is concatenated to the output of the last convolutional layer. A one-hot encoding of the trial would also be included in the in-context setting.• Shared input layers:-Conv (5, 32), 1 × 1 filters, stride 1, padding 0 -Conv (32, 64), 1 × 1 filters, stride 1, padding 0 -Conv (64, 64), 1 × 1 filters, stride 1, padding 0 -FC (576 + 3, 256) -S5 (256, 256) Code can be found at https://github.com/FLAIROx/cultural-accumulation.Our use of "episode" and "trial" aligns withBauer et al. [2023], which is the reverse ofDuan et al. [2016].For the oracle to capture sub-optimal, but goal-directed behaviour, we inject correlated noise by corrupting the privileged information in o O .This is an external selection mechanism. We investigate the emergence of selective social learning in Appendix C.We initialise the second generation entirely from scratch so that it can learn to learn socially and use resetting with accumulation thereafter. 


Decision: This paper investigates the hypothesis that cultural accumulation may be possible in RL. In other words, agents may benefit not only from their own experiences while interacting with a given environment but also from knowledge implicitly passed down from previous generations. The authors proposed, investigated, and empirically analyzed two ways in which accumulation could occur.

All reviewers had an overall positive impression of this work, highlighting the well-conducted experiments designed to test the hypotheses investigated by the authors (as mentioned, e.g., by 9itL, 2ZBi, and h1KJ). Reviewer R6qB and others also argued that the proposed approach is novel and interesting. The reviews were thorough and insightful, and the discussion phase helped clarify various important concerns raised by the reviewers—particularly claims made by the authors that were perceived as overly broad and unclear. Some reviewers, such as 2ZBi, also argued (even post-rebuttal) that it is not immediately apparent how well the proposed approach compared to other methods that made similar assumptions.

Overall, however, the consensus among the reviewers was that this paper does contribute important insights that may benefit the NeurIPS community and that points of contention that were brought up were not critical and are addressable in a revised version of the paper. The reviewers strongly encouraged the authors to improve their document by adding further discussion and improving clarity to ensure that the insights introduced in this work become more apparent to a broader audience.

Reviews: review: paper_summary: paper_summary: The paper studies cultural accumulation within the context of RL agents. The techniques involve social learning based on in-context learning and in-weights learning. Each agent is modelled as a POMDP within a larger POSG. The techniques are applied to memory sequence, goal sequence, and TSPs, where they are compared to RL^2.
strength_weakness: strength_weakness: It does not appear obvious how to implement such a training setup on a physical system.

Traditional evolutionary algorithms pass policy parameters from the previous generation to the next, and these techniques are not discussed. And indeed, there is a big field integrating RL techniques with evolutionary algorithms. 

Observing other agents is also not new and this is commonly seen in multi-agent reinforcement learning. For instance, sharing observations:
https://arxiv.org/pdf/1812.00922
The benefit of sharing hidden states is not clear.

TSPs are best solved by evolutionary algorithms. There is no comparison to such algorithms.

My initial reading of the POSG section was that there is an error in the definition of the reward function as it does not index the individual. However, as mentioned in the Limitations section, the authors basically use the same R for all agents. So I believe that indexing R_i and then mentioning that all the R_i’s are the same would be more clear. It does cast some doubt on why the POSG formalism is more suitable.

It is said that each member of a given generation learns on an independent instance of the environment with frozen agents from the previous generation and that for each agent inidividually, the problem is a POMDP. So there appears to be no benefit in introducing POSGs.

Baselines: only one baseline RL^2. And the performance difference can likely be explained by the assumption of a full state oracle. It does not seem a fair comparison to assess the algorithms with different environment access assumptions.
“To facilitate (1) during training, but not at test time, we assume access to an oracle −i with fixed −i parameters θ̂ . This oracle is able to condition on the full state s, which we refer to as privileged information. Agent i can observe the behaviours of −i as they jointly act in an environment.”

The RL^2 seems to be an essential part of the algorithm (Appendix A) yet it is not explained. Moreover, RL^2 too passes on hidden state information so the comparative benefit should be explained as well.

The need for two separate phases of evolution is not clearly explained. Also it is not clear how they connect.

It is not clear how RL^2 and the proposed technique are comparable in terms of the total number of evaluations but also in the observations and learning setting.
Algorithm 1: 
 p_obs, phi, and theta are not being used
 
 there is no need to repeat line 19 for all i. It can be put at the end of the generation since it is the same for all i.

 l.11: index n is undefined

There is limited theory development.

There is no explanation of why the noisy oracle can work well.
questions: questions: The authors mention the two variants of RL^2 show the learning for the length of a single lifetime as well as the full length combining generations (equivalent to the length of their own algorithm). If so, then how come Figure 4 shows the same length for RL^2 in both cases? 

Why does the noisy oracle work well? Is this because in the evaluation phase, only an observation can be provided so there is less discrepancy between the phases?

“We model cultural accumulation as taking place within POSGs. Each member of a given generation gm+1 learns in parallel on an independent instance of the environment, which also contains static members of the previous generation, gm.  Therefore, from the perspective of a given agent i in gm+1 this is a POMDP, since the prior, static generation can be considered part of the environment.”
Can this be clarified? For example, does this mean the policies of other agents are frozen? And why do we need the framework of POSGs if actually the problem to solve is only POMDP.

What is the importance of having the POMDPs being sampled from a distribution to demonstrating the proposed approach? I don’t clearly get this from the text. Also, it appears that actually also the environment is just a POMDP with observations indicating the task.

What are the hyperparameters for RL^2

What is the benefit of sharing the recurrent state at one step vs sharing the observation at each time step?

How would you implement this in a physical setup?
limitations: limitations: The authors mention limitations but not the strongest ones: 
    • the presence of the full-state oracle.
    • The applicability
paper_summary: paper_summary: The paper "Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning" introduces the concept of cultural accumulation in reinforcement learning (RL), where agents benefit not only from their own experiences but also from the knowledge passed down from previous generations, akin to human cultural evolution. The authors propose two models for this accumulation: in-context accumulation, which occurs within single episodes, allowing fast adaptation, and in-weights accumulation, which embeds knowledge in neural network weights over longer training periods. Through experiments on tasks such as Memory Sequence, Goal Sequence, and the Traveling Salesperson Problem (TSP), the paper demonstrates that agents trained with cultural accumulation outperform those trained without it. This work represents the first demonstration of general models achieving emergent cultural accumulation in RL, opening new directions for more open-ended learning systems and providing fresh insights into modelling human cultural processes with artificial agents.
strength_weakness: strength_weakness: 1. Complexity of Models: The proposed models may be complex to implement and require significant computational resources, which might limit their applicability.
2. Scalability Concerns: While the models work well in the presented tasks, it is unclear how they will scale to more complex or real-world scenarios.
3. Limited Real-World Applications: The paper primarily focuses on theoretical and controlled environments. More discussion on potential real-world applications and implications would strengthen the paper.
questions: questions: 1. How do you anticipate the scalability of your models to more complex or real-world tasks beyond the experimental environments used in this paper?
2. What are the computational resources required to train these models, and how do they compare to traditional RL methods?
3. How do you determine the optimal balance between social learning and independent discovery during training?
4. How do your models handle diverse or rapidly changing environments where cultural knowledge might quickly become outdated?
limitations: limitations: 1. While the paper demonstrates the models on specific tasks, there is limited discussion on the scalability and complexity of the models in more complex or real-world scenarios.
2. The authors do not provide a detailed analysis of the computational resources required for their models, which could be a limitation for practical applications.
3. The potential real-world applications and implications of the proposed models are not thoroughly explored. Discussing these aspects would provide a clearer picture of the practical relevance of the work.
paper_summary: paper_summary: This paper introduces the problem of modelling cultural accumulation in populations of deep RL agents, with evolution happening through non-communicative social learning. The paper introduces 2 setups for cultural accumulation: one where the agents learn in-context from other agents within an episode composed of several trials through meta-RL (called the in-context setup), and one where the agents learn from other agents whose behavior is sometimes visible in the agent's observations (called the in-weight setup).

In the in-context setup, agents are first trained alongside a noisy oracle (with access to the oracle annealed to 0 over the course of the episode) but imitating the oracle is not enforced in the loss; social learning is implicit. The oracle is kept noisy to encourage agents to learn on their own, so that cultural accumulation can occur. At evaluation time oracles are replaced by the previous generation of agents. There is no separate phase for training in the in-weights case.

The paper tests algorithms for cultural accumulation in three experimental environments. The first is a sequence memorization task, the second is a goal sequence memorization task in a gridworld with egocentric view, and the third a multi-agent, partially-observable traveling salesperson problem. Baselines are RL$^2$ algorithms with no social learning (single lifetime) for the in-context setup, and single lifetime RL training for the in-weight setup. The paper shows consistent improvement on all tasks when cultural accumulation happens with respect to single-lifetime baselines.
strength_weakness: strength_weakness: * The paper could be clearer: the fact that there are multiple approaches on multiple environments makes the paper complex and difficult to parse. Maybe the in-weights results could go in the appendix and more space could be devoted in the main text to giving examples or longer explanations of the tasks, as well as of the meta-RL setup? I think that some notations might need polishing, see Questions section below.

* The paper distinguishes itself from [Perez et al 2023, 2024] by adopting a model of cultural evolution based only on imitation of non-pedagogic behavior, i.e. agents imitating other agents doing the same task. While this is a completely valid form of social learning and definitely present in human learning, the first introduction paragraph suggests that this is the main mechanism leading to humanity's success as a species. This is not the case, and it is known that language as a means for instruction and rapid communication of knowledge, explicit teaching, and shared norms and institutions are an important (and human-specific, whereas imitation of behavior occurs in great apes) component of cultural evolution (see Henrich 2019 as well as Tomasello 2019). This should be discussed in the paper.
questions: questions: * What is the deep RL algorithm underlying both the in-context and the in-weights setup? I am not sure this is mentioned in the main body of the paper;

* What is the extent to which these results could serve as a model for human evolution, knowing how much RL agents and humans differ, and how different the current setup is compared to human learning setups? How do such experiments compare, as far as modelling of human evolution is concerned, to simple models from the cultural evolution literature like those of [Enquist et al, 2008] (or of [Fogarty et al 2015] in innovation)? Could more realistic experiments be devised?

* Couldn't there have been simpler ways to model cultural evolution using LLMs instead of RL agents, since they already have meta-learning capabilities? (for instance, LLM agents with skill repertoires like LMA3 [Colas et al 2023] or Voyager [Wang et al 2023]).

* (minor) L181 the claim about populations should say whether this is a simulation or human result;

* Why does the TSP agent not benefit that much from more than 1 transmission event, do you think? Is there an exploration problem?

* Why is cultural accumulation better than single lifetime for these tasks? Is there a plasticity problem with the single-lifetime agent? In evolution a part of the advantage of generational transmission is that environments are changing and populations need to be adaptable. There are also aging processes and the difficulty of maintaining lifeforms for arbitrary amounts of time. This is not the case in simulated agents for fixed environments, so how come cultural accumulation provides an advantage in this setting? Are there differences in the amount of exploration in the generational vs single lifetime experiments?

### Notations
(minor)

* l68 what is the observation space? shouldn't the observation function be defined on $O: \mathcal{O} \times S \rightarrow [0, 1]$, defining a joint distribution on observations and states (assuming that $\mathcal{O}$ is the observation space)? 

* The $g_m$ notation is not clear when first encountered, are these model weights?

* l70 do the parameters depend on the time t?

* l105 Not sure I get this notation: $g_{m+1} = f (·|g_m)$, is $g_m$ a distribution? Why not just $f(g_m)$?
limitations: limitations: * The paper is concerned with tasks where a clear reward function is defined beforehand. While this is important and models some aspects of tasks early humans had to face, they are not open ended and not subject to variation. Studies of cultural evolution in simulation must eventually tackle open-ended domains, and even better, domains where the population of agents themselves alter the task landscape as they evolve (see for instance how language or the invention of water containers has influenced human evolution).

* The skills learned by agents are not compositional, and thus do not model the stepping-stone aspect of human cultural evolution that allows populations to build upon existing sets of knowledge or combine them to create larger cultural adaptations (the way the Voyager agent implements skill compositionlity by writing programs could be a source of inspiration here);

* The exploration/innovation aspect of cultural evolution is under-represented in this paper. Could exploratory behavior also be meta-learned as it leads to higher long-term innovation and thus reward? Are the test environments complex and variable enough to test this? 

* The in-context setup requires access to an oracle to steer agents towards learning to imitate other agents behavior. I acknowledge that this paper does not claim to model the emergence of social learning during evolution, but it would have been significant to get the agents to imitate one another without access to optimal behavior on the task.
paper_summary: paper_summary: The authors introduce two methods for learning agents to trade-off imitation and exploration across generations, by incorporating the behavior of noisy oracles and/or the best-performing agents in prior generations into the observations of the next generation of agents. Two settings are studied: in-context learning and in-weights learning.  The in-context learning involves:
- a training phase, in which a meta-RL algorithm learns a recurrent RL algorithm that uses observations from a noisy oracle to learn to solve a POMDP by updating a hidden state
- a test phase in which the RL algorithm is frozen but the hidden state can still change, and the oracle is switched out with the agent with the highest performing hidden state from the previous generation. 

The in-weights accumulation has no train-test split or hidden state, and just trains the RL algorithm from scratch at each generation, allowing it to observe both the oracle and the best agent from the previous generation. Noise is added to the oracle and the probability of observing it is decreased over time, so the agents must learn to explore and act independently.
They demonstrate that their methods outperform non-cumulative/single life baselines in three simple partially observable tasks.
strength_weakness: strength_weakness: - The writing about the algorithms is quite unclear, especially the training algorithm of the in-context accumulation and the in-weights accumulation algorithm are only partially explained in the main text while the pseudocode is relegated to the appendix (see questions section) the paper could benefit from more thorough explanations of the methods in the main text 

- The ability to observe an oracle with access to privileged information (even with the addition of noise) seems like an unrealistic choice, artificially making the problem easier especially in the in-weights setting, since the oracle can be observed throughout rather than only at training time. Even worse, the method is highly sensitive to how much noise is added to the oracle, and the optimal noise amount also varies significantly across settings. Why can’t the agents in this approach learn to learn robustly from prior generations without an oracle?

- There is only one baseline used in each setting, and it is unclear how much of an even playing field they’re tested on. How much were they tuned, e.g. by adjusting the learning rate decay if premature convergence was a major factor? Could they access the oracle demonstrations in any way, or could only the accumulation algorithms observe the oracle? Equation 1 does not account for the population size N, just the number of generations- shouldn’t the baseline’s experience budget be multiplied by another factor of N?

- Cultural accumulation is not just passing down information about the unobserved state, but can also involve passing down procedural knowledge (e.g. mathematics or dance). the authors claim that in-weights accumulation can be interpreted as updating procedural memory, but the environments they test in do not seem to have the right type of complexity to demonstrate this. It would help if the authors tested in a setting that is fully observable but with more complex dynamics requiring the development of skills, such as half-cheetah

- Only observing the best performing agent from the previous generation seems unnaturally limited - humans get to observe a range of behaviors from other humans and may learn from the mistakes of those who perform poorly, or could combine together two suboptimal agents’ strategies to make an optimal one.
questions: questions: - How exactly is the oracle’s behavior incorporated into the observation? 

- How is a^{-i} used by the in-weights algorithm? 

- How sensitive is the approach to the population size?
limitations: limitations: yes

Comments: title: Thank you
thank you for further clarifications
Taking on feedback and responding to comments
Taking on feedback and addressing further concerns
thank you for the response
comment: We thank the reviewer for engaging in this discussion, considering our responses and raising their score. We are certain that the points raised on clarity and the additional baseline run in response to the reviewer's comments will have a positive impact on the quality of the final manuscript.
I commend the authors for their thorough discussion and additional work during this period. There are no major red flags so I will increase my score to 5.
Thank you for your kind response and for highlighting points worth clarifying in the updated manuscript; we greatly appreciate the time taken to engage with our work in depth and provide suggestions that will no doubt improve the quality of the final version.

We are glad that you see our paper as providing a significant contribution and thank you again for your efforts in ensuring its impact by strengthening its presentation.
Dear Authors,

Thank you for your detailed and thoughtful response to my initial comments. I appreciate the clarifications you have provided regarding the complexity, scalability, and potential real-world applications of your models. Your explanations have helped to address many of the concerns I raised, and I commend you for the thoroughness of your rebuttal.

I would like to offer a few additional comments and suggestions:

Complexity and Computational Efficiency:

Your clarification on the minimal changes required to existing RL algorithms and the efficiency gains from cultural evolution is valuable. It might be beneficial to highlight these aspects more prominently in the manuscript to preempt concerns about complexity and resource demands from other readers as well. This could be particularly useful in the introduction or discussion sections to frame the work within the context of practical RL implementations. Please ensure that these details are updated in the paper.


Scalability and Broader Applications:

The results you presented for Atari Breakout, Freeway, and Brax Half-Cheetah are promising. Including these results in the main text, or at least in an appendix, could strengthen the paper by providing concrete evidence of the models' scalability beyond the initial set of environments. Additionally, a more detailed discussion on how these findings might generalize to other real-world tasks would enhance the paper’s impact. Please consider updating the manuscript to reflect these new results and discussions.
Real-World Applications:

Your response about modeling human cultural accumulation “in-silico” is compelling. To further enrich this discussion, you might consider elaborating on how these models could be adapted or extended to address specific real-world challenges. For example, could the approach be tailored to specific industries or applications, such as healthcare or autonomous systems? Including a few concrete examples or case studies could make the potential impact of your work even clearer. It would be valuable to update the paper with these considerations.


Balancing Social Learning and Independent Discovery:

The insights you provided on balancing social learning and independent discovery, particularly regarding the role of oracle noise, are intriguing. It could be useful to explore this balance further in your experiments or discussion, perhaps by analyzing the sensitivity of your models to different levels of oracle noise or by suggesting guidelines for selecting this parameter in practice. Incorporating these insights into the paper would provide further clarity for the readers.


Overall, I believe your paper makes a significant contribution to the field of reinforcement learning, and these additional points are intended to further strengthen its presentation and impact. I recommend that the paper be updated with these additional details to ensure that it fully reflects the comprehensive nature of your work.

I have decided to leave my score unchanged.


Thank you
Thank you for bringing to light points that will improve the paper's clarity and for raising your score. We greatly appreciate the time spent reviewing the paper and engaging in this discussion.
We would like to kindly ask that the reviewer considers our most recent comment and ask if they have any further questions or concerns.
Regarding Breakout MinAtar, given the amount of fluctuation in the return I do not find it convincing that generation 3 performed better than previous generations just because the return at step 1200 is higher (it appears to dip below generation 1 less than a hundred steps earlier).

I am satisfied with the authors' other replies. I believe the clarity of the paper will be greatly improved if the authors make the changes they describe in their response, and my major concerns have been addressed, thus I will increase my score.
Thank you for your kind response. We humbly ask if you will consider raising your confidence rating and/or score in light of having addressed your questions and comments. Equally, if there is any further information we can provide, please do let us know.
Thank you for taking the time to read our rebuttal and for sharing your further concerns. We strongly believe that these are addressable and endeavour to do so within this comment.

> It should be made more clear in the text that the figures relate to the evaluation and not the training phase.

We take this onboard and adapt our captions accordingly.

> How come even in in-weights training, the plot for the single-lifetime training is essentially a fixed line?

Thank you for pointing this out! To ensure that our single-lifetime baseline is fair, we use the same population size (i.e., number of seeds) as for accumulation and report the best performing agent's (i.e., argmax over seeds) maximum achieved return over the duration of its training as the dashed single-lifetime baseline. We now explicitly state this in the manuscript.

> We currently have no view on the learning curves of RL^2.

We can provide these in the Appendix. We would like to have been able to include them in the rebuttal if they are of importance to the reviewer, but they were not initially requested and we cannot upload new figures or amend the pdf during the discussion period.

> There are potentially a lot multi-agent RL techniques to compare to [...] techniques for POSGs, implicit and/or explicit communication in multi-agent RL...

We would like to humbly remind the reviewer that we use the setting of a POSG to introduce the formalism for our algorithms before explaining how the setting reduces to a POMDP because the policies of previous generations are fixed. This means that we are training successive single-agent RL policies with PPO and using the methods we introduce to achieve performance improvements via cultural accumulation. For in-weights accumulation, the baseline of partial network resets [1] therefore makes sense, which we show *additionally* improves accumulation itself in Figure 5 (right).

Approaches to implicit communication in MARL [2, 3] assume that agents are incentivised to *learn to communicate* due to cooperative rewards. We make no such assumptions, as each agent in our study is simply maximising its own independent reward. We believe that combining our work with implicit or explicit learned communication in cooperative settings would be an exciting direction for future work. 

> ...or indeed cultural transmission across generations...

Could the reviewer please clarify what they mean by a baseline here? We are precisely exploring cultural transmission across generations, where previous work has explored a single step (i.e., one generation) of cultural transmission [4, 5].

> It is not clear what are the comparative scores of the techniques compared to the state-of-the-art.

We would like to emphasise that prior work on social learning and cultural transmission in RL [4, 5] has focused on learning from an expert for a single generation of transmission, both in a setting similar to our in-weights setting [4] and where human experts provide demonstrations at test time [5]. In both cases, the reported baselines are ablations of their method, as the purpose was to demonstrate that social learning can be achieved and provide performance benefits in RL. In our work, we extend these ideas to span multiple generations, whilst making no additional assumptions and removing the assumption of having an expert at test time from [5] by creating this generational bootstrap.

The goal is therefore not to achieve state-of-the-art performance on a set of benchmarks, but to demonstrate that cultural accumulation can be modelled in RL and that it can reap performance benefits relative to running the same algorithm for one continuous lifetime.
 
In an effort to further address this concern, we provide the final performance of successive generations of policy distillation (i.e., generational training) [6] in comparison to the final performance at each generation of in-weights accumulation.

| Task                           | Generational Training     | In-Weights Accumulation |
|-------------------------|-----------------------------|------------------------------|                                  
| Memory Sequence    | 0.6, 2.7, 6.6, 7.5, 8.5      | 0.6, 4.2, 7.9, 8.3, 11.4      |
| Goal Sequence         | 0.7, 1.8, 2.9, 3.1              | 0.7, 3.7, 4.2, 4.3               |

> ...do not seem to mention which noise level is used for the results in the evaluation plots.

We take this onboard and now include them in our stated hyperparameters.

[1] The primacy bias in deep reinforcement learning, Nikishin et. al, 2022

[2] Learning to communicate implicitly by actions, Tian et. al, 2020

[3] Foraging via multi-agent RL with implicit communication, Shaw et. al, 2021

[4] Emergent social learning via multi-agent reinforcement learning, Ndousse et. al, 2021

[5] Learning few-shot imitation as cultural transmission, Bhoopchand et. al, 2023

[6] Open-ended learning leads to generally capable agents, Stooke et. al, 2021
Thanks for the response. I have some further comments.

I think it should be made more clear in the text that the figures relate to the evaluation and not the training phase. Also, if there is no distinction between the evaluation and the training phase, how come even in in-weights training, the plot for the single-lifetime training is essentially a fixed line even though the agents have the same number of environment interactions? We currently have no view on the learning curves of RL^2.

There is still only one comparison in the paper, to RL^2, which in a way is an ablation. An ablation study is important but this is quite minimal.  There are potentially a lot multi-agent RL techniques to compare to but this has not been done. Empirical comparisons to techniques for POSGs, implicit and/or explicit communication in multi-agent RL, or indeed cultural transmission across generations, would be interesting to see but are absent. Also, it is not clear what are the comparative scores of the techniques compared to the state-of-the-art.

Last, the authors show the effect of different noise levels but do not seem to mention which noise level is used for the results in the evaluation plots.

While I appreciate the authors’ extensive responses and additional experiments, the claims remain overly broad and it is not clear how well the method fairs when compared to algorithms making similar assumptions.
Thank you for taking the time to read our rebuttal and for sharing your further questions. We endeavour to address these in this comment.

# Further clarifications

In addition to providing the following clarifications, we will update our manuscript to explicitly state them. 

> Line 13 of "Algorithm 3 In Weights-Accumulation" says "Set oracle visible in $o^{i_n}$" - so I guess this must be a typo?

Thank you very much for identifying this! Yes, this can be considered a typo, here we simply mean the previous generation by 'oracle'. We will adjust our manuscript to reflect this and avoid further confusion.

> Are you saying that for population size N, you train N different seeds for the baseline model and select the best one? [...] Is the value plotted in the horizontal dashed line the highest return ever achieved?

This is correct and thank you for pointing out that this has not been explicitly stated. Taking the argmax over seeds is also the reason that there is no error region on the single-lifetime in-weights baselines.

> Are you implying that you did just as much tuning for both the baseline and the accumulation? [...] if there was specific attention paid to tuning of the learning rate and learning rate schedules...

We indeed carefully tuned learning rate schedules for the single-lifetime in-weights baselines. As mentioned, we found that tuning hyperparameters for single-lifetime baselines also contributed to improved performance for accumulation (i.e., performance improvements sustained over more generations), including the tuning of learning rate decay. Given that this has emerged as important information to the reviewer, we can include these sweeps in the Appendix of a camera-ready version of the manuscript.

We would also like to state that in-context accumulation overcomes an entirely different issue to premature convergence, one of effectively exploring a new environment over long contexts.

> I don't see this in appendix D? It doesn't comment on results, just environment details.

We sincerely apologise for this oversight. In our updated manuscript, we have included this information in the Appendix but it was not present in the original submission. We provide the contents at the end of this comment.

> ...except for breakout MinAtar where generations 1-3 appear to perform the same...

Generation 3's final performance is 50.8 whereas generation 2's final performance is 44.3, but we concede that generations 1 and 2 are approximately the same. 

> Could you please also show the error bars and the single lifetime baseline performances in your plots?

We are unable to upload new figures or update the pdf during the discussion period, so we provide the baselines (once again as a maximum over the same population size) numerically here and provide the numeric maximum difference in returns between seeds (over the training of all generations) to provide some information regarding error bars.

| Task                           | Single-Lifetime Baseline | Maximum Return Range |
|-------------------------|-----------------------------|------------------------------|
| Minatar: Breakout      | 36.5                                | 8.2                                    |
| Minatar: Freeway       | 39.1                                | 3.2                                    |
| Half-Cheetah             | 4773.8                            | 219.6                                |

# Appendix on selective social learning

In the Goal Sequence experiments, we select the best performing agent of the last generation for the
current generation of agents to observe during training, automating the selection process. In human
and animal cultural accumulation, this selection is instead learned through prestige cues Barkow et al.
[1975], Horner et al. [2010]. Thus, in the Memory Sequence experiments, we do not automatically
select the best of the past generation for the agents to observe. Instead, the agents can observe the
entire last generation. 

Jerome H Barkow, Akinsola A Akiwowo, Tushar K Barua, MRA Chance, Eliot D Chapple,
Gouranga P Chattopadhyay, Daniel G Freedman, WR Geddes, BB Goswami, PAC Isichei, et al.
Prestige and culture: a biosocial interpretation [and comments and replies]. Current Anthropology,
16(4):553–572, 1975.

Victoria Horner, Darby Proctor, Kristin E Bonnie, Andrew Whiten, and Frans BM de Waal. Prestige
affects cultural learning in chimpanzees. PloS one, 5(5):e10625, 2010.
Thank you for answering my questions and addressing many of my concerns with the paper. I still have some remaining questions and concerns: 

>we make no use of an oracle for in-weights accumulation

Line 13 of "**Algorithm 3** In Weights-Accumulation" says "Set oracle visible in $o^i$" - so I guess this must be a typo?

**Clarifications**

> Populations train in parallel in independent environments for baselines and the main algorithm.

Are you saying that for population size N, you train N different seeds for the baseline model and select the best one? I can't see this specified anywhere. I'm also just noticing now that there are no error bars shown for the single lifetime baselines- is the value plotted in the horizontal dashed line the highest return ever achieved? 

> The baselines use the same underlying RL algorithm and have the same hyperparameters. The best hyperparameters found for baselines and accumulation were the same.

This does not actually answer my question: I asked how much you tried tuning the baseline's hyperparameters, and specifically if you tuned the learning rate decay. Are you implying that you did just as much tuning for both the baseline and the accumulation? I would be satisfied with that, if there was specific attention paid to tuning of the learning rate and learning rate schedules. My underlying concern is that cultural accumulation could be no more effective than a well chosen learning rate schedule (especially one with warm restarts, e.g. [1]), if the main issue with the baselines is premature convergence. 

> Appendix D details how the results on Memory Sequence assess the selective social learning described by the reviewer. 

I don't see this in appendix D? It doesn't comment on results, just environment details

**On Environments**

Thank you for sharing results on the environments involving more complex control. It is encouraging to see that each generation seems to outperform the last, except for breakout MinAtar where generations 1-3 appear to perform the same (why do you think it is failing there?) could you please also show the error bars and the single lifetime baseline performances in your plots?

Loshchilov, Ilya, and Frank Hutter. "Sgdr: Stochastic gradient descent with warm restarts." arXiv preprint arXiv:1608.03983 (2016).
I am glad that you took the time to answer my questions and incorporate my comments. I hope the paper is accepted and am looking forward to see follow-up work in more complex transmission setups.