Bias Amplification in Language Model Evolution: An Iterated Learning Perspective
Authors: Yi Ren
Shangmin Guo
Linlu Qiu
Bailin Wang
Danica J. Sutherland
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
LLM Agent
Self-improvement
Cognitive Science
Bayesian
Iterated Learning
Primary area: natural_language_processing
TL;DR: As LLMs undergo self-evolution, it is imperative for us to deepen our understanding of this process.

Abstract: With the widespread adoption of Large Language Models (LLMs), the prevalence of iterative interactions among these models is anticipated to increase. Notably, recent advancements in multi-round on-policy self-improving methods allow LLMs to generate new examples for training subsequent models. At the same time, multi-agent LLM systems, involving automated interactions among agents, are also increasing in prominence. Thus, in both short and long terms, LLMs may actively engage in an evolutionary process. We draw parallels between the behavior of LLMs and the evolution of human culture, as the latter has been extensively studied by cognitive scientists for decades. Our approach involves leveraging Iterated Learning (IL), a Bayesian framework that elucidates how subtle biases are magnified during human cultural evolution, to explain some behaviors of LLMs. This paper outlines key characteristics of agents' behavior in the Bayesian-IL framework, including predictions that are supported by experimental verification with various LLMs. This theoretical framework could help to more effectively predict and guide the evolution of LLMs in desired directions.

Bias Amplification in Language Model Evolution: An Iterated Learning Perspective
Yi Ren UBC renyi. joshua@gmail com
Shangmin Guo University of Edinburgh s. g.guoddd..o ac.
Linlu Qiu MIT linluqiudmit. edu
Bailin Wang
Danica J. Sutherland
MIT
UBC & Amii dsuth8cs. ubc.
kailin.wang28@maiil.com
Abstract
With the widespread adoption of Large Language Models (LLMs), the prevalence of iterative interactions among these models is anticipated to increase. Notably, recent advancements in multi-round on-policy self-improving methods allow LLMs generate examples for training subsequent models. At the same time, multi- agent LLM systems, involving automated interactions among agents, are also increasing in prominence. Thus, in both short and long terms, LLMs may actively engage in an evolutionary process. We draw parallels between the behavior of LLMs and the evolution of human culture, as the latter has been extensively studied by cogmitive scientists for decades. Our approach involves leveraging Iterated Learning (IL), a Bayesian framework that elucidates how subtle biases are magnified during human cultural evolution, to explain some behaviors of L.MMs. This paper outlines key characteristies of agents' behavior in the Bayesian-lI. framework, including predictions that are supported by experimental verification with various LLMs. This theoretical framework could help to more effectively predict and guide the evolution of LLMs in desired directions. The code for experiments available at https: : //gtthub..ttttt// com/ Joshua Ben/iICI.
Introduction
Recent large language models (LLMs) have shown remarkable instruction-followin ability and an increasing number of applications; it is thus reasonable to expect they are likely to become more widespread. Moreover, interactions between LLMs (either multiple models, or different generations of the same model) may also become very commonplace in the near future. In fact, many recent works consider iterative on-policy gelf-data-augmentatte of human- generated supervisions, e.g., self-instruct (Y. Wang et al. 2022), self-refine (Madaan et al. 2023), hypothesis refinement (Qiu al. 2024), self-distill (C. Xu al. 2023), self-instruct (Y. Wang et al. 2022), self-reward (Weizhe et al. 2024), self seff-feedback (W. Xu et al. 2024), RAFT (Dong et al. 2023), ReST (Gulcehre et al. 2023), iterated DPO (Xiong, Dong. Ye, Zhong, et 2023), OAIF (Guo al. 2024), SPIN (Z. Chen et al. 2024), and many more. Whether the model's '' knowledge is updated through in-weight or in-context mechanisms, these methods involve an LLM learning from a corpus (comprising data examples evaluations) generated by another LLM (or itself), and subsequently transferring this acquired knowledge to others. Looking towards the long term, the future Internet may (for better worse) contain substantial portions of LLM -eMerreeddted text, which will, in turn, be employed for training the subsequent generation models. thus seems important to begin studying what this process will mean for future models.
solutions break through the bottleneck
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Although these self- -efroviig methods demonstrate considerable improvements on various bench- marks. a systematic understanding of why they work and what is their limitations is still missing. Some analysis of knowledge distillation might bring insights (Mobahi et al. 2020), as learning from data generated by another model a type of distillation. But precisely analyzing the L.LM's s behavior on specific samples becomes increasingly difficult as it grows more complex. Instead, a behavioral- level analysis might be fruitful, akin to how the Bayesian framework can aid in comprchending the human cognitive system (T. L. Griffiths et al. 2023). By conceptualizing the LLM as an intelligent agent, we can draw parallels between its behaviors and the cultural evolution observed in humans. Iterated learning (IL), framework proposed to study the evolution of knowledge and beliefs through a chain of learning among Bayesian agents (Kirby et al. 2007), stands out as a promising candidate for achieving our goals.
In this paper, we start by introducing the Bayesian-II. framework, demonstrating that agents engaged in such a process gradually amplify bias in their priors. This amplification process can be steered by introducing an interaction phase that "filters" or "re-ranks" the messages generated by the agents. Next, we theoretically justify that the in-context behavior of LLMs can be approximated by Bayesian update, establishing a crucial link to the LLM system. To validate our claims, we conduct numerous experiments across different settings. Depending on the beneficial or detrimental nature of the bias, we propose various strategies to guide the evolution of LLM. The key contributions of this work are: 1) establishing the first Bayesian amalysis of the full interactive learning process (including an interaction phase); 2) applying this framework to LLM agents and describing their evolution theoretically: 3) validating the theory and demonstrating how to guide LLM's evolution using experiments. We believe that our analysis can enhance our understanding of LLMss, and aid in designing more effective algorithms for alignment, bias mitigation amplification, and similar tasks.
2 Background and Related Work
2.1 Iterated Learning
Iterated learning (II.) is a hypothetical procedure to simulate how the lendency of specific properties of human culture or language gradually emerges and becomes dominant. based on studying the behaviors of a chain of intelligent agents. From the perspective of an individual agent, the process involves initially acquiring knowledge from its predecessor (imitation), refining its beliefs while using them to conduct tasks (interaction with the world), and subsequently imparting its knowledge to the agents in the next generation (transmission). Cognitive scientists have applied this framework to explain various evolutionary phenomena of human society, including the emergence of compositionality in human language (Kirby et al. 2015), patterns in human object categorisation (T. L. Griffiths et al. 2008), and the evolution of color naming systems (Carlsson al. 2023). The framework has also seen recent success with neural network agents, including in emergent communication (Guo et al. 2019; Ren et al. 2020), machine translation (Y. Lu et al. 2020), visual question answering (Vani et al. 2021), large vision-language models (Chenhao et al. 2024), and representation learning (Ren al. 2023), indicative that this framework could also be useful for more general deep learning systems, like LLMs.
2.2 Related On-policy caff-data-augmeettttin Methods
While the theoretical guarantees for the Bayesian-IL framework studied in this paper rely on several assumptions, we posit that the behaviors observed for many recent "iterative eeff-dtt-augmmnnation" methods in LLM can be at least partially explained by the theory. We will now overview the basic assumptions, and how they fit with recent LLM approaches (more discussions in Appendix A).
First, the theory assumes "self-evolution," where all agents in different generations share the same initial knowledge. Methods like self-refine (Madaan et al. 2023) and hypothesis refinement (Qiu et al. 2024), which require the LLM to refine its output by the feedback from an identical LLM for several rounds, satisfy this assumption. Self-distill (C. Xu et al. 2023) and self-instruct (Y. Wang et al. 2022), if the models involved in different generations are the same, do as well. On the contrary, the super-alignment setting (Burns et al. 2023), where a stronger model is trained using the data generated by another weaker model, do not strictly fit with our analysis. However, if all the models are trained using a similar corpus, so that their initial knowledge should be similar, our analysis might still hold partially.
2

---

The theory also assumes the information transferred among agents is in the form of data examples, as in RAFT (Dong al. 2023) and ReST (Gulcehre et al. 2023). Both methods consider multi- generation transfrransr process, during which the bias is introduced by re-ranking the transferred data. Methods like self-reward (Weizhe et al. 2024) and self-refine (Madaan al. 2023), which requires one agent to evaluate another agent's ' response, do not directly fit this assumption. However, if we also consider the evaluation as part of the data generated by the agent, the Bayesian-IL framework can still bring some insights. Furthermore, as analyzed in Ren and Sutherland (2024) that many preference alignment methods like direct preference optimization (DPO, Rafailov et al. (2024)) will naturally amplify the preference hidden in the pretrained model's prior. Then, those multiple- generation DPO variants, e.g., iterative DPO (Xiong, Dong, Ye, Z. Wang, et al. 2023), might face a more serious risk of amplifying malicious bias.
In summary, although the assumptions of our Bayesian-Il. framework might not be satisfied by all practical algorithms, the general trends depicted by it, e.g., the bias amplification, the necessity of a good interaction phase, etc., would still hold. Please refer Appendix A for more discussions.
Bayesian Analysis of Iterated Learning
3.1 Notations and Basic Behaviors of Bayesian Agents
We denote a data pair as d = (z, y). where d E D = X' x y. with I E X' and E Y The (I, y) pair can be question and answer QA problem, the input and label in a supervised setting, or any type of prompt and output for in-context learning. The hypothesis / E H. : X' + y describes the mapping between all possible x and their corresponding y. Note that h can be either explicit or implicit, depending on the task. For instance, in inductive reasoning. h represents the rule determining the output from input examples and is explicit, as the model can directly generate using natural language. Conversely, in is likely to be implicit. this context, h can be highly abstract with varying interpretations, such as the level of conciseness, helpfulness, or even the writer's preference for using rhyme.
where x is topic and is paragraph generated based on
Consider general Bayesian agent whose behavior can be depicted by two basic procedures: learning and sampling. Learning involves updating the agent's knowledge based on observations, while sampling is procedure wherein the agent generates data based on its knowledge. In this context, the agent's ss knowledge is encapsulated by its posterior over the hypotheses, i.e., (h).
In Bayesian learning, we assume the agent holds prior Po (h) at the beginning. Its posterior after observing d = (xi, yt N 1 is calculated as
Pim (h) = P( | d) x p(d h) Po(h,,
(I)
where p(d h) is the likelihood of these N data pairs under a specific h; this usually hard to calculate in practice.
Assume the agent holds a posterior (mm(h) during sampling. Then, given the input signal x, we can sample the corresponding y ~ Pim. (y) z). Based on the fact that h determines the relationship between and y, the above sampling procedure is equivalent to y EE-/,.... (h) [p(y h, h,,) which can be rewritten as h Pt.. (h): y h ply h,) Following the definition of d and the assumption that x is uniformly distributed, the sampling procedure above is equivalent to d ~ Pim (d) x p(d| h)) Pmm(h). If we instead first decide the most probable h rather than sampling h from the agent (maximum a posteriori (MAP), as is perhaps common subconscious behavior for humans), we then generate d by
d~ p(d h²), h" argmax Ptm (h)- NCH
3.2 Iterated Learning of Bayesian Agents
Iterated learning is a hypothetical process simulating how human language gradually evolves to become more efficient when transferred and utilized across generations. Typically, iterated learning repeats of the following three phases, as illustrated in Figure 1: an imitation phuse, where ignorant agent learns from the data generated by its predecessor; an interaction phase, where this agent uses the knowledge to accomplish the task, and hence refine its knowledge; and a transmission phase,
3

---

Multi generation mutti agent system
On policy fineturing algorthmms
Sayesian formulation of the multi gen ite otlve learn
Earcousto TEE
Lett Mo Sigh-g-eeeee li
puthi rnge
Heae of purdag mvrttiing
Canarnt SIro
right warikih
A
20
fowtzuhiinn of Lei
de
dF
Pu
FuLh)
Py(A)
Figure Examples of practical LLM systems that require knowledge transfer among different generations and how we use Bayesian agents to approximate their behaviors. C, 2, and 3 denotes the imitation, interaction and transmission phases respectively.
where this agent generates useful data for the next generation. Combing with Section 3.1, we can get a picture of how h and d evolve as follows.
Initialization: at the beginning of the tth generation, new agent,, whose belief on h follows prior distribution Po(h), is initialized. In lab experiments, (h) represents the belief of a well-educated participant who has not been previously involved in the target experiment. In in-context learning, well-trained LLM also holds a complex and informative (h) based on the enormous corpus it is trained on and the task instructions in the prompt.
Imitation phase: after initialization, agent, then updates its knowledge by observing N data samples d' Following the above learning procedure, the model's posterior should be P(h d d'
Interaction phase: in this phase, the agent will accomplish specific tasks to refine its knowledge. The tasks involved in this phase can be diverse and complex. For example, in lab experiments (Kirby et al. 2015) and emergent communication (Ren et al. 2020), the agent plays a Leiws referential game (Lewis 2008) to rule out hypotheses representing non-bijection between X' and ); in representation learning (Ren et al. 2023), the agent directly conducts supervised learning on the downstream task to inhibit insufficient representations. Although is hard to precisely formalize the behavior of the agent under these tasks precisely, their goals are consistent: we expect to "rule out" unsuitable hypotheses with carefully designed interactions. In an idcalized setting, we might expect the agent's 's posterior to become proportional [(h E Hen) P(h| d where 1(-) is an indicator function and Heif H is the subset of hypotheses that can accomplish the tasks. Broadly speaking, refining using the feedback from humans, LLM, or the environment, which is common in the aforementioned iterative -ir-datt--augmenattion methods, is also type of task implicitly constraining h Hetf-
filtering d'
Trunsmission phase: agent, now comes to the transmission phase, where generates multiple data samples d' for the next generation. The agent will first select the most probable hypothesis based on its current belief and then generate data samples, i.e., df ~ p(d h²*), where ht* = argmax, P(h d This accomplishes one generation of iterated learning.
3.3 Amplitying Biases is a Double-Edged Sword
In iterated learning, we repeat the phases mentioned above get better h and d. The limiting behavior of this process can be described by the following proposition. In short, the bias in Po (h) is guaranteed to be amplified generation- mnmteaation-b-ggeerrtttee by-generation. Imposing appropriate Het (mainly through a carefully designed interaction phase) can control it.
Proposition 1. Consider several Bayesian agents sharing the same prior Ph(h) are conducting iterated learning for T generations. IfT is sufficiently large, any agent, with & > T will have
(hm(h) + 1(h = =h²²)
where TT is a stationary point (e.g. a local maximum) of Po(h) subject to h E Heg.
To prove this, we first analyze iterated learning without the interaction phase. By drawing parallels between IL and EM (expectation maaximizatinn) algorithms, we can prove that h² converges to the with the maximum prior probability. We then consider the interaction phase, which introduces a "selection" pressure to rule out those h & Her. By proving this process does not break the converging behavior of non-interacting iterated learning, we achieve this proposition. (Please refer to Appendix B for more details.)

---

This proposition describes an inevitable bias amplification procedure as long as the model keeps learning from the data sampled from itself (via Bayesian update, distilling, or imitating, as long as the learning increases the model's confidence in these samples). However, in practical applications, we should bear in mind that bias amplification a double-edged sword. If this bias is beneficial, like the simplicity bias in compositional language experiments, IL. will help the model generate the "correct messages" more robustly. Imagine if we only have two possible hypotheses, i... hgood and hoad. where Po(1)goo)) is slightly larger than Po(bbaa). Then by sampling y ~ p(y h, Po(h), half of the chance we will get incorrect y coming from hbad. Although we can get y = argmax, p(y | h,,) (h))) by using an extremely small temperature, the diversity of y provided by the likelihood will then disappear, which is not what expected. How could we get samples that are both diverse and correct? Iterated learning can help with this problem by providing a posterior where Pim (Ayyod)) >> Pim (hood). With this posterior, sampling from ply h, Pmm (h) would be similar to sampling from p(y Regood, z), which solves our problem.
Conversely, amplifying bias also negatively influences the system in several ways. Besides the cases where the bias is malicious (it can be solved by designing an appropriate interaction phase where hhad & Hen). it also influences the model's 8 creativity. Imagine we have multiple good h where Po(hgg1) > Po (ogg))) > Pohhaad), then II. will let us lose h22 even its prior is only slightly smaller than hit. Such a mode decay phenomenon is quite similar to the "recursion curse" mentioned in (Shumailov et al. 2023): a more peaky Piom (h) will make those non-dominating h have a smaller probability, hence it is harder to keep these modalities during evolution. Touvron et al. (2023) also mentioned that iteratively fine-tuning would harm the creativity of the model. The solution could be early stopping the iterated learning or manually introducing more y that comes from hyz during imitation.
In summary, to guide the LLM to self-evolve in an expected direction, we need a good Po(h), a carefully designed interaction phase, and an appropriate evolving time.
4
LLM-based Agents in Iterated Learning
4.1 LLM Behaves like a Bayesian Agent when Sampling
To transfer the Bayesian-IL IL analysis to LLM, we start by showing that the sampling and learning behaviors of an LLM agent can be depicted by Bayesian inference, following a few-shot in-context learning (ICL.) scenario demonstrated in (Xie et al. 2022). In this setting, the message feed to the agent would be an instruction prompt w followed by N examples, i.e., dx = (#t, y In other words, sampling y given the prompt, the examples, and the question Fees can be represented as:
y Py Destts dN. w) A Pema Fiest dv),
where Pinoo is the model's belief conditioned on the instruction w. If we call dN as d (i.e., assume the examples are generated by agents in the previous generation) and assume the test question Tist is uniformly distributed, sampling new data based on instruction and few-shot examples can be expressed as d' ~ Ptmmold d' which is similar to the transmission phase in iterated learning.
Formally linking ICL and Bayesian-IL IL poses non-trivial challenge, however, because the theoretical guarantee of Bayesian-IL relies on obtaining the MAP estimate of h at each generation. This is not immediately evident in ICL. Inspired by Xie et al. (2022), we first de- de-marginalize this posterior predictive distribution using the latent variable h, and then achieve the following proposition:
Proposition Consider that agent A is conducting in-context leurning. If the prompt examples in d' are generated by another agent B with the same prior knowledge (e.g., they come from the same checkpoint and use the same prompt), sampling from the posterior predictive distribution of agent
A. i.e., d' ~ Pimeed d'-1), can be decomposed into: 1.) ht' + argmax,, Pomw d'-l), ), and 2.) d' (umo(d) h(t), where h is a hidden variable that describes the mapping between x and y.
The proof is in Appendix B.3. This proposition bridges LLM and Bayesian agents using its in- context behavior, which we believe is a ubiquitous procedure in any LLM system, irrespective the subsequent information-updaie updating strategy or the final target. For example, in an L.M-ageet system, where no in-weights update exists, the model interacts with other agents (e-g., the human, the internal block of an LLM agent, or the environment) by generating responses based on the prompt and dialog
5

---

history. For LLM's finetuning, where various parameter updating strategies exist, the model also generates responses given the prompts, which is well depicted by the in-context behavior. Although the assumptions in this proposition do not exactly hold for all L.M systems, we believe our analysis can still roughly depict important trends of them. Please refer to Appendix for more discussions.
4.2 LLMs in Different Algorithms have a Similar Target to Bayesian Learning
We then check the learning procedure. First, in pure in-context learning setting like self-instruct (Y. Wang et al. 2022), self-refinement (Madaan al. 2023), hypothesis search (Qiu et al. 2024), etc., the learning can be modeled by calculating the posterior Poww (h d which is identical to the Bayesian learning discussed previously. Then, for those algorithms that require in-weights updates, like self-reward (Weizhe et al. 2024), self-play instruction tuning (Z. Chen et al. 2024), iterative DPO (Xiong, Dong, Ye, Z. Wang, et al. 2023), etc., the LLM might update its Pimw (h) using different loss functions. However, as all of these methods contain procedure that encourages the models to increase their likelihood of the training samples generated by their predecessors, we should expect (d' to be increased after learning. As a result, the equivalent posterior Pimw will implicitly favor those h that can generate d'-1, which aligns with the Bayesian learning target.
5 Experimental Verifications when the Hypothesis is Explicit
We directly verify our analysis above using an inductive reasoning task called Abstract Causal REasoning (Chi Zhang et al. 2021, ACRE), where all LLM agents update their knowledge via ICL. In this task, the model needs to infer and generate the shared rule by summarizing several input-output pairs. Specifically, assume there are M different objects, say [A, M,,,CJJ. CJ. One data pair d = (z. y) is composed of an input x, i.e., list of subset of these objects, and an output that represents the status of the light (could be on, off, or undetermined). In this experiment, the existence of specific object triggers the light be on. The roles played by different objects expressed by the rule h. For example, in the learning stage in generation-l, the model sees three data pairs d' ([B, C] undetermined), (IB] off),, and ( ([A,B, C] on). We then expect the model to guess rule like h' {A: o:fff, C: u:udetermined)), which means A can trigger the light to be on, B cannot, and C is not sure. In the sampling stage, we will feed the above h together with the instructions to the model and hope it generates more examples following p(d h²). Hence the model's 's output might be ([A,C) on), ([A,B), on), and ([C), ,undetermined), Treating the above examples as d', the model in the next generation can induce the corresponding h1+1 by selecting the hypothesis with the largest of Pimwe (h d'). To ensure the generalizability of our analysis, we conduct experiments on GPT3. 5, GPT4, Claude3-hatle and Mixtral-8an -8x7b. Please refer to Figure 2 and Appendix D.3 for more details.
5.1 How the Knowledge of LLM Agents Evolves
Convergence of the posterior. We start from the guarantees mentioned in Proposition I under an imitation-only setting. In this experiment, we choose M = to better illustrate the posterior ability, all LLMs we considered always return rules in the correct format, where the probabilities of all cormat-related tokens are almost one. We can then calculate Pimue (h) or Ph (h d) by multiplying the probabilities of specific tokens in their response (see Appendix D.1 for more details).
distribution Pimw((h (h) (there are 3 = 243 possible h). Thanks to the cntrruction-oolloiiii
We first demonstrate the convergence of Pm (h). i.e., Pm) - 11.), which can be supported by the decreasing of the posterior's entropy, i.e., H (Pitow (h)). As illustrated in the first panel in Figure 3, H (Pmma (h)) gradually decreases to almost zero as iterated learning goes on, which verifies our theory that Pi will converge to a one-hot-like distribution'. Smaller temperature T makes the convergence faster, which matches our intuitions well. To better illustrate how different h evolves during iterated learning, similar to what we did for the compositional language experiment in Appendix C.2, we also provide the probability of all possible h E H in similar fashion. Note that for this problem, it is impossible to get the prior distribution (h)), because we must give the model at least one example as d'. So in the rightmost two panels in Figure 3, we compare the posterior of the first and the sixth generations and that the posterior becomes sparser.
Note that the entropy of uniformly distributed A and one-hot& l roughly 5.34 and 0, respectively.

---

UND
Learing
B
8 C
UND
Gen1 enie
d'
Gen2
UND
Your rule cannot oxplain
A A.C.C. Please refinee
UND
(wokld lor LLM)
Figure 2:
Demonstration of conducting iterated ICL on the ACRE task.
Converged h under different likelihood and priors. We then show how iterated learning amplifies specific biases implied in the prior, i.e., 112 - argmax, (b))), and how the bias and likelihood inftuence the converging behavior. Note that (h) represents LLM's belief given the instruction prompt w, where the few-shot examples are not included. Thanks to the phenomenon mentioned in (McCoy et al. 2023), where the confidence of LLM's prediction is heavily influenced by its degree of familiarity with the output phrases, we can manipulate the prompt to create spurious correlations and hence implicitly control bias in Ph(h) Specifically, we change the name of the last object from "E" to "screen" and add a sentence like "Turn off the screen after the experiment. " in the instruction prompt. Then all h with screenooff would have higher prior under this prompt. We use six different prompts to introduce different levels of biases (see Appendix D.2 for more details).
We then control the strength of the likelihood by selecting different h", i.e. the ground truth rule we want to recover. For the strong likelihood case, we select h² where four objects are being on while there is only one in the weak likelihood case. The status of screen in both cases is undetermined Due to the nature of the ACRE task, i.e., the existence of on-object in the input will trigger the light on, there might be more examples whose outputs are on when the likelihood is strong, Then it is harder for the model amplify the prior bias that favors the status of screen be off. Because the likelihood and prior compete with each other during iterated learning, as illustrated by Equation (I).
This competing relationship can be well depicted by the middle two panels in Figure 3, where track the probability of Pinme (screen: : off) at the end of each generation. The converging speed under different settings correlates with the level of prior bias well. Furthermore, we find it is easier for the bias to be amplified when the likelihood is weaker, as five out of six curves converge to one in the right panel. This trend is more clear in Figure 12 and 13, where curves with the same level of bias are shown together. These results give us a good picture of how the likelihood and prior bias interact with each other during evolution and also verify the correctness of the Bayesian- framework for LLM agents. Plus, we plot the histograms of Pimw (h) under weak-likelihood soioodhhhh-baas case in the rightmost two panels in Figure 3, which also demonstrates the amplified bias (the blue region grows).
Table 1: Results when adding different interaction phases. Column "BOTH" represents the ratio of converged hT who correctly predict all examples in d and have screen off (i.e., r20=off). The Mixtral model does not have relf-refine results, as it violates the instructions too much.
Od Turbo 1166
New Turb 0125
C Clonle 2ar---nnnn--222222000
Misiral
Con.
off
BOTH
Corr. d'
off BOTT
Coer d"
off
BOTH
Con: d' Ps)
ROTH
imitations-nnly
T.56
70%
15%
5.6e1.11
6421.0
90%
30%
5.54.01
20%
10%
w. self refine
7.0+0.60
40%
20%
6.621.11
95%
35%
7.0.0.20
60%
15%
W. hypo search
7.740.21
80%
45%
7.410.66
1008
55%
7.520.67
90%
50%
6.511.97
30%
30%
Influence of the interaction phase and Her. Finally, we introduce the interaction phase and show that l1T' + argmaxhe (o).) Two mechanisms are considered in this experiment: self-refine (Madaan et al. 2023), where the feedback comes from the model's own response; and hypothesis- search (Qiu et al. 2024), where the feedback comes from an external ground-truth interpreter. We can consider the self-refine as using an imperfect Her- In both settings, the LLM refines its proposed h the end of each generation by checking and reporting whether this he can explain all samples in d (details in Appendix D.3).
In this experiment, we give the model different examples in dº, where all these examples can be explained by both h² and h. We first select an from all 162 candidates (3+ x 2) and then create h by changing the value of screen to off. Under this setting, both h² and belong to Han (i.e., mappings
"This phenomenon also inspires to manipulate the prompt inject useful bias in LLM's evolution.
7

---

Shong Le Poodd
Wesk Bolteed
OFTE (N)
Caneration
Condration
Concrston
Figure 3: Leftt: the mean and standard deviations of H (P (h)) of experiments with different h* and d (5 different seeds). Middle two: the probability of screen being off, where different colors represent six different levels of spurious bias. Right: the histogram of all Pime (h) in the first and sixth generation, where the bars are colored based on the value of the last object in h.
with perfect training accuracy in d and h" is what we want our model to converge to. See Table I, where we run experiments under 10 different h and report three quantitative metrics of the last iteration, i.e., 1t=". We first report the number of correct predictions (mean and standard error) in d', which demonstrates how well the method constrains 112** E Hett. The imitation-only method performs the worst, which warns us if the LLM keeps learning from the corpus generated by the agents in previous generations without any evaluation or filtering, even the training accuracy given d" would be harmed. Because hallucination or incorrectness can aggregate through generations. Adding the interaction phase can mitigate this problem efficiently, which is why most of the related works contain a "data-sclection" or clata-reranking" phase. The fact that the hypo-search outperforms self-refine indicates the importance of an appropriate Han. which means good reward (or evaluating) model is crucial for these iterated training methods. Another metric the ratio of ht² with screen : off, which measures how well the bias is amplified (we here assume this bias is beneficial and wish to be amplified, as the compositionality bias in emergent communication example showed in Appendix C.2). We find all these methods can amplify the bias to some extent and hypo-search also performs the best. Last, combined with the requirement of good training accuracy and amplifying bias, report the ratio that the algorithm successfully chose AT* h" As illustrated in the last column of the table, adding an interaction phase with good Hen always brings benefits.
In summary, this section verifies the correctness of the proposed analysis in LLM agents when the hypothesis is observable. The results remind us to pay more attention to whether the bias is beneficial or not and to design better interaction phase as well.
Experimental Verifications when the Hypothesis is Implicit
Section demonstrates that the Bayesian- IL framework can predict the behavior of LLM agents when h is explicitly defined and utilized when generating new examples. This section considers hidden h scenario that is more general in most LLM systems. We start from a few-shot self- data--aggeentttion examples to augment the data pool. In this process, h is
task, where the LLM keeps generating implicitly selected when the few-shot examples are given, as stated in Proposition 2.
Experimental settings. We choose scenario where on-policy self-data-augnee augmentation is repeated for several generations. Consider using an LLLM to generate multiple examples of an acronym- brainstorm task, where each example d is composed of an acronym and the corresponding word list, c.g., Acronym: IL; List: ("infinite" "]oop"]. The h determines the properties of d. We hold a data pool 2 poot. which contains 20 random samples as d' at the beginning of the experiment. In each generation, the model will generate 20 extra examples based on the data generated by itself in the previous round, i.e., d' ~ Pmmma (d d' d' The generated d' will be pushed into D poot. which simulates a scenario in which the available data keeps growing when we conduct self-data augmentation. In this experiment, h is hidden and might have different interpretations. We consider that h represents two types of acronyms, i.e., Neasy. where the acronym is a common word and hpad otherwise. As the training data of the LLMs in experiments is private, we instead use the ranking of the frequency of a word that appeared in common English corpus² as an approximation. We categorize a word as "easy" if its ranking is below 60,000; otherwise, we label it as "hard".
Bias in prior is amplified during IL. Many recent works observe that the LLM prefers to output more common words (i.e., those with higher frequency in the pertaining corpus) (McCoy et al. 2023;
The frequency and ranking comes from Corpus of Contemporary American English (COCA) (Davies 2008).
8

---

Paris Average makk
Auerage stoutbut
Langth
Wih cate apainet STT
14
2,2
4,4
350 200
3033
180
200
600
200 150 300 65
experiments in Section 6. First: how the ratio of easy samples changes in d'. .N is the number of easy examples in d Second: how the average ranking of acronyms changes. Third: how the average length of acronyms changes. Rightmost two: results of on-policy DPO in Section 7. Fourth: average length of the responses. Fifth: win rate against the SFT baseline.
Figure 4: Leftmost three:
Z. Wu et al. 2023), which can be considered as bias towards heas,, i..., Po(heasy) > Po(haad). Since h is hidden and we cannot directly observe it like in the previous experiment, we instead track three quantities: 1.) the proportion of easy samples in all 20 samples for each d'; 2.) the average ranking of d', where all hard examples are ranked 60,001; and 3.) the average length of the acronyms for d'. As in the leftmost three panels Figure 4, the aforementioned bias is gradually amplified during iterated learning whatever the initial proportion of the easy samples in d is.
Interaction phase when h is hidden. As h is inaccessible, which forbid us directly apply hypo- search or self-refine, we instead add a filter on the transmitted data across different generations, which plays similar role as the interaction phase. Specifically, we use a sampled d ~ poo h E Hottt to replace original d in the "imitation-only" setting. Based how we sample d, different constraints on Hell are implicitly imposed. We compare the behavior of five different settings, they are: 1.) Hrandom. where d is randomly sampled from Poooi: 2.) Hhant where only hard examples can be sampled; 3.) Heasy. opposite to the hard setting; 4.) Heasylong. where the easy acronyms with longer lengths are more likely to be sampled; 5.) Heasyshort opposite to the easy-long setting.
See the first several columns of Table 2 that show the ratio of easy examples in d'. Compared with the random setting, all methods expect Hhard finally converges to d' with more easy examples, which means the bias towards easier acronyms would be amplified when Han doesn't 'it impede it. On the contrary, using Hhard successfully restrain this bias, as the average number of easy samples in d' is even lower than that in dº. We can also design composite Heir by choosing two properties of the data. For example, Hewsylong restrains the samples with hard and short outputs, which is why they have more easy but long examples in their d'.
In summary, this experiment verifies that the Bayesian-IL IL framework still works when h is hidden: the bias is amplified generation by generation, implicitly imposing Hen can still guide the evolution direction. Please also refer to Appendix E for more results and discussions.
Table 2: Results when adding different Helr We color the highest and lowest numbers in each column.
No is the number of easy examples in d". Results under different settings are in Table 4 and 5.
satt--aay
Angezat
Arplengn
Randon
Ban D1
n 500ud
a00200 m
0.335 66
13519
37309
7687
1n:25
15871
5.435a IM
4 4.825-0333
5630a1 55
S0iiei 20
47136063
8438
90540 m
0.9550000
056340 6o
35235
3,57
5001
0075
1.500 36
1377-1 40
18840
5 24
Ban
1.80ut pm
19910
31.5
2383
2524
2039
302248
4.3400
Reyfiing
0.97500 0:
0:958-0000
1.808-n 60
1065
9113
5600
6666
7a3d
12066
5.88-6.33
681861.10
7 Experiments on In- In-Weights Learning: On-Policy DPO as an Example
Besides the manually designed experiments in the previous two sections, here we verify our analysis in real preference-tuning task using on-policy DPO (Guo et al. 2024). In each round of the training. the model first samples multiple responses given the prompt (similar to the sampling stage in IL). Then, these responses are evaluated and ranked by another LLM annotator based on their level of helpfulness (the interaction phase in IL). Finally, we select the highest (lowest) ranked samples as the chosen (rejected) response and use a standard DPO algorithm (Rafailov et al. 2024) to train the policy network (the imitation phase in IL.. As described before, each update of the on-policy DPO algorithm can be considered as one generation in iterated learning, because the model keeps updating its parameters using the responses generated by itself. Ranking the responses based on helpfulness is equivalent to imposing Haclpful. As result, the phenomenon of bias amplification, the guiding

---

effect of the interaction phase design, and the influence of spurious correlation, should still hold in this practical setting.
To verify our analysis, we finetune a pretrained 11ama-2-7B model (Touvron et al. 2023) using Antropic-Hil HH dataset (Bai al. 2022) following the on-policy DPO recipe provided in (Guo et al. 2024). We study the length bias demonstrated in (Dubois et al. 2024), which means the LLM tends to prefer longer responses when answering questions. We first show that such a bias will be significantly amplified by multi-generation self-improvemen method (on-policy DPO) compared with a non- self-iterated method (RLHF, (Ouyang et al. 2022)). As demonstrated by the blue curve and the dotted line of the fourth panel in Figure 4, the average length of the responses from the model trained using on-policy DPO is much larger than the SFT baseline and RLHF counterparts. With the increase of the win rate against SFT', the average response length also keeps increasing. To restrain this bias, we impose Hanon by adding a sentence like "you are a laconic agent and prefer concise answers" to the annotator LLM, just like how we manipulate the spurious correlation between screen and off in Section 5. Then, combining with the existing interaction phase that requires h E Hhelptull this design is equivalently imposing constraint of h E Haelpful n Hhhott Hence as illustrated by the orange curves in the last two panels, the on-policy DPO can then generate shorter responses (the increasing speed is also restrained) while keeping a high level of helpfulness. However, if our constraints of the length are too strong, which makes Haelptul n Mveryshort = D, the model's helpfulness will then be significantly harmed, as demonstrated by the pink curves in these two panels.
In summary, we find all our analysis on the Bayesian- still holds for a practical preference-tuning task: the biases would be amplified and a suitable interaction phase can control it as long as we can figure out them. However, some biases are inevitably hidden and are also amplified during LLM's evolution. Hence how to pinpoint these biases, or finding method that can restrain malicious biases even without explicitly knowing them, would be interesting directions to explore in the future.
Conclusion
This paper examines the potential and ongoing evolutions of LLM agents by drawing parallels with human cultural evolution, where the latter is a well-established subject in cognitive science. By demonstrating that the sampling and learning procedures of LLMs in various algorithms can be effectively approximated by Bayesian inference, we successfully apply the Bayesian-IL framework to elucidate and steer the evolution of LLM agents. The presented theory and accompanying experiments not only provide deeper insights into LLM behavior from a top-down perspective but also hold the potential to inspire the design of more efficient self-evolution algorithms.
Acknowledgments and Disclosure of Funding
This work was supported in part by the Natural Sciences and Engineering Resource Council of Canada, the Fonds de Recherche du Québec Nature et technologies (under grant ALLRRP57708- 2022), the Canada CIFAR AI Chairs program, the BC DRI Group, Calcul Quéhec, Compute Ontario, and the Digital Resource Alliance of Canada
References
Walter R Gilks, Sylvia Richardson, and David Spiegelhalter (1995). Markov chain Monte Carlo in practice. CRC Press.
Søren Feodor Nielsen (2000). "The stochastic EM algorithm: estimation and asymptotic results."
Bernoulli, pages 457-489.
Aad W Van der Vaart (2000). Asymptotic statistics. Volume 3. CCambridge university press. Simon Kirby, Mike Dowman, and Thomas L Griffiths (2007). "Innateness and culture in the evolution of language." Proceedings of the National Acudemy of Sciences 104. 12, pages 5241-5245.
Mark Davies (2008). "The Corpus of Contemporary American English." URL: www eng ish
corpora. org/ coca/.
"We use GPT4 to compare the helpfulness between responses generated by different models, as did Rafailov et al. (2024).
10

---

Thomas L Griffiths, Brian R Christian, and Michael L Kalish (2008). "Using category structures to test iterated learning as a method for identifying inductive biases." Cognitive Science 32.1,
pages 68-107.
Simon Kirby, Hannah Cornish, and Kenny Smith (2008). "Cumulative cultural evolution in the laboratory: An experimental approach to the origins of structure in human language." PNAS 105.31,
pages 10681-11800 100666.
David Lewis (2008). Convention: A philosophical study. John Wiley & Sons.
Aaron Beppu and Thomas Griffiths (2009). "Iterated learning and the cultural ratchet." Proceedings
of the Annual Meeting of the Cognitive Science Society. Volume 31.
Florencia Reali and Thomas L Griffiths (2009). "The evolution of frequency distributions: Relating regularization inductive biases through iterated learning." Cognition 1.3.3 pages 317-328.
Nicolas Fay, Simon Garrod, Leo Roberts, and Nik Swoboda (2010). "The interactive evolution of human communication systems." Cognitive science 34.3, pages 351-386.
Amy Perfors, Joshua B Tenenbaum, Thomas I. Griffiths, and Fei Xu (2011). "A tutorial introduction to Bayesian models of cognitive development." Cognition 120.3, pages 302-321.
Elizabeth Bonawitz, Stephanie Denison, Thomas L Griffiths, and Alison Gopnik (2014). "Probabilistic models, learning algorithms, and response variability: sampling in cognitive development." Trends in cognitive sciences 18. 10, pages 497-500.
Vanessa Ferdinand, Simon Kirby, and Kenny Smith (2014). "Regularization in language evolution: On the joint contribution of domain sppecifc biases and domain genneall frequency learning." Evolution of Language: Proceedings of the 10th International Conference (EVOLANG10). World Scientific. pages 435-436.
Simon Kirby, Monica Tamariz, Hannah Cornish, and Kenny Smith (2015). "Compression and communication in the cultural evolution of linguistic structure." Cognition 141, pages 80- Danielle J Navarro, Andrew Perfors, Arthur Kary, Scott D Brown, and Chris Donkin (2018). "When extremists win: Cultural transmission via iterated learning when populations are heterogeneous." Cognitive Science 42.7, pages 2108-2149.
Vanessa Ferdinand, Simon Kirby, and Kenny Smith (2019). "The cognitive roots of regularization in language." " Cognition 184, pages 53-68.
Shangmin Guo, Yi Ren, Serhii Havrylov, Stella Frank, Ivan Titov, and Kenny Smith (2019). "The emergence of compositional languages for numeric concepts through iterated learning in neural agents." The 3rd workshop on Emergent Communication, NeurlPS.
Yasamin Motamedi, Marieke Schouwstra, Kenny Smith, Jennifer Culbertson, and Simon Kirby (2019). "Evolving artificial sign languages in the lab: From improvised gesture to systematic sign." Cognition 192. page 103964.
Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin (2020). "Countering language drift with seeded iterated learning." " International Conference on Machine Learning.
PMLR, pages 6437-6447.
Hossein Mobahi, Mchrdad Farajtabar, and Peter Bartlett (2020). "Self-distillation amplifies regular- ization in Hilbert space." Advances in Neural Information Processing Systems 33, pages 3351- 3361.
Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, and Simon Kirby (2020). "Compositional languages emerge in a neural iterated learning model." International Conference on Learning Representations.
Ankit Vani, Max Schwarzer, Yuchen Lu, Eeshan Dhekane, and Aaron Courville (2021). "Iterated learning for emergent systematicity in vqa." International Conference on Learning Representations. Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu (2021). "Acre: Abstract causal reasoning beyond covariation." " Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10643-10653.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Ammanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. (2022). "Training a helpful and harmless assistant with reinforcement learning from human feedback." arXiv: 2204. 05862.
Yasamin Motamedi, Lucie Wolters, Danielle Naegeli, Simon Kirby, and Marieke Schouwstra (2022).
"From improvisation to learning: How naturalness and systematicity shape language evolution." Cognition 228. page 105206.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. (2022). "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems 35, pages 27730-27744.
11

---

Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi (2022). "Selffinstruct: Aligning language model with self generated instructions." arXiv: 2212.10560.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma (2022). "An Explanation of In-context Learning as Implicit Bayesian Inference." International Conference on Learning Representations.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. (2023). "Weak-to-strong generalization: Eliciting strong capabilities with weak supervision." arXiv: 2312. 2312.09390. Emil Carlsson, Devdatt Dubhashi, and Terry Regier (2023). "Iterated learning and communication jointly explain efficient color naming systems." arXiv: 2305 10154.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and
Tong Zhang (2023). "Raft: Reward ranked finetuning for generative foundation model alignment."
arXiv: 2304. 06767.
Thomas L. Griffiths, Jian-Qiao Zhu, Erin Grant, and R Thomas McCoy (2023). "Bayes in the age of intelligent machines," arXiv: 2311.10206.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. (2023). "Reinforced self-training (rest) for language modeling." arXiv: 2308.08998.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan. Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. et al. (2023). "Self-refine: Iterative refinement with self-feedback." arXiv: 2303. 2303.17651.
Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas Griffiths (2023). "Embers of autoregression: Understanding large language models through the problem they are trained to solve." " arXiv: 2309. 13638.
Yi Ren, Samuel Lavoie, Mikhail Galkin, Danica J. Sutherland, and Aaron Courville (2023). "Improv- ing Systematic Generalization using Iterated Learning and Simplicial Embeddings." " Thirty-seventh Conference on Neural Information Processing Systems.
Ilia Shumailoov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal. Nicolas Papernot, and Ross Anderson (2023). "Model Dementia: Generated Data Makes Models Forget." arXiv: 2305. 17493. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. (2023). "Llama 2: Open foundation and fine-tuned chat models." arXiv: 2307.09288 .02288.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim (2023). "Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks." " arXiv: 020777.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang (2023). "Iterative preference learning from human feedback: Bridging theory and practice for rhhf under kl-constraint." " ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models.
Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang (2023). "Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf." arXiv: 2312. 11456.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley (2023). "Baize: An open-source chat model with parameter-efficien tuning on self-chat data." " arv: 2304.01196.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu (2024). "Self-play fine- tuning converts weak language models to strong language models." arXiv: 2401.01333
Zheng Chenbao, Zhang Jieyu, Kembhavi Aniruddha, and Krishna Ranjay (2024). "Iterated Learning
Improves Compositionality in Large LaniunnLanguage Models." " arXiv: 2402. 047922.
Yann Dubois. Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto (2024). "Length-controlled alpacaeval: A simple way to debias automatic evaluators." arXiv: 2404. 04475.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. (2024). "Direct language model alignment from online ai feedback." arxiv: 2402. 04792.
Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, et al. (2024). "Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement." " International Conference on Learning Representations.
12

---

Rafael Rafailov, Archit Sharma. Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn (2024). "Direct preference optimization: Your language model secretly a reward model." Advances in Neural Information Processing Systems 36.
Yi Ren and Danica Sutherland (2024). "Learning Dynamies of L.M Finetuning." " arXiv: 2407. 10490.
Yuan Weizhe, Pang Richard Yuanzhe, Cho Kyunghyun, Sukhbaatar Sainbayar, Xu Jing, and Weston Jason (2024). Self-Reeardiig Language Models." arXiv preprint 2XXi:2441111022
Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang (2024).
"Perils of celf-Feedback: Self-Bias Amplifies in Large Language Models." arXiv: 2402. 11436.
13

---

A Discussions of the Proposed Theory and its Applicability to Real Methods
A.1 Assumptions of Bayesian-IL. and practical Scenarios
As typical in theoretical machine learning research, some assumptions needed to prove results about models' behavior; these assumptions are often not exactly satisfied by practical algorithms. So, we elaborate here on the important assumptions we made and when practical algorithms break them.
Assumptions for the theoretical analysis. To derive the guarantees of Proposition we first model the interaction phase as a binary filter on la E Helf and also assume a shared prior (h) among all agents involved in Bayesian-IL. We also model the L.L''s in-context behavior as a Bayesian agent and assume the number of samples during the imitation phase is sufficient.
Assumptions we can break for iterative ICL experiments.
Binary filter on h E Hat- All our LLM experiments break this assumption (the pure Bayesian example in Appendix C.2 does not). For example, in the ACRE experiments, we use self-refine and hypothesis search the interaction phase. Self-refine asks the model to evaluate the responses, and the hypothesis search uses an external interpreter: they both manipulate h by feeding messages the LLM, rather than binary filter. (When using an external interpreter, h is usually filtered before forming the refinement feedback.) For the experiments in Section 6 and 7, where h is implicit, re-rank all the generated samples in Pooo and take a weighted sample during imitation, similar to re-ranking the generated examples in ReST. Since all these interaction designs are commonly applied in the community, and our theory describes their qualitative behaviors well despite strictly violating the assumption, we believe our methods can shed more light on other practical methods with similar designs, like self-reward (Weizhe et al. 2024), iterative-DPO (Xiong, Dong, Ye, Z. Wang, al. 2023), etc.
Identical Po(h) for agents in different generations. Although this assumption makes easier to derive Proposition 1, slightly relaxing it will not change the whole story: we only require different agents to share a similar tendency towards a specific bias. To verify this, we conduct several experiments when the agents in different generations are different LLMs (e.g., GPT3. 55 plays with Claude3 Appendix and E). The phenomena claimed by the theory still hold.
The Bayesian learning assumption, i.e., Pmm(h) (h) = P(h d) x p(d (M). Although this assumption is necessary for drawing a parallel between iterated learning and the EM algorithm and hence getting a guarantee for the amplified bias, the practical in-weights learning (IWL for short) method usually does not strictly follow this assumption, because people usually early stop the training before the model perfectly learns all d'. However, results in Section 7 match our analysis well, which means the iterated IWL can also be depicted by iterated learning of finetuning methods with different targets or loss functions, their aims are consistent: increasing the likelihood of pdrrain h) under instructions, which aligns with Bayesian targets well. Furthermore, we find the increased bias or decreased creativity during iterated finetuning has also been extensively mentioned in many related works (Touvron al. 2023; W. Xu et al. 2024), which also supports our analysis.
some extent. That is because although there are plenty
A.2 Why we Start from Two "Toyish" Tasks
The experimental settings in Section and 6 are relatively manual and toyish. The main reason for us to start from them is that we want to directly observe some quantitive numbers described by the theory, which we believe would provide stronger support for the analysis. For the explicit case, we chose the ACRE task because of its simple H. making it possible to observe the distribution and entropy of all possible h. We believe observing the model's 's logits supports our theory more directly than merely observing the accuracy other quantitative metrics.
For the implicit case, we chose the acronym task, which a prototype of celf-datt-auggmenttioon in self-instruct. We initially consider the conditional creative writing task (quite common in many related works), where the model needs to write a passage (i.e., the list in our settings) based on several topic words (i.c., the acronym). However, constrained by the context length of LLMs,
14

---

we can't generate more than 4 examples in one generation, which makes it hard to calculate the statistics of d'. Remember the model will generate 20 extra d' based on 20 d' in our acronym experiments. In summary, although the experiments studied in our paper look artificial, they are reasonable approximations of real tasks.
Last, in concurrent work W. Xu et al. (2024), the authors study practical applications like machine translation, creative writing, math reasoning, etc, in an iterative ICL setting. Their observations match our theoretical analysis quite well: bias is amplified generation by generation, and introducing external feedback can mitigate it. However, due to the complexity of the tasks they considered, they can only observe the average bias and the skew level using several conclusive quantitative metrics. Hence we believe that by combining our theoretical analysis, detailed observations on artificial examples, and the evidence from real applications in W. Xu et al. (2024), one can draw a good overview of how LLM would evolve in an iterated ICL setting.
A.3 How our analysis brings benefits to practical algorithms
Besides the method proposed in Section 7, where we manipulate the instructions prompt of the annotator LLM during the interaction phase, experiments and analysis also provide the following potential approaches to guide the model's evolution:
Select d that makes more confident and correct predictions on the target task. Manually selecting good in-context examples is intuitive. Our analysis, though, suggests taking the model's confidence (i.e., the logits) into account, because the theory claims that the likelihood and bias in prior are competing with each other during evolution. From Figure 3, we see the model evolves faster if the likelihood of d hh" is weaker. The results in Figure also provide similar insights: the related biases are amplified slower when the number of easy samples in d decreases.
Designing a good interaction phase is important: more accurate Herf leads to better per- formance. This can be supported by comparing self-refine and hypothesis-search in our paper. The paper W. Xu et al. (2024) also claims that external feedback with more accurate assessments or feedback from a larger model can reduce the amplified bias.
Manipulating the instruction prompt: in our analysis, both (h) and Pmu (h) are the model's predictions conditioned on the instruction prompt w. Hence adding preference in the task instruction (or changing the system prompt) during evolution could be an effective way of guiding the model's evolution. Our ACRE experiments show the feasibility of this: remember we can introduce spurious correlation by adding one sentence to the instruction. Hence it is also possible guide the model's evolution by feeding appropriate prompts during learning and sampling.
Manipulating the temperature: Bayesian-IL theory studies the evolution of the distribution, the temperature should also be an important factor for the evolution, as illustrated in Figure 3. We left the exploration between temperature and different phases in IL in the future.
B Proofs related to Bayesian Agents
B.1 Recap the Proof Expectation-Maxiinzatioo Algorithm
To get clear picture of the asymptotic behavior of imitation-only iterated learning, we first recap how typical inpectation-mmximizatii (EM) algorithm converges when the target function is posterior e distribution'. Consider a statistical model that generates a set of observable data samples (x, 1m and the corresponding hidden variables {z, ym The generating mechanism can be expressed as P(x z, 0), where is a set of unknown parameters determining this distribution. To get a
'The MLE (maximum likelihood estimation) version of the EM algorithm is common in textbooks.
15

---

o7 P(4))),0-11)
argmax B. Ex-oi oi [log (x. a/10)/g""" (+)
Ot-1
eC-1
0
Q
EM:
E-Step
M-Step
a argmax Re-inint 1y flog P(d)n)))) (" 00)
4f - P(dl
ht-1
d
h
dt
IL:
Hypotheup
data
Hypothesis
data
learn
A-entt-1
Agent
Agent
Figure 5: Illustrations of typical EM algorithm and an imitation-only iterated learning method.
MAP(maximum a posterior) estimation 0, we need to optimize the following target function:
(()
log P(O x .m))
log Po(0) + log P(xi...... Xm
() log P(x1...... Xm)
m
log Po(
Σ i=1
log P(x: llog P(xi....... Xm)
m
log ((x: 0)P² (0)) loog Xm)
where (0) the prior distribution of parameters. As the marginal distribution P(x; 0) hard to calculate due to the existence of the hidden variable our target function can then be expressed as
É(() = L()) const
log Σ P(x:, 1x (0)
where the term log P x is eliminated doesn't 't depend on The target function above is still hard to tackle due to the summation inside the logarithmic function. To solve this, we first introduce an auxiliary funetion Q,(z;), which is probability distribution over 2,, and reformulate the target
((0)
m72
log
[.((Z))
P(x; PO (0)
Σ
m
log
BRiNQ,
P(x;, 2, O) (0)
Q1(z,)
Q.(z,)
(6)
Given the concavity of the logarithmic function, we can use Jensen's inequality to get a lower bound of (()):
m
P(x;, z; 1)) (0)
E(0) 2 J(0
Σ E E
log
Qi(z,)
The FM algorithm then maximizes this lower bound by alternatively optimizing 0 and Q, for several rounds.
In the E-step, as illustrated in Figure 5, we use the estimated 01-1 in the previous round to find Q: argmax (6t-1, Q). Specifically, we need to push this lower bound to be tight by making the equality in Equation (7) hold. Following the properties of Jensen's 's inequality, the equality only holds
when 7 1p. (0) is constant. Combining this requirement and the fact that Σ. Qi(z) = 1, (2(z.)
we can calculate the optimal (z, = P(z; x; 0(-1,, which the posterior distribution of z; given the observable data x; and the fixed parameters -r-i
16

---

In the M-step, we plug in the estimated Q to [ Q) to get the target function as:
I(0:Q" = Σ E2 Qz
log
P(x:,
Zi P (0)
Σ RR Q [log
(((x Zi 10) (0)]
Qi
(8)
where c = Q; (Qi] is a constant term and can be neglected while optimizing 0. In this step, we can calculate 6 = argmax, [(; Q") using gradient descent or other parameter estimation methods.
In summary, the E-step ensures tight lower bound [(0,Q] Q) and the M-step finds better 0 to make it larger. The two steps cooperate to ensure a series of estimations of 0 for which (()) is non-decreasing. Finally, the estimation of parameters will converge the one that maximizes the posterior distribution, i.c., E((") = argmax, P(0 xi..... ), if it is convex.
B.2 Proof: Convergence Behavior of Bayesian Agents in Iterated Learning
Proposition 1. Consider several Bayesian agents sharing the same prior Po(h) are conducting iterated learning for T generations. If'T is sufficiently large, any agent, with T will have
(h) 1(h 1hh= = h²*)
where T is a stationary point (e.g. a local maximum) of Po(h) subject h E Heg.
Proof. The proof of this proposition can be divided into two steps. In the first step, we show that imitation-only iterated learning shares similar convergence behavior with standard EM algorithm. In the second step, we show the "selecting" pressure introduced via the interaction phase doesn't 't break the necessary conditions of the convergence in the first step. Merging these two steps leads to the proposition.
Step 1: imitation-only iterated learning special EM
Recall the imitation-only iterated learning illustrated in the bottom part in Figure 5, where the hypothesis held by the agent in the (t - 1)-th generation is represented by /t-1 With this hypothesis, the agent will generate data samples using P(d h²-1), denoted d [d ..... d. In the 1-th generation,
agent will first update its posterior probability using P(hld' ) and then select h' by picking the one with the largest posterior. As there are multiple data samples in d' this process can be expressed as
ht argmax log P(h di- d1-1 1
h
p(ap-1 dh ...., 1 Po)))
argmax log
P(d ..b ..... dmm
argmax log Po(h) II p(d (1)
h
[=1
1 m log Po(h) + m
log p(d -1 (t)
argmax
m
s argmax
E d Dog Po"i (h)) +
[logp( p(d] [)]
h
d=
argmax h d-pidht-1
E
log h) h)P (h)].
(10)
Based the analysis above, notice that the imitation-only iterated learning and the EM algorithm are almost identical: by replacing 0 and z to and d, and by removing the random variable x", we can also have similar theoretical guarantees for the imitation-only iterated learning algorithm.
To prove this, we can first verify the equivalence between the imitation phase and an M-step. By comparing the target functions when calculating hidden variables (h and 6) in these two algorithms,
'It is unusual to apply EM with no observable data, but removing it doesn't 't violate any assumptions in the derivation of EM.
17

---

i... Equation (8) and (10), we can find two major differences. First, the expectation of observable samples (i.e., Ex,) disappears in Equation (10), as we assume there are no "observations" in iterated learning. We can also introduce dummy variable named x to the iterated learning process, and find that the existence of x doesn't 't influence the aforementioned caleulation at all, Second, in II., we can only approximate B [1] by sampling d from p(d h N'=1), while in EM, the posterior distribution P(z: x,
x, 0*) is usually analytically calculated. Of this discrepancy. if our d is a good approximation
of p(d h ). the imitation phase in IL is equivalent to M-step in EM.
We then verify whether the transmission phase is good approximation of an E-step. The first thing to check is the tightness of the lower bound generated via Jensen's 's inequality, which guarantees the non-decreasing update the target function across multiple generations. we can first assume the target function of the whole iterated learning process is [(h) = log (h)), and derives its lower bound J (h; Q) following a similar procedure in EM:
(h) = log Po(h)
m log P (h)
= m log Σ p(d. PM" (h)
m log Σ Qi(ss) p(d; Ph (h)
(:)d,)
p(d: Po" (h)
m log Edi
(i(d,)
p(d: h))P"" (h)
2 Ed:-Q log
a (h,
(1I)
Qi(de)
The equality of Jensen's inequality holds when p(d:l)) (h) (d.)
is a constant. Using the fact that Ed, (((c) = 1. we can have the optimal Q; p(d: h). In summary, as we sample each d. p(dd 1²11), the transmission phase in IL equivalent an E-step in EM.
Another interesting parameter to discuss is m, i.e., the number of data samples generated by an agent in each generation. The choice of determines how well the sampled d can represent the ground truth p(d h). For large enough m, we can prove the convergence guarantee using the above procedure. When et al. 1995. The authors of Nielsen (2000) proved that in stochastic EM,, A in different generations form homogeneous Markov chain whose stationary distribution over hypotheses is approximately centered on the maximum-likelihon solution. In other words, when f > T for sufficiently large xm)]. ]. and similarly, IE[4"] is the optimizer of Po)h). In other words, the dominating hypothesis in imitation-only iterated learning converges to the one with the highest prior, which is equivalent the large case.
1, the standard EM algorithm becomes stochastic EM approximation Gilks
T. [[] optimizes [P() xi..... xm
Although m = doesn't 't influence the converged estimation of h, a too small m will make the variance of estimation large, and hence impede the converging speed of h. Then should choose m as large as possible? The answer still no: too large m will also impede the converging speed. We can get some intuition by observing Equation (10), where p//m(h) (h) determines the effect of the prior when selecting optimal and the preference encoded in the prior cannot influence the choice of h in this generation much the likelihood term ((d] h) will dominate." Hence the resulting h² would be quite close to h--1, which means the evolution of belief on h would be slow.
each generation. m is too large, this distribution would be very flat
Actually, the choice of m is usually considered as the "bottleneck" parameter in different iterated learning algorithms. Almost all the related studies point out that the bottleneck should not be too wide
"A similar trend also exists in Bernstein- von Mises theorem (see e.g. Van der Vaart 2000), which claims the posterior p(O) æ1,..., 2 ) = A/(0, n -=1(6))-) for n + cc. In other words, P(b]d dra.) would become peakier m increases, and hence / will be closer A after the imitation phase.
18

---

or too tight (like experiments in Kirby et al. (2015) and Ren et al. (2020)). Our Bayesian analysis provides a theoretical explanation of the effect.
Step 2: the influence of introducing Hem- Let us first check the imitation phase, i.e., the E-step where h is calculated in Equation (9). Assume we have perfect interaction phase that can rule our all h & Hetfs then the target function is:
h' argmaxlog P(h| di d2 ,...,, d
(12)
AEHai
argmax log P(h| | di d22 ...., d 1(h E Hen)
log [h) (Px (h) 1(h E Hai)]
argmax
E dop(d(he-1)
logp(d h) Po (hi]
argmax
E dmp(d[hh-1)
where we define a "regularized" prior as (h) c- Po(h) (1 E Har). Then, by substituting this prior back [(h) defined in Equation (11), we find the optimal Q7 = p(d; h) still holds. In other words, as long proposition still hold. Furthermore. this proof also provides us an insight that adding constraints on Heff is required in both imitation and transmission phases. Hence having a powerful "data-filter" or "data-ranking" design for the transmission phase would also make the evolution more robust, like those applied RAFT (Dong et al. 2023) and ReST (Gulcehre al. 2023).
same d ~ p(d] 1' where h Hetf. all the conditions required for this
B.3 Proof: LLM Bayesian Agent
Proposition Consider that agent A is conducting in-context learning. If the prompt examples in d' are generated by another agent B with the same prior knowledge (e.g., they come from the same checkpoint and use the same prompt), sampling from the posterior predictive distribution of agent A, ie., d' ~ (d ) can be decomposed into: 1.) ht + argmax, Pane d'-1), and 2.) d' (d ) where h is hidden variable that describes the mapping between x and y.
The proof of this proposition can be divided into two steps. In the first part, we de- de-marginalize the posterior predictive distribution hidden variable h, and then show that the model automatically "selects" a hypothesis = that generates the prompting examples. In the second part, we show that when the examples in d are generated by another LLM with the same prior belief over h, the MAP (maximize posterior) estimation of h can approximate h well.
Proof.
Step 1: In our paper, we assume the query and answer sequences in each example, i.e., d; (I,, ys). are controlled by the hidden hypothesis h, which plays similar role to the "concept" parameter 0 mentioned in Xie et al. (2022)3. Then the posterior predictive distribution can be decomposed as:
/ p(d d'd'd=- d'=1 P() d dh
P(d P(d|d'
p(d h)P d dh
h pld (d) (h)P((h) dh
h p(d h) p(d' h)
p(d'-11'')) Po(h) dh
pld h))xp(n r. (b)) dh,
(13)
PA) where ra (h) A 1 log p In Equation (13), the second line follows the Markov property of hypothesis and data samples, the third line follows the Bayesian rule and drops constant term, the
"A we only need the "selection" mechanism mentioned in this paper, the HMM (Hidden Markov Model) assumption is not required in our setting.
19

---

fourth line is generated by dividing constant p(d" | },*,,, where h is a hypothesis that generates d Now we see the r., (h) has almost the same form as (0) in Xie et al. (2022). By reusing the derivation in that paper (mainly in Section 3.2 and the proof of its Theorem 1), we can conclude that
exp(n - (h)) - O for any h & h²* and exp(n (n(h")) - 1. Hence Equation (13) becomes:
P(d| d d== P(d h²)).
(14)
a general in-context learning setting, the prompting examples d' are usually sampled from an unknown distribution P prompp: For example, by analyzing different Tust. the researchers can manually design effective prompting examples sharing similar chain-of-thought structures with the target question. Obviously, such I' prompp is impossible to parameterize analyze accurately. To approximate it, Xie et al. (2022) first assumes that both the prompting examples and the pre- training corpus are natural languages, and
Step 2:
LLM can approximate this "natural language distribution" well. Then, Prompt can be well approximated by P(- (8)) under some 8", which is the prompt concept in that paper.
In our settings, as we assume the prompting examples d' are generated by another agent-B sharing the same prior, then the h* triggered by feeding d' to agent-A would be exactly the same as that feeding that to agent-B, i.e.. h t argmax PegmaxPPmmlll d dd
(15)
h
where Poow is model's belief after receiving the common instruction w.
Combining these two steps, we can decompose the sampling procedure d' ~ P(d | d' into two parts: first, inherently select h based on observations generated by another agent in the previous generation; then sample new d conditioned on this h"* which matches the Bayesian-IL procedure discussed in this paper.
C Experiments of Iterated Learning on Different Domains
To provide a panorama of iterated learning and its applications in different fields, this appendix will first give an intuitive explanation using some lab experimental results. Then, experiments on the emergence of compositional language among Bayesian agents are introduced to verify all the theoretical hypotheses.
C.1 Iterated Learning in Lab Experiments using Lewis Language Game
Aggrstio usddi teilk expairene
Irarrctioo of ass- Lenci Lar guige Gurnec
Langs age in the fins generstion
Propure ispor images and available vocsbulary Ty-
condidites, call them Alisetl Bohitl festonon Plone
If 5 then
Abodi komns from the datect Da..
end It
Initaaton phate taachhngg
fetemattion Plase
requiring unerrhigunas Come with Bolkil Alodil and Bulli) swithh the role and play the
O'to for
finm Alice.]
Largls ape The 1Dd gurandonn
Abrolel pla the Lewic
end for
Tranomisioon Plase
Alioitl generats the dataent D3
end
Reanil wwwarbinneeed intial aflae
AliosTT... T
images
Figure 6: The lab experiments (algorithm, settings, and results) conducted in Kirby et al. 2015.
As denoted in Kirby al. 2008, iterated learning is process where one individual learns by observing learning
the output of another individual, who learned in the same way. In this procedure, the shared language (i.e., h) among learning agents will gradually become more systematic under the compressibility pressure (imposed during imitation, embodied in Po and expressivity pressure (imposed during interaction, requiring h E Hen)) To simulate this process, authors of Kirby et al. 2015 design a two-phases learning procedure illustrated in Figure 6. In the interaction phase,
20

---

the speaker (Alice) and listener (Bob) must cooperate to accomplish a Leiws referential game (see the middle panel in Figure 6). Specifically, Alice will first create name for the given objects and talk that to Bob. After receiving this message, Bob needs to select the correct object shown to Alice among some candidates. If correct, both of them are rewarded. This phase terminates when they can achieve high enough success rate. To succeed in this game, the shared language should be expressive enough to avoid any ambiguities - we expect the language to be a bijection. Then, we select another new naive candidate (Alice[t+I]) and let it learn the naming system created by Alice[t] and Bob[t]. In this phase, those highly structural mappings should be easier for a human to remember, which is how the compressibility pressure is imposed. After that, Alice[t+ 1] will play the same game with another Bob[t+1] and the interaction phase starts again. We provide the languages generated by Alice[O] and Alice[10] in the right panel in Figure 6: it is clear that an interesting structure emerges in the language generated by Alice[10]. There are also plenty of similar lab experiments that support the "two pressures" and cultural evolution hypothesis using II.-like training methods, e.g., Fay et al. 2010; Ferdinand et al. 2019; Motamedi et al. 2019; Motamedi et al. 2022. Although these methods have different types of input, game designs, learning procedures, vocabularies (like gesture language), etc., the conclusion of them is quite consistent: compressibility and expressivity pressures are crucial for the emergence of systematic mappings, iteratively learning and interacting can amplify these pressures a lot, which matches the Bayesian explanations well.
C.2 Iterated Learning of Bayesian Agents (re-implementation of results in Kirby et al. (2015))
To verify that the emergence of systematic mappings in iterated learning is not an accident, authors of Beppu and T. Griffiths 2009 provide guarantee by analyzing the behavior of Bayesian agents. There are also plenty of related works in cognitive science, like Reali and T. L.. Griffiths 2009; Perfors et al. 2011; Bonawitz et al. 2014; Ferdinand et al. 2014; Navarro al. 2018, discussing the influence of and theories behind iterated learning and Bayesian analysis.
To give the readers a better understanding of how iterated learning works, we re-implement the Bayesian experiments mentioned in Kirby et al. 2015. Consider the following toy example, where we have four different input objects: X' {blue circle, blue box, red circle, red box}. and four possible names: ) = {00, 01, 10, 10, 10,11]. The hypothesis h is defined as h E H X' + V In this example, we have [H] 256, which means P(h) can be parameterized by categorical distribution with 256 dimensions. In this analysis, we assume the prior distribution of mapping negatively correlated with its coding length a, i.e., P(h; a, e) xx 2- where is a normalizing constant to make sure the prior distribution is not too peaky. Usually, the easier-to-leam mappings (i.e., more systematical ones) have higher prior. In Table 3, we demonstrate how to calculate the coding length for the three typical mappings. Note that the mapping that has the highest prior is a degenerate mapping, where a = 18 and Phh) & 0.6. The (h) for all possible mappings are demonstrated in Figure 7.
Table 3: An example of coding the mappings, where is how many characters (including space and unique symbol. e.g., - and :) are used to express the grammar. systematic mapping
A holistic mapping
A degenerate mapping
a 43
a 56
a 18
z2, zl
72: 0
-
blue
S:
00 blue circle
72:
1
-> red
S:
01 red circle
S: 00- - Everything
zl:
0
-
circle
S: 10 red box
z1:
1
+ box S: 11 + blue box
In this experiment, the knowledge of an agent is encoded in its posterior distribution, i.e., Pim (h). We will observe how this distribution evolves when the agents conduct iterated learning. Recall the sampling behaviors discussed in Section 3. To get data sample d (z, v). we first randomly sample x E X from a uniform distribution and then sample y based on the given x,
d~ (mm((x))) Pomly x) x p(y h, Pi (h).
(16)
21

---

197
degenerate
holistic
others
systematic
10
Prior
10:14
150
The Index of 256 mappings
Figure 7: The prior probability of all possible ht E H. The systematic mappings are sandwiched between degenerate and holistic mappings, which means Po Po(heegn > Po(heys) > Po Some mappings in the "other" group also have relatively large prior, because they contain degenerate components (e.g., mapping two or three objects to the same message).
The likelihood pt /, x)) is defined as:
1
if y is mapped to a
ply h, x)
(17)
otherwise,
D
where c is small positive value describing the systematic error during communication. For the learning behavior, the agent will update the posterior based on the received data samples
N
Pom(h) = P(h d) x p(d h) Po(h) x Po(h)
II
p(y: h, x4).
(18)
Now, with the definition of learning and sampling for these Bayesian agents, we can describe how they conduct IL:
Initialization: at the beginning of the t-th generation, new agent is initialized by Po
Imitation: agent-f learss from d which is generated by agent in the previous generation following Equation (18)
Interaction: to impose expressivity pressure, let agent-t (Alice) play communication game in this phase. Specifically, we first create another agent Bob by copying Pim (h) from Alice. Then, Alice samples data pair d = (ar, y) on a randomly chosen x and sends it to Bob. Bob will estimate the object based y. If the estimated x' x, the game succeeds and data pair (æ, y) is added to buffer named deomm After several rounds, Alice updates its knowledge by learning from deomm- Note that in this phase, the pressure of h E Helf is induced implicitly: for the ambiguous h, where multiple x are mapped to the same y. Bob's 's reconstruction x' might not equal with high probability. Hence deomm will finally dominated by the samples generated by those h E Hett-
Transmission: after the interaction phase, Alice will generate multiple samples d' for the next generation.
In the above procedure, the compressibility pressure is embodied the prior distribution where more reusable principles lead to higher probability, which aligns with the simplicity bias in the human cognition system. The expressivity pressure is imposed in the communication game because d only contains the unambiguous mappings. Under this setting, proportion (i.e., the summation the posteriors) of different types of languages and observe how they evolve during iterated learning, as illustrated in Figure 8. It clear that the systematic mappings gradually dominate as the learning progresses.
calculate the weighted
To further verify our theory, we consider compressibility--nt case by removing the interaction phase, and an expressivity-only case using a uniform prior P(I) 1/256. The results in Figure match the theory quite well:
22

---

Degenerate
expersovity
Proportenn
holletic systematic
Gareraiions
Generations
Generations
Generations
Figure 8: Ratio of three different types of mappings during iterated learning (curves are the average of different runs, shadow region is the variance). Left to right: 1.) d is a holistic mapping: 2.) d a degenerate mapping: 3.) Starting from a holistic d no longer conduct an interaction phase during training. Hence the degenerate language, which has the highest prior, will gradually dominate; 4.) Ablating the compressibility pressure by using a uniform prior distribution.
Generation
prob-1.0
degenerate
Posterior
holistic
others
systematic
Gemeration
Posterior probabiitt
Posteriar probabiltty
256 mappings
Figure 9: The posterior probabilities of at the end different generations.
In an imitation-only iterated learning case, i.e., the third panel in Figure 8, hT converges to a degenerate mapping, which has the highest prior as illustrated in Figure 7;
Introducing the interaction phase will rule out those ambiguous mappings (i.e., those f & Hor), and hence h² converges to systematic mapping, which has the highest prior among all Hem:
By comparing the first and second panels in Figure 8, A² always converges to systematic mapping no matter d holistic or degenerate.
Furthermore, can directly observe the dynamics of (h) from Figure 9, which provides more detailed illustration how the posterior of all mappings changes during training. In the first generation, we see the dominant mapping is holistic one, which our d Then gradually, under the two pressures, the posterior of systematic mappings gradually increases and finally dominates.
23

---

D More on GPT-based ACRE Experiments
D.I How to Calculate the Model's Posterior on All Hypotheses (Figure 10)
Thanks to the instruction-oolloinng ability, the GPT can always provide responses following the given format, as illustrated in Figure 10. The experiments demonstrated in this part come from OpenAl's playground. The model we use is gpt-3. 100-335-turde 0.1 and the probability feedback is enabled. We let the model return probabilities of the top candidate tokens for each token in the response, as illustrated by the three sub- panels in the figure. Then, the posterior of specific can be calculated by multiplying the probability of all tokens with corresponding values. For example, Pi (h = on, B:und, C:off, D:... can be calculated by P(rs- = on)- - P(rg- = und). P(r13 = off) .... where 13... are the tokens denoting the corresponding values of A, B, C. To further show the feasibility of this approach, we conduct the following two verifications. First, we calculate II P(r.) and find that this value is always close to one (Tramat denotes the indexes of those format-related tokens, e.g., Rule, {, :, A, B, C, etc). This means the model reliably follows the instructions when generating responses. Second, we calculate Pimuo (l)) for all possible 243 different / and verified that EnEN Prmmn(1) is always close to one.
The temperature is
Let un play mirile-fe wollowoooiigg game. need generate
mape the following
iop the their conmepenniing outputi generite ecumplex Sallowing a given rule, Each eccample is an inpuut outpout pai.. The reput a lust otjecta. the presence of cantare otjects will trigger the light to furn on. The ourputt either 'on"" "off "undeternineet" ind cating
Inputic,
Outpution
und i StUTS
the light. For wach obiject, determitee whether trigpers only
Outputiorr
Inpatt
und - 1464
4152%
otjects game: 1C: 10,,
Outpution
010%
You new howe mnne encmmpies,, geoeaate nule that mape the sill thriliputs (nncurding
on C17%%
thoee presisue nounsti) to their comneporndin nulputt..
Please format
(skject
9960% pnbabiiity unverd rap Frogiaa
InputiE
BuNc (Acons R
Cren
Ampult C.D.E
Outputt on
Output on
Imputic, D
Output
(- OutG
outputtoff
Fput-C. D
C+ O14%
InpullA
Pput:AA
Doample- D.0%%
0.00%
Please formet your rute as rotrows return Rute: fotject con/erttnnd, 2t consrttundb... ..3
return the ruich
Total 000 log on thokens
Rullo (otject
Please form
Total: 0000 logprob on tolkenn
Decon und Cros. on.
(6con. LuddCcoon DI offff
Figure 10: How GPT provides the rule following the given format, which makes it possible to calculate (h) for all h.
D.2 How to Control the Bias in the Prior
We demonstrate how to manipulate bias in the prior (h) in the ACRE task by adding spurious correlations in the prompt and by changing the name of the object. The prompt we use is almost the same as that in Figure 10. We will analyze the value of P Prrzu), i.e., the probability of the token denoting the status of the last object in the rule. The d in these experiments
all the same (i.c., [B,D] -> on; [B,C] C] -> und; D,. -> on, as stated in the last several panels in Figure 11). Some subtle modifications under different settings will be described one by one in the following:
1: the default setting, where the object list is [on,off, und]) are [11.8% 6.46%, 81.68%). This makes sense, all these 3 statuses of E can describe all examples in d"
EJ. The Po(r2o =
D: compared with O, we change the object E to screen, and extra text is added to the prompt. Then, as the screen is likely to be turned on during the experiment, Po (r = on) dominates the prediction;
2: compared with D. add sentence "Turn off the screen after the experiment" to the task instruction. This misleading sentence introduces a bias towards screen: off by creating a spurious correlation;
3: compared with @, we use synonym "close the screen" to replace the "turn off the screen" in the prompt. As the word "off" does not exist the prompt, the bias towards screen: off is weakened;
24

---

D: compared with 2. we change the name screen to Sony screen in the example, but left the prompt unchanged. We see the model clever enough distinguish which screen we refer to, and hence keeps the preference of Po(220) demonstrated in Di;
: here we change the object to another name John, which also keeps the preference of (r20) demonstrated in ();
c: compared with 2 we add the sentence "John will turn off the screen after experiment".
Then we find the bias towards John off slightly increased, but not as strong as that in
2. which provides us another way to control the strength of the bias;
7: in the following three cases, we put the position of the misleading sentence before the examples dº. Compared with (2, the bias towards screen: off is significantly amplified. This might be because the attention mechanism lets the model recite the fact that the screen is off before reading the examples (remember that screen : off also explains all examples);
8: compared with ④ where the object is also Sony screen. Here the bias is stronger than ④ but weaker than O. which verifies that adding spurious correlation before examples can amplify the bias while modifying the object name can reduce the bias;
D: compared with 3, which also uses the synonym in the prompt, the bias becomes stronger.
In summary, we have several principles when controlling the strength of the bias in Po(h):
Adding spurious correlation before d provide very strong bias;
Using synonyms rather than phrases containing specific states (e.g., close/open v.s. turn off/on) weakens the bias;
Using two slightly different object names (i.e., Sony screen v.s. screen) weakens the bias;
Using indirect spurious correlation (c.g., John v.s. screen) weakens the bias.
stiject
Figure 11: An example of how to manipulate Po(h) by adding spurious correlation in the prompt. The task instruction is the same as that provided in Figure 10, except the object name (in the blue box) and the added hints (in the red box).
For the experiments in Figure 13 and Figure 3, the six different levels of prior bias are controlled by the following prompts:
Very high: add "Turn off the screen after the experiment. before and after d' is given;
High: add "Turn off the screen after the experiment." " before d';
Medium: add "Turn off the screen of the monitor after the experiment." before d';
Mild: add "John will turn off the screen after the experiment. " before d':
Low: add "Close the screen after the experiment." " before d';
" Very low: add "Close the screen after the experiment." affter d';
25

---

Piscreen off
Strong Stong
2.00g medium
Strong
2200ng n0R2
2rorg 2200g
Snogl lock
Piscreen off
1.3
modluu_stronn
medmumm
medium_rone
Piscreen off)
1.8
Weak_stron
#caCmmcolum
Weak_none
Plerrenn off)
Fanc_strong Fonc_medim
none_non:
noro_
Generation
Generation
Generation
Figure 12: A more detailed analysis of the competition between the likelihood p(d h)) and bias in (h)... The first 4 columns are results from the new GPT3. 5 and the last column is from GPT4. In each panel, the two curves represent strong and weak likelihood cases, which controlled by the ground truth h". The h" in strong cases contains 3 objects being "on" while the weak cases only have 1. The text in each panel represents the level of spurious correlation introduce before and after d by manipulating the instruction prompt. For example, strong_ stroonggstrogggggoo means we put strong bias, i.e., "Turn off the screen after experiments" before and after d' in each generation. The trend of these panels aligns well with our previous results and analysis.
Wey b.
High b.
Medium NiNs
Gonsiation
Gensration
Geteret on
Figure 13: The follow-up experiments of the one mentioned Figure 3. In this figure, we put curves of the same level of prior bias (but with different likelihoods) in the same panel. It is clear that in cases, the stronger likelihood will weaken the influence of the bias (that why the orange curve is above the blue curve).
D.3 The Prompt Design for ACRE Task (Figures 14 to 16)
Please also refer to the three figures and the log. txt file in our code base.
E More on Self-data-augmentatton Task (Table 4 and 5; Figure 17, 18, and 19)
In this appendix, we first introduce the prompt design of the experiment on both imitation-only settings, as well the experiments with five different Hefr. Please refer to Figure 17 for more details.
26

---

System:
owing same. You need gone are that maps the tollowing Inputs thair corresponding outputs, generate examples tol owing gven rule. Each evample is on input output pair. The input is let of ects. ann objects with status on in the inout, the output should be on, if all objects the inout are off, the out oft If thonlly abjects with off and und in the ist, the should be undetemined lund for short) The ourput is either ton", "oert or undete minoe. We only
us play
"und", Indicating state
light. each ablec, determine whether triggers light
dogar
have 5 objects this game: IN, BB, 'c, 'F].
Gen1 assistant:
Gen1 user:
Rule: "un'""" "a":'on" "C": "Und"" "o":"'n"" "on", "C":"on")
ht h tra (1))2))
You have more examples, generate a rule that maps inputs (including previous rounds) to their corresponding outpurs.
Gen1- user:
The rule you provided "und", "eet "on", "C'' "und", "t"; "on". "E": on"": Based on this rule, can you before? Ramember the input list objects.
with citterent inguts these objects with stacus on input, outout should be on. oojects input
Input:: A Output:ond
outout
only obyects with off and und in the ist, the outout should be undetermined lund for short).
Input: 8, Output:on on
d -p(dhh"
Gen1 - assistant:
Gen2 assistant:
Ingut: A,
You have more examples, generate rule the the fineluding those previous rounds) their corresponcing rg outputs.
Input:
Input: & D,F
Output:on on
Output:
Output: on
Input:A, B,C Output: and input:,,
Input: A, B, Output:on Input:
Please format your following format.
Rule: ("abjet it: "oo"""of""""nn "object <'onf yundf""
Output:
input:
Output und Input: A, B,
Outputt on
Output
d' d'~P Ptm (allie)
Output: on
Please fomat your rule in the following format.
Rule: I"object 1", ""on"ooor"""""ddd "osject 2"a"n" "ott"""""""""mnn /undd"
Figure 14: Prompt design and example dialogue for the imitation-only iterated learning on the ACRE dataset. The shaded region and arrows represent that we copy specific text to form the message. The messages starting with the role of system and user are sent to GPT, while those starting with assistant are the feedback from GPT. For this multi-round chat, we will feed all historical information API, see our code for more details.
Then, we provide the full results of the experiment in Table 4, 5 and Figure 18, from which we can derive more interesting findings. In general, the figure demonstrates how different metrics evolve for different generations while the table reports the converged values at the last generation.
From the first column in this figure, we observe that other than the Hhard setting, all curves in other settings show a clear trend of convergence towards the top of the figures. This means iterative learning indeed amplifies the hidden bias of Po (heaay)) > Po Po(auar)) when the imposed Heir doesn't 't impede this bias. In the Hrandom setting, the d is sampled from all the data generated by the previous generation, which makes those hard samples more likely to be sampled compared with the imitation-only settings or those with Hessylongsshon- Hence we observe the converging speed of it is slightly lower than these settings. On the contrary, when Hhard is introduced, the bias towards heasy is successfully restrained. We observe a clear competition between these two pressures: the curve first goes up, which means the bias towards easy samples is stronger. However, as the learning goes on, the curves turn down again as we later have more hard samples in D pool-
The second column of the figure demonstrates the average ranking of words in d'. We observe a similar trend in the ratio of easy samples, although our Hon never explicitly constrains it. This phenomenon hints to us that when conducting an iterative self-data-augmentatto algorithm, some unknown bias would be implicitly amplified although we already designed another Her for other properties. Imagine we are conducting the ReST algorithm (Gulcehre et 2023). We can pursue the correctness of d' by ranking all examples by training a reward model that prefers more correct responses. However, some other subtle biases, like conciseness, informativeness, etc., might be ignored by the algorithm designer and are hence unexpectedly amplified. In summary, we should bear in mind that identifying the good and bad bias in (h) is quite important for an appropriate evolution.
Finally, we use the last column and the average length of the acronym to show how make a composed Hof by combining more than one attribute of the data. It is clear that both Heasylong and Heasyshon did their jobs quite well: the converged de contains the samples with desired properties as expected. Another thing that heavily influences the results is the ratio of easy examples in dº Although the theory claims that the converged results are irrespective of d', the converging speed and the difficulty of amplifying specific bias heavily depends on d". This claim can be well supported by
27

---

System:
Same with the mibation-only case
Gen1 user:
Gen1
Same with the mitttion-oni only case
ded
(und"" "offt, "und", andd., "0' "E":"on") Rasid on nulee can you glve more esamptes??
A
Output
d
Same with the im tation only case
Please format your rule follows (only "eturn the rulel: Poosecct 1% sttonny chtttrruunn ("und', 2""ject
Gen1 assistant:
"ooft" snd
Imput Output
Gen1 assistant:
h (hlld)
Input: B. C, D Outputt off
d' bm (ellett))
Input: E-9 un undetermined
(MA))
Fund" "D". 10tt
Input: R, D,F Dutput on
Gen1 - user:
The rule you give. (TAT: aand" 'B": "oftt tCt: "und", "D" "off, "E"; "on") Hease verrfy whether
Gen2 assistant:
explain the exer ples above if not, refine all
You now have more exomples, generate that maps all Inoucs (Incluing those previous rounds) to their corresoonding outputs.
your
following fammat
Rule: ('blecc 1" etan''"tt (oott"""" and"s, "cblec: ""ott""""""""
Input:A, B. C-C Output: undesermines Input: B, Outpus
Gen1 assistant:
h self-
Input:A, E Outputt undetrrmeee
Acclogles for the oversight Let's reevaluate the exmples:
Input: R, D,F F- Dutput on
Input Output: undetermined (Rule: "und""
Hesse your follows tonly Bull: ("ojject 1h sfonf/"flf (aandto, "object Polfy"/uny
(Rute: folf", "off
Input: A, C-+ Output: undetermined [Rule: (1M's "Lnd", "C:: (und")) Input D Output: (tule (*0": off) "D"; ooft, "E' "on'll
Rule: "8" "otl" CC: "Ctd"und" "D" "ott" TE": "on"
Figure 15: Prompt design for iterated learning with self-refine (Madaan et al. 2023) as the interaction phase. The text in gray is the same as the default imitation-only setting. Note that the format of the examples might be changed (like the examples in Gen-1: assistant), which doesn't 'oss influence the experimental results.
System:
def Generate_feeddaacCCaatt h): refine_ List, refine [], for in data: X, y =dd if h(x) 1= y:
the milationn-nlly case
Gen1
Same with the mitation-only case
Anouti Output: und
d
refine["Inpue refine (rfine[[Prediie [Prddict"] h(x)
Pewas format rule follows (only return the rilej: ("ooeett 2""="on"""ff ("und's, 2object "oofff"fff/ "ooff//uudd suul
refine['Actual
Gen1 assistant:
14-Pammle (
External Interpreter
refine_list.ee www.llisttpp...... append return refine_list
(World feedback)
WW ooff off "CT "0 offf.
Gen1 user:
Gen1 - user:
"off" "p"; Applying the following inputs ooes not produce expected outpots.
you give:
rule you provided [MA"; "und", "off"off nch "und"' "0"t"off" ooff) "t" "on'")
Same with iitation only examples?
A, Predict output
ha E Helt
Gen1 assistant:
ourput: und
Input: B. D,D.E
Please refine your rule based on the above meddbacknn to make sure it ned tormat your rule sollwwngg formas.
haass
An B.C
Output und
Dutput
d'' d'-PPm(d/1t1) (c)1222))
Rule: (" ('object 1h; "oott" und's, "object 2" <ton"" Mort/ooudd b..d
Anput: B.0
Dutput
Gen1 assistant:
In
Iuule: (*A "und", "p*. "off "und"" "D" "off" "on")
Output
Figure 16: Prompt design for iterated learning with hypothesis search (Qiu et al. 2024) as the interaction phase. Compared with self-refine, it introduces an external interpreter to refine h proposed by the model, where Hall is the ground-truth one.
the fact that when N₂ is small, amplifying the bias of heasy is significantly harder than the large No case.
28

---

System:
System:
Gen1- holloing ass
You pettern following aookder
Gen1 user:
d (d(ht))))
pool
Here are some scrmnymtntt pains. The for contains
ithocome
sepersl words The anenym is the concateration
of the first letter weeeswordd in the bet. For
Gen1 assistant: d (dld")
example:
PAcromyett SSAATEER "aat Stnnder, Uhte with intor anseelyy
Anrony:: STRUBLE
List: subzoro, tocu roaping, uncouth, Mustorr
GenZ user:
koning, sarthen
d (dd'pd
hara ars some (thesame with intacos-erlyl
(19 more examptes)
Dulput STRISLE
d~D (d)
Messe provids more exampp ez following this pasom, wmere the output comos fiss.
Impult suboers, legull, reaping, smosuth, broste loaning, barthen
Hease stridly
following larps
which has
more exmpless
Ket
AAreramm stt, fist-latter concater abinn oflst the
Micao prowid |the some with tottioon only)
the words
moumple:
("Auunami: "Lai": laree, ward)..."
I"Acronmm" "Lat": worst, word?..."!
I'Aemmymms "ow"" "Lat" Sopiineiir Ioabbe;
Gen1 - assiisannt:
more mommpies d iin
['Aorosmmt "SPLATTER"; "Lss": "strident, plobie..." ..7) exmmlees)
d' d'mPPa(ddd"
Initation-only
Interaction with different Mar
Figure 17: Prompt design for iterated learning on the acronym cata-generatiin task.
Claddo3 Caudoó
GP135 Turbo Tuko
Awarange seggh
coe
knadl:
Anmeg rent
Rntwred
Figure 18: Results when adding different interaction phases (4 different seeds). All three settings demonstrate similar evolutionary trends, which match our theory quite well.
Iuth Sumeer Pootron byatledhy ceer, contney citer p.A
witt
Chcrage": "amms"
dadia berry, "ia gress Mal))
saauur cnth maloe tul, Hn wit)
"ilee
("emean Settogaee clemeee the, tein letus Mary hylmee))
jusllay ceea
arat, gou. encuge
"eemmn"" jestie tegins hisl this
the, life to
"Joor 110h anioc talng come,
Terroge" Tilly scctrtio wagg parr pery, oller
Temmy" mm)
dove tas, pronads atmeaty oflien, hor?
straderry)
agants with comm. lone,
modee eilaas onlitt thile)
cnigr)
rats, by segerias tillo siate. witenasoe's
Typiral womples penwisted Mictrallthh (let)) Midrak-hh (nght) Compared with
untele adoretii pillin comsa she sellie cimes honon chermy berr""
and Cla Mitrral tend provide eomplee in aiphabet order Mitrri-2 uzually visatte the Instructins (e.g. 12 not roch scromvm, generate more than 20 ocmoles stc).
Figure 19; An interesting observation of Mixtral series models: they have bias toward alphabet examples. However, as the Mixtral model usually has typos in their response (like the right panel), do not have the full results of these models.
29

---

Table 4: Claude3-haie haiku results under different Herr. We color the highest and lowest numbers each column differently.
Argakapis
200 15
27 366530111
151TT
1272
1176
5717
6-150-21
6.35501.00
6-225e0.22
Earsking
8300000 am
1.88849009
3913ed
(.9340001
1:0001.00
30016
1938
2010
1530
9.835
11,07748040
18:01269.. um
11,50343887
Buraken
Table 5: GPT3.5-Turbo 0125 plays with -laude3-haiku results when adding different Hoff.
Argaza
Aog-keph
41335
11200
1233
1011
Mad
8.00000
0183500.00
0.0005000
100020
60001
27032
69011
69911
8:23522.06
6.0008227
5.50
6.00e 18
344040
1.85540091
1.895.0000
1.00u100
15867
5387
6346
11.8258855
18:56-8.55
11.975.2.10
Boyskes
047560
1.888:0:00
1/00H9
0313:000
10000
14159
19015
16300
12356
4.0638000
3835.4
3,80768
30

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims are justified.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss this throughout.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

Justification:
Theorems include full statements and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Code reproduce the experiments is provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification:
Our code is publicly released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppccceeccccccccccccccat / ssssssooiccyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We describe settings in general in the main body, and details are provided in the appendix and in the published code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Results have error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
33

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We did not carefully track our computational resources, but they were mostly limited to modest numbers of API calls to existing LLMs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: pettps://neurppeee //nuurrrrs..../
Answer: [Yes]
Justification:
We follow the code cthics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper about the evolution LLM agents and potentially steering that evolution; this has roughly the same potential for impact as any L.LM work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any novel datasets or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Datasets, models, etc. are cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code for experiments is fairly simple, with documentation inline.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "yJzCHVaIAb", "reviewer": "Reviewer_Zowd", "summary": "The paper explores the evolution of Large Language Models (LLMs) through the lens of Iterated Learning (IL), drawing parallels with human cultural evolution. The authors propose a Bayesian framework to analyze how biases in LLMs are amplified over generations of learning. They introduce the concept of Iterated Learning in the context of Bayesian agents, demonstrating how subtle biases can be magnified as knowledge is transmitted across generations. The authors argue that understanding the evolutionary process of LLMs can help in designing more effective algorithms for alignment, bias mitigation, or amplification.", "questions": "- In lines 51-52 of the introduction, you mentioned, \"We believe that our analysis can enhance our understanding of LLMs, and aid in designing more effective algorithms for alignment, bias mitigation or amplification, and similar tasks.\" Could you provide more concrete examples to demonstrate how this analysis plays a role in downstream tasks and applications to strengthen this claim?", "limitations": "NA", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper has a good theoretical basis.\n- The paper's focus on bias amplification in LLMs is timely and important, given the increasing concern about fairness and bias in LLMs.\n- It is interesting from the perspective of Bayesian framework and iterative learning.", "weaknesses": "- Do you think that the Bayesian framework and iterative learning are effective and robust across different large language model agents? Could these methods lose their effectiveness with changes in model architecture or parameter scale?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. Do you think that the Bayesian framework and iterative learning …\n\nThanks for this good question. In short, the proposed framework doesn’t care about the implementation and training details of LLMs. That is also the most important benefit of using a highly abstract way (i.e., Bayesian behavior) to analyze LLM’s evolution. Based on our theory, as long as the LLM’s in-context capability is strong enough and can be approximated using a Bayesian update (as argued by Xie et al.), the subsequent analysis will hold regardless of the model's details. (Our results on different LLMs have similar trends, also showing the generalizability of this framework.) Furthermore, as stated in Appendix A, even when some assumptions are violated, some important principles uncovered by our theory still hold. For example, we experimentally showed that when the agents in different generations are different LLMs, e.g., a GPT agent playing with a Claude agent, the studied bias is also amplified. That is because although these two models have different $P_0$, as they are both trained on the datasets collected from the internet, some biases are shared in their priors. Hence their evolution still follows our theory well. \n\nAdditionally, in Section 7, we showed that when the model conducts in-weight learning instead of in-context learning (where the Bayesian learning behavior assumptions do not strictly hold), the bias could also be amplified, and a good interaction phase can mitigate that. This experiment uses a smaller open-source model (7B), suggesting the effectiveness holds even if we change the parameter scale (and setup). In summary, we are confident that the main trends described by this paper are generalizable to many practical scenarios.\n\n> 2. In lines 51-52 of the introduction, you mentioned …\n\nThanks for this suggestion. The first hint from our analysis is that the interaction phase can play an important role in guiding the model’s evolution. The result in Section 7 is a good example. In this experiment, we use carefully designed prompts to change the preference of the model in the interaction phase (i.e., let the judgment model prefer more concise responses). The results show that combining this bias and on-policy DPO makes the model converge to helpful and concise responses, and successfully constrains the amplification of length bias in $P_0$. \n\nOn the other hand, note that in our framework, $P_0$ is the model’s confidence given the instruction prompt. Hence we can design clever prompts when the model generates new examples for the next generation. The example in Lines 258-267 shows the feasibility of this idea. This example verifies that we can manipulate the bias in $P_0$ by adding a sentence in the prompt. As a result, if we can pinpoint the potential bias we want to mitigate, we can design a corresponding prompt when generating new examples from the model. We leave further exploration of this interesting direction to future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H3qd31D5co", "reviewer": "Reviewer_JuEo", "summary": "The main idea of this paper is similar to Xie et al., as both attempt to incorporate in-context learning of Large Language Models (LLMs) into the framework of Bayesian inference. This paper extends the concept to include more general multi-agent, multi-round self-improvement of LLM systems within the framework of Bayesian Iterated Learning. Based on the proposed Bayesian-IL framework, the authors prove LLM's bias amplification during self-improvement and propose a potential solution by introducing hypotheses. Experimental results confirm the existence of bias amplification during LLM self-improvement and demonstrate the effectiveness of the proposed solution in specific tasks.", "questions": "Questions are asked in the previous section.", "limitations": "Limitations are discussed in the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The attempt to conceptualize self-improvement of LLM agents with Bayesian Learning is relatively novel.\n\n2. The proposed framework provides a systematic perspective for understanding bias amplification in LLM agents and potential solutions. It conceptualizes the design of agent-based LLM systems into initialization, imitation, and interaction phases, providing guidance for reducing bias amplification.\n\n3. The experiment section presents comprehensive empirical evidence of bias amplification during LLM's self-interaction. The experiments are well-designed to cover diverse setups, task scenarios, and base LLMs. Results confirm the existence of bias amplification during LLM self-improvement and demonstrate the effectiveness of the proposed solution in specific tasks.", "weaknesses": "1. The presentation of this paper could benefit from reorganization for better reading flow. Specific comments include:\n\na. Provide a running example in the introduction, such as the ACRE experiment, to help readers better understand the concepts discussed in the theoretical proof.\n\nb. Reorganize Sections 3 and 4 to make the argument more concrete and concise. There is no need to repeat similar proofs available in previous literature (e.g., Xie et al.). Focus on the novel aspects of the proposed work, such as bias amplification.\n\nc. Add a connection paragraph before Section 5, summarizing the planned experiments and their connection with the proposed theoretical framework. Describe task environments and their nature, or refer readers to the appendix if needed. For example, the compositional language experiment should not be mentioned without context. The nature of ACRE tasks should not be discussed in the middle of a sub-experiment.\n\nd. Move Appendix A into the main text, as it contains valuable discussions highlighting the theoretical and practical value of this work.\n\n2. Both the theory and experiments do not consider multi-agent scenarios, where embodied LLM agents might be competing/collaborating, heterogeneous/homogeneous, or have partial/global observation\n\n3. In the proof of Proposition 1, the authors claim that $h \\in H_{eff}$ can be guaranteed by adding constraints during imitation and interaction phases. This process might not be applicable to all Bayesian agents during iterated learning in general. For example, if agents are unaware of task completions, it is non-trivial to \"rule out\" unsuitable hypotheses at the end of each iteration. Consider multiple embodied LLM agents interacting with a task environment with sparse rewards, where they might not receive feedback at each timestamp.\n\n4. In the proof of Proposition 2, the Markov property of hidden hypotheses and the LLM sampling process used in the second line lacks proof. Can we always find a hidden hypothesis $h$ such that $P(d|d_{t-1}, h) = P(d|h)$, meaning LLM's sampling only depends on this variable instead of the prompt input?\n\n5. There is a gap between the theoretical framework and empirical evidence. As claimed in Appendix A, most of the assumptions used in the theoretical proof are violated in the experiments. How do experimental results support the proposed Bayesian-IL framework? The claim in line 314 that the proposed framework can predict LLM's behavior is less accurate. The framework actually predicts the phenomenon of amplifying bias during LLM's self-improvement, rather than individual LLM agent's behavior (e.g., when they will make mistakes given certain inputs). The proposed framework is one explanation of bias amplification among many others.\n\n6. The solutions authors propose to reduce bias amplification all depend on domain knowledge about the task. What if the intrinsic biases of LLMs are unknown and the criteria for task completion are difficult to measure?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 0. The main idea of this paper is similar to Xie et al. …\n\nWe'd like to first highlight some differences from Xie et al. They show that LLM agents behave like Bayesian agents when doing ICL, linking LLM evolution to our Bayesian-IL framework. However, our paper focuses on LLM’s knowledge evolution when the model keeps learning from its predecessors and generates new samples for their descendants, different from Xie et al. Plus, the results in this paper also have the potential to be extended to the in-weights learning scenario, not mentioned in Xie et al.\n\n> 1 - (a, c, d). The presentation of this paper could benefit ...\n\nThanks very much for the great suggestions; we will update the next version based on them.\n\n> 1 - b. … There is no need to repeat similar proofs available in previous literature (e.g., Xie et al.). …\n\nAs stated in the previous response, we only use the technique in Xie et. al. in Proposition 2. We believe all other discussions are necessary to understand the whole story of the paper (and are not discussed in Xie et. al.). For example, the discussions of Bayesian agents and iterated learning framework in Section 3 do not make any assumptions on how the agent is implemented: it can be a pure Bayesian agent like our experiments in Appendix B, a human whose behavior is well approximated by Bayesian rule (common in cognitive science), or LLM doing in-context learning (as in Xie et. al.).\n\n> 2. Both the theory and experiments do not consider multi-agent scenarios …\n\nThanks for pointing this out. We believe for any type of multi-agent scenario, the interaction between two agents (one agent can be the environment) is the building block. Hence this paper starts from this atom of behavior to understand the whole process. \n\nWe originally included some experiments on multi-agent settings, particularly auction games, where more than two agents cooperate or compete in the bidding process. We add different personas to different agents (e.g., some are conservative, some are radical, and some are superstitious always bidding for a number ending with 8). We find that after several rounds, the agent’s preferences are indeed amplified. However, as we already have several experiments under different settings, making the story a bit hard to digest, we decided to focus only on the theory part and the most fundamental 2-agent settings to verify the theory. We will add a discussion on multi-agents in our next revision.\n\n> 3. In the proof of Proposition 1, the authors claim …\n\nThanks for this insightful question. It is true that in most practical cases, designing a perfect interaction phase that imposes $h\\in \\mathcal{H}_{eff}$ is almost impossible. However, as we mentioned in our Appendix A and Section 7, a filtering (or ranking) design during the interaction phase plays a similar role of imposing this constraint: the practical algorithm will assign more weights and credits to those examples coming from such $h$s. If this filter is strong enough, it is equivalently that we have this constraint. If not, the strict guarantee of the theory no longer holds, but the general trend that “a good interaction phase can down-weigh bad examples” still holds.\n\nNote that the theory only provides a guarantee in the ideal case with assumptions. Since our model and tasks are becoming more complicated, some of these assumptions are very easily violated. Then, the experimental results support that when specific assumptions are mildly violated, some important principles uncovered by the theory still hold, which could be a guide for our future system. For this paper, specifically, we want to highlight that a well-designed interaction phase and manipulating prompts when generating new examples are two efficient ways to guide LLM’s evolution.\n\n> 4. In the proof of Proposition 2, the Markov property … \n\nThere is a small misunderstanding here. First, we claim sampling $d\\sim P(d|d_{t-1})$ is equivalent to first sampling $h^*$, then $d\\sim P(d|h^*)$. The $P(d|d_{t-1},h)=P(d|h)$ never occurs in our paper.  That is important for the implicit $h$ case because the model can only get new samples based on the data generated in the previous generation. Using this proposition can extend our Bayesian-IL analysis to these more practical cases. The ACRE task, where $h$ is explicit, does not need Proposition 2; only Proposition 1 is enough.\n\n> 5. There is a gap between the theoretical framework and empirical evidence …\n\nThanks for pointing that out. We will change the imprecise claims throughout the paper and also move the discussions about assumptions and practical systems in the main context. We also believe the carefully designed measurements in sections 5 and 6 (although they are a bit hard to understand) support the results well.\n\n> 6. The solutions authors propose to reduce bias amplification all depend on domain …\n\nThanks for this great question. We also believe that the domain knowledge about the task is important for figuring out the hidden bias in the prior. However, when the bias is hidden, our method provides a reasonable approach towards probing and pinpointing the potential bias.\n\nNote that based on our analysis, if we kept doing self-improvement in-weight learning, like self-reward, for several generations, some hidden bias would also be amplified. If that occurs, we might have to re-train the model and waste all the computing resources. However, based on our theory, we can first conduct the iterated in-context learning for several generations, during which, we can define different prompts to elicit the potential bias in the model’s prior. After pinpointing such biases in the prior, we can then design a corresponding interaction phase for the self-improving in-weights learning, and hence avoid amplifying them. Furthermore, even when the intrinsic biases are unknown, we could still design reasonable heuristics for filtering, e.g. using a reward model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WmYTKE1Kmn", "reviewer": "Reviewer_YSn2", "summary": "The paper discusses how the widespread adoption of Large Language Models (LLMs) and their iterative interactions can lead to an evolutionary process similar to human cultural evolution. It leverages the Bayesian Iterated Learning (IL) framework to explain how subtle biases in LLMs are amplified over iterations. The authors outline key characteristics of LLM behavior in this framework, supported by experimental verification, to predict and guide the evolution of LLMs.", "questions": "1. Can you provide more details about the experimental setup, specifically the choice of datasets and evaluation metrics used to validate your framework?", "limitations": "See weaknesses above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The application of the Bayesian-IL framework to LLM evolution is innovative and offers a new perspective on understanding and guiding LLM behavior.\n2. The paper provides a solid theoretical foundation and validates its claims with comprehensive experiments across various LLMs.\n3. The proposed strategies for guiding LLM evolution can be highly beneficial for designing more effective algorithms for bias mitigation and alignment.", "weaknesses": "1. The framework relies on several assumptions that may not hold in all practical scenarios, potentially limiting its applicability.\n2. While the framework is validated on specific LLMs, its generalizability to all LLM systems remains to be fully explored.\n3. The paper primarily addresses explicit biases, and while it acknowledges the challenge of implicit biases, it does not provide concrete solutions for detecting and mitigating them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. The framework relies on several assumptions that may not hold …\n\nThanks for this question. The theoretical proof indeed needs a lot of assumptions which might not be true in practical systems. However, as we stated in Appendix A, we only expect the general trend (i.e., the bias amplification and the constraining role of the interaction phase) to hold for practical models. The latter is well supported by our experimental results in different settings.\n\nThe main message we want to highlight is that theory only provides a guarantee in the ideal case under assumptions. Since our model and tasks are becoming more and more complicated, some of these assumptions are very easily violated. Then, the experimental results support that when specific assumptions are mildly violated, some important principles uncovered by the theory still hold, which could be a guide for designing our future systems. For this paper, specifically, we conclude that a well-designed interaction phase and manipulating prompts when generating new examples are two efficient ways to guide LLM’s evolution (i.e., help amplify good biases or restrain bad ones).\n\n> 2. While the framework is validated on specific LLMs …\n\nOur in-context learning experiments are indeed API-based, and only verified on several commercial LLMs like GPT, Claude, and Mistral. The in-weights learning experiment in section 7 is on LLama2. Our experiments cover the representatives of SOTA closed-source and open-source models with different scales. We believe this suggests our method could generalize across different models. We would also like to extend our analysis to more general LLMs and tasks in our future work. \n\nMore experimental results on various LLMs will definitely argue for or against our proposed theory, and we expect future work may do so. Nevertheless, as we mentioned in the related works of the paper, we think we’ve found much evidence (experimental results across multiple settings, theoretical analysis) to support our claims.\n\n> 3. The paper primarily addresses explicit biases …\n\nThanks for this great question. We also believe that understanding the hidden bias in LLM prior is important for applying it to different domains. We believe a general principle for manipulating the bias is that: we must first pinpoint what the bias is, then find some method to mitigate it. The method provided in this paper has the potential to figure out such a bias efficiently. Note that based on our analysis, if we kept doing self-improvement in-weight learning, like self-reward, for several generations, some hidden bias would also be amplified. If that occurs, we might have to re-train the model and waste all the computing resources. However, based on our theory, we can first conduct the iterated in-context learning for several generations, during which, we can define different prompts to elicit the potential bias in the model’s prior (just like how we manipulate the prior in the ACRE task). After pinpointing such biases in the prior, we can then design a corresponding interaction phase for the self-improving in-weights learning, and hence avoid amplifying them.\n\n> 4. Can you provide more details about the experimental setup …\n\nThanks for pointing this out. All details of the in-context learning experiments can be found in Appendix D. We will add the details of the experiments in Section 7 in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CgQmHCczbZ", "reviewer": "Reviewer_1qvU", "summary": "This paper investigates iterated learning with large language models (LLMs). The authors present theoretical analyses demonstrating how bias is amplified when LLMs are chained in an iterated learning process. Moreover, they conduct experiments with LLMs that confirm the amplification of bias during iterated learning. When the weights of LLMs are allowed to be updated during this process, the authors find that manipulating the information passed through intergenerational transmission can train LLMs to exhibit a desirable bias, such as providing responses that are both helpful and concise.", "questions": "First of all, the font sizes in the tables and figures are too small to read. Second, the results in each table and figure are not self-explanatory. Terms like “Corr”, “r_20”, and “BOTH” in Table 1, as well as the numbers in Table 2, lack clear definitions and justification. What do these terms mean specifically? Are smaller values in Table 2 better? The text does not explain these terms either. Due to the extensive theoretical analysis, the key experimental results received insufficient attention in the main text. Moreover, the paper ends abruptly without sufficient discussion and lacks sections on limitations and future directions.", "limitations": "This work contributes useful empirical knowledge to the field of LLMs. However, I would like to suggest that the authors place greater emphasis on the empirical results and ensure these results are clearly presented. A key missing piece is the important comparison between in-weight iterated learning and model collapse. It would be beneficial to identify when in-weight iterated learning is effective and when model collapse is likely to occur.\n\nReferences\n\nGriffiths, T. L., & Kalish, M. L. (2007). Language evolution by iterated learning with Bayesian agents. Cognitive science, 31(3), 441-480.\n\nXie, S. M., Raghunathan, A., Liang, P., & Ma, T. (2021). An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080.\n\nZhang, L., McCoy, R. T., Sumers, T. R., Zhu, J. Q., & Griffiths, T. L. (2023). Deep de Finetti: Recovering Topic Distributions from Large Language Models. arXiv preprint arXiv:2312.14226.\n\nShumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2023). The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17493.\n\nLu, Y., Singhal, S., Strub, F., Courville, A., & Pietquin, O. (2020, November). Countering language drift with seeded iterated learning. In International Conference on Machine Learning (pp. 6437-6447). PMLR.", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "The experiments were conducted in three successive steps: in-context learning with both explicit and implicit hypotheses, followed by in-weight learning. I appreciate this structure, although the results are somewhat difficult to interpret (as detailed below). These three experiments are quite novel.", "weaknesses": "The paper devotes extensive text (Sections 3 and 4) to presenting their key theoretical results: (1) iterated learning with Bayesian agents leads to a stationary distribution that effectively samples from the agent’s prior, and (2) in-context learning can be understood as an implicit Bayesian inference process. I found both results to lack novelty as they are essentially paraphrasing existing findings. For instance, Griffiths & Kalish (2007) have already provided the result (1), and both Xie et al. (2021) and Zhang et al. (2023) have presented the same result (2). Therefore, I do not believe the theoretical results should be featured in such detail given the established prior work.\n\nDue to lack of clarity, I am also uncertain about the results from Sections 5 and 6. The results from Section 7 are the clearest to me, where controlling the transmission channel to pass on more helpful and concise data allows the LLM to amplify this good bias. However, I am unsure if this is a viable long-term approach, as it still involves training a model using data generated by the same model. Numerous studies, including Shumailov et al. (2023) and Lu et al. (2020), have shown that this method can eventually lead to model collapse. The authors might not have observed model collapse because the iterated learning chain was not run for a sufficient number of iterations. Furthermore, the evidence that imposing too strong of a constraint (line 384) negatively impacts the model’s helpfulness during iterated learning seems to support this idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. The paper devotes extensive text (Sections 3 and 4) …\n\nThere is an important misunderstanding in this claim. The Bayesian-IL framework has been discussed in the cognitive science community for decades; Griffith et. al. formally describe it using the Bayesian framework. In that paper, sections 1-4 prove the case when $h$ is sampled from the posterior. The converging distribution is just the prior. In sections 5-6, they experimentally showed that when considering $h$ that maximizes the posterior, the converging distribution is a delta distribution centered at the $h$ with the highest prior. That paper provides an idea of proving this; we formalize that in our appendix.\n\nMore importantly, most of the works in Bayesian-IL including Griffith et. al. only consider an **imitation-only IL**. However, as demonstrated in many related works, the interaction phase is very important in avoiding mode collapse or enhancing the quality of knowledge. As a result, formalizing this phase is necessary. That is one of our main contributions. Since the interaction phase usually has different forms, we introduce $h\\in H_{eff}$ as one specific formalization that roughly makes sense across tasks. We verified that by imposing this constraint, the converging posterior will be the argmax of $P_0$ within this $H_{eff}$. If this $H_{eff}$ rules out degenerate knowledge, our algorithm will not degenerate. If this $H_{eff}$ rules out harmful responses, our LLM will evolve to be less harmful.\n\nIn Section 4, we use a similar proof technique as Xie et al. to study a different problem. We show that when $d_{t-1}$ is large enough, sampling $d\\sim P(d|d_{t-1})$ is equivalent to first sample $h^*$, then $d\\sim P(d|h^*)$. Plus, we also discuss the in-weight learning scenario in Section 4.2, not done by Xie et al.\n\n> 2.1 Due to lack of clarity, I am also uncertain ... \n\nThe ACRE experiments include several subtle designs, which are powerful for directly supporting our theory rather than only observing the final accuracy: when $h$ is explicit, we can directly measure the evolution of all $P(h)$.\n\nThe word-generating experiment simulates a creative writing task. However, this design makes the bias easier to evaluate by counting the length or ranking difficulty of the generated words. We explain why these two experiments are so important for our theory in Appendix A2, and will highlight that more in the next version.\n\n> 2.2 However, I am unsure if this is a viable long-term approach …\n\nWe believe self-improving, or more generally learning from AI content, will continue to become more important in the future. Such methods have the potential to break the constraints of limited data, as stated in many recent works. Our theory suggests a general trend of bias amplification as long as there some some common bias in the prior of agents in different generations. Since self-interaction is flourishing and bias amplification is unavoidable, our work could inspire more efficient methods to solve this problem.\n\n> 2.3 The authors might not have observed model collapse because the iterated learning …\n\nIt is true that if we amplify the bias too much. $P_0$ will degenerate, i.e., some $h$ in the model will disappear. This does not contradict our analysis. Actually, in Table 1, the low correct prediction on $d^0$, when only imitation exists, falls into the model collapse scenario. However, as implied by our theory, and many related works about iterated learning, adding a good interaction phase is very efficient for avoiding model collapse. The experiments in Appendix B also clearly show that good interaction avoids converging to degenerate languages.\n\n> 3.1 First of all, the font sizes ... Second, the results in each table and figure are not self-explanatory. …\n\nWe'll fix font size and add more detailed explanations of the notations in the captions. For the terms used in Table 1, all the numbers are evaluations of the converged $h$. An explanation of Corr. d0 can be found in line 295, i.e., the number of correct predictions of the $h$ in d0. The r_20 is the ratio of how many screen:off occur in the final $h$, because following our prompt format, the value of the last object “screen” always occurs at the 20th token. BOTH means how many $h$ satisfies both requirements mentioned above. The results of these measurements all match our theory well. That is why we believe although this experiment is hard to understand, they give stronger support for our theory than only observing the final accuracy.\n\n> 3.2 Are smaller values in Table 2 better? …\n\nFor results in Table 2, the numbers verify that we can control the bias amplification using the method suggested by our theory. For example, the imitation-only amplifies the model’s own bias, hence the ratio of easy works increases to 96%. However, if we believe this bias is not good and want to guide the model to generate more hard words, we can design the corresponding interaction phase. So in the line named “hard”, the ratio of easy works is the lowest among all settings. The average rank and average length do similar things. We believe combining Figure 3 and the explanation in Section 6 helps to understand Table 2 better.\n\n> 4. This work contributes useful empirical knowledge to the field of LLMs. However, I would like to suggest that …\n\nThanks for this suggestion. The main motivation of our paper is to provide theoretical support to understand existing self-data-augmentation methods. Since the theory in this paper has important differences from the existing ones, we decided to talk more about the theory and toy experiments.\n\nRegarding the question of when mode collapse occurs: our answer is “the method is beneficial before mode collapse”, similar to “the model keeps improving before overfitting”. Similar to early-stopping in traditional systems, we can also stop the model’s evolution when we observe mode collapse. Techniques for this, though, are out of the scope of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors seem to have addressed the various concerns the reviewers expressed in their initial reviews.  Ultimately, three of the four reviewers side with accept, while the fourth reviewer maintains a "reject."  From my perspective, the authors have satisfactorily addressed many (but potentially not all) of this reviewers' questions.  I believe the paper addresses an interesting and important topic and makes a solid contribution.  Therefore, I advocate for the paper being accepted if there is space.  I encourage the authors to implement what they indicated they would do in the discussion to further improve the paper.
Decision: Accept (poster)