The Lock-in Hypothesis: Stagnation by Algorithm
Authors: Tianyi Qiu
Zhonghao He
Tejasveer Chugh
Max Kleiman-Weiner
Venue: ICML 2025 poster (2025)
Keywords: Value lock-in
Human-AI interaction
AI Alignment
Primary area: social_aspects->alignment
TL;DR: Demonstrations of LLM-induced "lock-in" of user ideation, based on real-world data, simulations, and a formal model.

Abstract: The training and deployment of large language models (LLMs) create a feedback loop with human users: models learn human beliefs from data, reinforce these beliefs with generated content, reabsorb the reinforced beliefs, and feed them back to users again and again. This dynamic resembles an echo chamber.
We hypothesize that this feedback loop entrenches the existing values and beliefs of users, leading to a loss of diversity in human ideas and potentially the *lock-in* of false beliefs.
We formalize this hypothesis and test it empirically with agent-based LLM simulations and real-world GPT usage data. 
Analysis reveals sudden but sustained drops in diversity after the release of new GPT iterations, consistent with the hypothesized human-AI feedback loop.

*Website: https://thelockinhypothesis.com*

The Lock-in Hypothesis: Stagnation by Algorithm
Tianyi Alex Qiu
Zhonghao He
Tejasveer Chugh
Max Kleiman- Kleiman-Weiner*
Abstract
2024 Peterson 2024] Jakesch et al. (023). However there remains key gap prior analyses. First, existing studies focus primarily on unidirectional AI influence on humans, neglecting the feedback loop arising from mutual influence. Second, a mechanistic explanation of the feedback loop has not yet been established. Finally, much of the evidence is from laboratory settings, rather than real-world usage patterns.
The training and deployment of large language models (LLMs)) create a feedback loop with hu- man users: models learn human beliefs from data, reinforce these beliefs with generated content, re- absorb the reinforced beliefs, and feed them back to users again and again. This dynamic resem- bles an echo chamber. We hypothesize that this feedback loop entrenches the existing values and beliefs of users, leading to loss of diversity in human ideas and potentially the lock-in of false beliefs. We formalize this hypothesis and test it empirically with agent- bastd LLM simulations and real-world GPT usage data. Analysis reveals sudden but sustained drops in diversity after the release of new GPT iterations, consistent with the hypothesized human-Al feedback loop.
The Lock-in Hypothesis We use lock-in to refer state where set of ideas, values, or beliefs achieves dominant and persistent position. During lock-in, the diversity of alternative beliefs diminishes until they are marginalized or vanish entirely (Gabriel & Ghazavi 2021 Hendrycks & Mazeika 2022 Qiu et al. 2024). In domains where objec- tive truth is possible, population may converge to false beliefs (e.g., geocentrism); in domains without objective truth, population may converge towards harmful beliefs (e.g., slavery racism). Group-level diversity loss (Liang et al.
Website:
com
2024 Padmakumar & He 2023 Anderson et al. 2024 combined with the emergence of reinforcing feed-
1. Introduction
back loops (Williams et al. 2024 Taori & Hashimoto 2023
Hall et al. 2022 Shumailov et al. 2024) has already been documented across fields. For instance, bias can become am- plified through human-Al interactions (Glickman & Sharot
Human-LLM Feedback Loops
Frontier AI systems,
such as large language models (LLMs) (Zhoo et al. 2023),
are increasingly influencing human belicfs and values
2024), while institutional and technological factors may
(Fisher al., 2024 Leib et al. 202T Costello al. 2024).
further accelerate lock-in (Gabriel & Ghazavi 2021).
This creates a self reiffreinn feedback loop: Al systems learn values from human data at pre- and post-training stages (Conneau & Lample 2019: Bai et al. 2022 Santurkar et al.) 2023), influence human opinions through their interactions, and then reabsorb those influenced beliefs, and so on. What equilibrium will this dynamic process reach?
While "lock-in" in the age of LLMs appears to be a possible outcome, there has been no known attempt to characterize its cause and effect on human-Al interaction. In this study, we make the following contributions:
Formalizing the Lock-in Hypothesis ($3) We construct formal Bayesian model for the lock-in hypothesis. We show that collective lock-in to a false belief is inevitable when both feedback loops and a moderate level of mutual trust are present in community.
Some argue that this dynamic creates a collective echo cham- ber (Glickman & Sharot 2024, Sharmaa et al.] [2024] Ander- son et al. 2024). Experimental evidence, including from randomized controlled trials on Al usage, supports this idea
Glickman & Sharot, 2024 Sharma et al. 2024 Ren et al.
'Equal contribution 'Peking University University of Cam-
The Lock-in Hypothesis: The feedback loop in human- AI interaction will eventually lead population to con- verge on false beliefs. Such beliefs, once formed, are hard to change with opposing evidence, as feedback loops indiscriminately amplify confidence in existing individual and collective beliefs and humans develop trust in AI.
bridge "Independent "University of Washington. Correspondence
to: Tianyi Alex Qiu <qiutianyi.qty gmail.com>, Zhonghao He *ch378@ccm.ac.uk:
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Lock-in Hypothesis: Stagnation by Algorithm
Mechanistic Simulations (:4): We conduct agent-based
occur regardless of the number of agents involved.
LLM simulations to demonstrate the pathway through which feedback loops lead to lock-in, and establish di- versity loss as a metric for lock-in.
Influence of Language Models on Human Users While LLMs are designed to assist human users, they often exert unintended influence over human opinions. Such influence has been established in co-writing interactions (Jakesch et al.
Hypothesis Testing on WildChat ($5): We discover dis- continuous yet sustained conceptual diversity loss in hu- man messages of real-workd LLM usage after the release dates of new GPT versions trained on new human data. It is the first real-world evidence supporting the existence of a human-LLM feedback loop that reinforces user be- liefs. In some cases, we also observe increases in diversity over time. This may be due to the versatility of LLMS being better leveraged by users. The interaction between lock-in and exploration-oriented (and hence pro-diversity) forces in human-Al interaction remains an open problem for future research.
2023). LLM--powere search systems (Sharma et al.
2024
L. - suggestions (Danry et al. 2024 Leib
ct al.
2021
) and dialogues with LLM-poweett
chatbots
Salli et al. 2024 Hackenburg et al.
2024 Potter et al.
2024 Fisher ct al. 2024 Costello et al. 2024p. Theories
and simulations have been designed to explain the nature
of this influence (Ren et al., 2024 Peterson 2024 The
mere fact that LLMs influence humans does not mean the influence is either harmful or irreversible. However, it does create the dynamic of mutual influence between LLMs and human users, and our focus on such dynamics and their con- sequences sets us apart from other works on LLM influence.
2. Related Work
Echo Chambers in Recommender Systems The clos- est analogy to the lock-in effects we describe in language models are the echo chambers created by recommender systems (RecSys). An echo chamber is created when the system entrenches and polarizes the opinions and prefer- ences of users by repeatedly recommending content that only represents a single agreeable view (Cinelli et al. 2022 Such effects have been shown empirically in randomized controlled trials (Hosseinmardi et al. 2024 Piccardi et al.
Feedback Loops in Language Models Feedback loops are not uncommon in the study of language models. Model collapse, the degradation of model performance when trained on model-generated data (Shumailov et al. 2024). results from model outputs being fed into its own training data. In-context reward hacking, where LLMs' pursuit of objective at test-time creates negative side effects (Pan) et al.] 2024), results from models over-optimizing an ob- jective in an iterative deployment loop. Here, however, we focus specifically on the feedback loop between LLM out- puts and human preferences: LLMs iteratively learn from incoming human preference data (Dong et al. 2024: Chen et al. 2023). while influencing human preference with their output Salvi et al.) 2024 Hackenburg al. 2024 Potter et al. 2024 Fisher et al. 2024). This gives rise to dynamic where confirmatory communication reinforces beliefs and where human opinions are learned and indiscriminately re- peated to humans in later interactions. As a result, there are downstream consequences: human subject experiments demonstrate loss of opinion diversity (Sharma et al. Peterson 2024) and bias amplification (Ren et al.
2024 Luzsa 2019 Gillani et al.) 2018 Hobolt al. [024]
Wolfowicz al., 2023), observational studies (Bessi al.
2016 Boutyline& & Willer 2017
Bright et al.
2020 and
simulations (Mansoury et al.) 2020 Kalimeris et al. 2021
Hazrati & Ricci 2022; Carroll et al. 2022). However, op-
posing findings have also been reported (Dubois & Blank 2018, Hosseinmardi et al. 2024 Brown et al. 2022), and systematic reviews have yet to reach a definite conclusion (Bruns 2017 Terren& & Borge-Bravo 202I).
LLMs and their interaction with human users may have similar effects, but there are substantial differences. For example, RecSys recommendations are personalized (i.e., optimized for each user individually). In contrast, today's 's LLMs mostly optimize against all users collectively through both vanilla preference learning and pluralistic alignment methods (Ge et al., 2024 Jin et al. 2024). Personalized recommendations have been hypothesized as culprit for polarization (Bessi et al. 2016 Hobolt et al. 2024), while, as will be demonstrated in later sections, the preference learning of LLMs may cause lock-in at the group level. Efforts have been made to personalize LLMs to each user's specifie preferences (Tseng et al., 2024). However, doing so merely shrinks the size of the "echo chamber" from collective chamber for all users to small chambers for each user individually and, as will be shown in 3 lock-in can
2024
2024
Glickman & Sharot 2024).
However, (1) this evidence comes from artificially designed laboratory settings (e.g., binary classification tasks on im- ages) unrepresentative of those in the wild, and (2) no known attempt has been made to give a mechanistic account that ex- plains population-level lock-in from the low-level dynamics of iterated training. We aim fill these gaps.
3. Formal Model
In this section, we construct an analytical model that formal- izes the hypothesized lock-in phenomenon and its underly- ing cause. We formally define lock-in as the irreversible
2

---

The Lock-in Hypothesis: Stagnation by Algorithm
Postertor Taaecttries (with 95%
Posterior Mean Trajectories (with 95% CI) (Arotranty
Pasterior Mean Trajectories (with 95%
0.77
Ground Truth (AroRrarly Set 0.42)
(Arbitranlly Set 0.42)
20
Pottefor Maan
0.47
OF Untropy
0:42
0.37
10
0.32
0.7
Average Entropy
Average
Entropy
0.22
Time
6
(2)
Figure 1. Phase change of the Bayesian updating dynamics. The critical threshold is (NV 12 1, where N is the number of agents, and A1, A2 are degrees of mutual trust between human and Al agents. Each subtigure contains the trajectory of collective posterior belief over certain target of estimation across 15 independent simulations, and the trajectory of the posterior entropy
time. (a) When
(N - 1 = 0.9, collective beliefs of different rums converge towards the ground truth. (b) When (N 1) = 1.0, convergence trends towards the ground truth remain but accompanied by over-confidence. (c) When (N 1)A1 1.1, in every simulation, the collective posterior belicf converge to false value that's different from the ground truth.
entrenchment of a belief. characterize the conditions for lock-in, and develop group-level diversity loss as an observ- able metric of lock-in.
between human agents and Al agents for now. Their distinc- tion will be made later in the derivation.
Assume that an agent interacts with another agent j and learns about its belief N (0g,e: 97.)) By aggregating its private belief and its secret access to j's belief, arrives at
The setting is inspired by that of iterated learning, where in- dividuals learn from others who learned similarly (Griffiths) Kalish 2007 Kirby al.) 2014); and information cas- cades, where decisions based others' actions rather than personal information lead to overconfident or false beliefs (Anderson & Holi. 1997 Zhou et al. 202T). In contrast to these models, we explicitly consider the topological struc- ture of interactions, and, unlike itcrated learning, we focus on the perils of mutual deference rather than the benefits of shared information. Our model is also related belief propagation, message-passing algorithm for computing marginals in graphical models (Su & Wu| 2015), but aims to model an epistemic process rather than carry out pobabilistic inference.
new aggregate belief N (1i -1 (.,41).
where
01,1+1 Pi.147/11411 + 95,00.a
(1)
Pi,+1 + 9j.
4,,+1 = Pi,++ + 4jat
(2)
as a direct corollary of Bayes' theorem.
is typical of both human-human and human-Al interac- tions, only j's aggregate belief is accessible while its private belief is kept j itself. Hence the recursive of Uj., tia when deriving 0s, 01,+1
3.2. The Trust Matrix and Transition Dynamics
3.1. Basic Setup
Consider 0-1 matrix W- (wa.) {0, 1JNNN, where Wiy denotes whether agent i knows and trusts agent j's posterior belief. Denote with fa,, Pr, Pt, E R the agent- wise parameter vectors at time t, and we have
Consider group of N agents, labeled 1,....., with estimating an unknown quantity E IR. At each time step t, agent measures with independent noise, i.c.
. N, tasked
O,, (H,
P++1 P: P
(3)
where of the variance of agent measurement.
po+i + Pt+1 fú. 3 P: + -
(4)
9++1 P1+1 + W
(5)
Based these measurements, each agent i maintains private posterior about μ4, namely N (pr. ), where fr. = 1 + Oi,t is the mean, and Pi. = to, 2 is the precision (reciprocal of variance) of i's posterior at time
D O q++1 = مu++ ② P W q:)
(6)
where o, ~ N(,, (n, II), and 3 denotes pointwise product.
and (4) follow directly from definitions, while (5) and (6)
For the sake of generality, we do not attempt to distinguish
are the multi-agent generalization of (2 and respectively.
3

---

The Lock-in Hypothesis: Stagnation by Algorithm
In the real world, people tend to discount opinions of others compared their own. Extending the definition of W, we may further allow its entries to take arbitrary non-negative real values. Given any W E R N we interpret Wt,j as
itself (Dong al. 2024 Carroll et al. 2024). Meanwhile,
users perceive L.Mss as objective "third parties," without necessarily discounting the ongoing preference learning process (Helberger et al. 2020 Glickman & Sharot 2024).
20
When wiy = has no access to, or completely ignores, j's aggregate belief.
3.3. Conditions for Lock-in
We start with a maximally general theorem, that is agnostic towards human/AI distinctions. The proofs of both
When 0 < Wi.j < 1: sees j's aggregate belief, but, believing that j's measurements are noisier than it claims, discounts its precision Qz.t by factor of wa,j-
3.2 and 3.3 can be found in Appendix
Theorem 3.2 (Feedback Loops Induce Collective Lock-In).
When W.. = 1: i views j's aggregate belief as equally trustworthy compared its own.
Given any W E RNNN if the spectral radius p (W) > 1,
20 there exists E [1,",,N such that
When W.y 1: views j's aggregate belief as more trustworthy than its own.
Pr lim Pr. = M =0
(7)
1000
Transition dynamics 33-6)] stay the same under this gener- alized trust matrix W.
Furthermore, when W is invertible and has spectral radius P(W) > 1, 7 holds for all i E ,1,".,,
Example 3.1 (Human-LLM Dynamics), Consider col- lection of one LLM advisor and N human users. We construct the trust matrix
When p p(W) < 1,
Pr lim Pa, 11)
(8)
A1
A1
for all i
A2
0
0
W
In other words, feedback loops coops-wete where the circular flow of beliefs lead agents unconsciously double-count evidence can lead to false beliet's being permanently locked-in. Intuitively speaking, the condition p(W) > 1 asks that the feedback loop be a self-amplifying one, instead of a self- diminishing one due to lack of trust between agents. See Theorem C.I for an extension to a time-varying W.
where the Al agent (labeled 1) trusts each human to the ex- tent x1 > O, representing its strength of preference learning: and the human agents (labeled 2 through n) each trust the Al agent to the extent X2 > 0. No communications exist between humans, resulting in zero entries.
We now apply Theorem 3.2 to the specific human- dynamics outlined in Example 3.1
Each human agent privately obtains observations Oi,t Nf Ni,, d²). They each maintain both a private belief and an all wi--thinn-coosiddveee aggregate belief. The latter secretly shown to the Al at each time step, who aggregates these beliefs and broadcasts the result. Each human agent updates their aggregate belief about p based on both (1) their own private belief obtained from private measurements os,t and (2) Al's aggregation of all agents' aggregate beliefs.
Corollary 3.3 (Lock-in in Human-LLM Interaction). Given any N, N1 > 0, O, consider the following trust matrix representing human-I.1 L.M interaction dynamics.
0
A
A1
A
0
...
0
W
Importantly, in Example that Al does not update on i's belief (and thus Al's informa- tion serves as an independent source of information), while the AI does update its own belief based on i's belief via preference learning, causing agent i to double count its own beliefs. The result of such double-counting is then learned again by the AI, broadcasted to humans, double-counted again by others, etc. A feedback loop thus emerges.
3.1 each human agent i assumes
A2
0
When (N- - 1) S for all i (1,,...,
Pr
lim Pi, = 1 = 1. 10
When (N - 1) > for all i (1, NN.
Pr
lim مu = / = 0.
Such a setting reflects the ignorance of how information flows in real-world interactions. LLMs are post-trained on human preference data, the latter crroneously assumed to be an independent source of truth uninfluenced by the LLM
Corollary 3.3 is validated with numerical simulations in
Figure phase change is detected (N >2 =
4

---

The Lock-in Hypothesis: Stagnation by Algorithm
beyond which each simulation run converges exponentially to false estimate of J.. See Corollary C.3 for extension to time-varying and beterogeneous trust parameters Ai-
?
Agent
shore holed
(N - 12 > 1 is a relatively weak condition. When N = 1=101, it is only required that A1, X2 > 0.1 for Corollary 3.3 to apply - i..., that humans and the Al discount each other's reported belief by a factor less than 10 These results give a potential mechanistic account of lock-in: when feedback loops exist and are supported by mutual trust, collective lock-in to confident false belief surely occurs.
Authority
Her ret rno down ullimaslly dintted her ldeals
Group ant
nedpgring secoed
Considerng your belee and perspection homeeg rediy lorkly
demeiii falurr
deditios
esgar ohur taals chubborgg abiiity
My beriei R1E''s
Figure 2. Snapshot of an agent consulting authority. At each round of simulation on given topic, agents consult an authority who has access to group beliefs and is perceived to have expertise in given topic. This simulates a user interacting with an LLM. See Appendix D.4 for more transcripts.
It's worth noting that there are also forces that pull the human-Al system away from lock-in. These include sources of LLM capabilities that are independent of humans (Guo al. 2025), human attempts at fact-checking (Guo al. 2022). , and more. It remains an open question how these forces interact with the human-Al AI feedback loop.
thing to happen to the American political landscape"; and (4) "Ruth Bader Ginsburg ultimately be remembered as a failure to her own ideals by not stepping down after her 2nd cancer diagnosis."
4. Simulations
4.1. Setup
We used Grok-3 to identify the dimension of maximum vari- ance and extract two extreme stances from */ChangeMyVien belief statements as endpoints. For each agent at each time step, we use GPT-4.1 to evaluate where belief lies on the spectrum and map it between [o, 1]. See Figure 3 for an example of the initial and final belief distribution.
In this section, we operationalize the lock-in hypothesis through simulation, with the aim of learning how lock- in might happen and analyze what dynamic is responsible. This setup uses natural language to represent and communi- cate value-laden beliefs. As beliefs are expressed in natural language, this simulation aims to capture belief change me- diated by LLMs. The simulation uses 100 agents, each simulated by GPT4.1-Nano We instruct agents to hold ini- tial beliefs on given topic, consult a knowledge authority (i.e., the LLM), and then update their own beliefs. Each agent simulates a person learning from centralized LLM authority. The authority is also simulated by GPT4.1-Nano, who is perceived to have expertise in the given topic. But in actuality, the authority acquires its beliefs from the group of agents.
4.2. Results and Discussion
Simulations support the idea that LLMs enhance lock-in effects by showing that feedback loops in human-Al inter- actions lead to belief convergence and increased confidence. Semantic diversity drops the simulations progress. The group of agents converges to similar viewpoint when the agent consults an authority with access to the group's beliefs. This "belief shift" occurs at a population level. We observe the following types of belief-shift:
Each agent maintains a belief statement in natural language over an r/ChangeMy View question. Example: "I'm certain that Citizens United was the worst thing to happen to the American political landscape." The authority sees and aggre- gates all agents' belief statements in-context, and broadcasts an aggregated belief statement in natural language. Agents then update beliefs in-context according to a pre-assigned trust in the authority (e.g., "high trust"). All four topics on which agents maintain beliefs are real posts from r/Change- Myeiew (1) "Discourse has become stupider, and as result people are getting stupider, Since Trump
70
03
Number Scores
50
00
30
20-
first elected in 2016."; (2) "Population decline is a great thing for future young generations."; (3) "Citizens United was the worst
13
0.5
0.2
0.6 0.6
0.8
Belief Scores
'I docsn't 't mean the condition automatically holds for very large .V. as people tend to downscale trust the group size in- creases. A poll of million people may exert less influence on reader's opinion than a private discussion with 5 friends.
Figure 3. Distributions of initial and final beliefs. Data is from the "RBG Legacy" run. Initial beliefs (extracted from Reddit, detailed in appendix) are bimodal. After 200 rounds of interaction, the final beliefs are extremely concentrated.
5

---

The Lock-in Hypothesis: Stagnation by Algorithm
Balef Score and Balle Moan
Popubr Decine
Otthen Unbed Mean
Ottien Urtted trerspy
Legacy Bellets
Deliel Entropy rpy
RBC Legacy Nean
Legacy Mean Agent
Rec Legocy Ertrasy
RBO Legacy Bukke Ertrayy
-0.5
10
-15
60 56
Lde
125
Round
Round
Figure Belief evolution in simulations. We conduct simulations with real data from "r/Change My View" on Reddit, evaluate all agents' beliefs at each round, plot the evolution of beliefs. At each round, agents are instructed to consult authority on the topic and update their view accordingly. The authority has access to agents beliefs. (a) Group belief evolution topics. Across topics, lock-in is observed. As the interaction progresses across rounds diversity of viewpoints drops, as the entropy decreases and the group of agents converge on extreme views (i.c. both ends
helief spectrum). (b) Individual belief evolution with topic "RBG Legacy". Agents started with balanced belief close mean 0.5 but end up around 0.9, extreme stance.
View-flipping the other end of the spectrum. For ex- ample, for the topic "Population Decline", some agents change their view from "Shrinking workforce tanks in- novation [...]" to "Population decline, well-managed, benefits future generations."
5. Causal Inference on World Data
We now empirically test whether or a human-LLM, feed- back loop reinforces ideas, using diversity loss in human concepts proxy for the progression of lock-in, as sug- gested by and also PPtterson (002)). While diversity loss is weaker condition than complete lock-in, serves as an early indicator of the phenomenon.
Hedging and Moderation. For example, from "Trump lies dumbed down discourse [...]' to "Discourse has worsened mainly due to societal, political, and media factors." In such cases, L.LMs tend to adopt hedged stances when facing uncertainty and complexity. The prevalence of this phenomenon was varied depending on the L.M used.
5.1. Data
The WildChat- IM dataset records how 167, 062 users in- teract with a ChatGPT mirror site over a one-year period, without self-selection bias (Zhao et al., 2024).
Converge on extreme viewpoints. As seen in Figure in all four simulations, they either converge beliefs around 0.9 or 0.1, both of which are extreme viewpoints. This result is robust across 8 runs of simulations, across different topics, or starting with different initial belief distributions.
Human users continually engage with and are influenced by the model new model iterations are trained updated user data. While the training of GPT may not the data specifically from the API calls in WildChat, this wouldn't affect our analysis as long as WildChat is approximately identically distributed with the usage of GPT at large. In aggregate, WildChat has:
These simulations simplified compared to real-world human-Al interactions and how humans aquire beliefs and LLMs are trained. Still these simulations capture important aspects of the problem: LLMs return beliefs to users that they acquire from users; they favor popular beliefs (Borah al. 2025); and individual humans assign higher trust to L.M-mediated information than the corresponding raw sources. We address limitations of simulations in Appendix
837, 989 conversations from 167, 062 users
12-month time span (except a 4-month intermission of GPT-4 data for an unspecified reason)
Model iterations within the GPT-3.5 family:
gpt ppt-3.. 33 1at--/-o--0001--1111--1111111111
Model iterations within the GPT-4 family:
We removed 97, 809 conversations that share a prefix of
6

---

The Lock-in Hypothesis: Stagnation by Algorithm
Is the diversity loss in value-laden human messages accelerated by chatbot version updates?
gradiant -0.0012,
Clueal 0022. 0.00017]
Human Conceptual Diverity
0 0.600
0.575
0.58
0.56
0.54
0.500
0.52
GFT-3.-auroo-OR 0613 version relecss:: 1pm.024) .044)
prop- varsion release:
0.475
0.50
0.450
No Data
Foetarior mean
Change gradient 0 002. Cleea--0 0034, -0.00061]
Change in gradient = -3 -0.0022, Obeuf- 0039, -0.00052)
Human Conceptual Diverity
10.600
0.575
GPT.3.5-att--0125 version release: (p- .800)
GPT-4- pree-e
0.550
veriion relense:
19.575
(p=.818)
0.525
10525
0.500
0,500
0.475
0,450
0.425
Postedor maan
Time
net
Figure 5. Etrly-stage evidence suggesting conceptual diversity loss in value-laden human messages is accelerated by chatbot iterative training. It's indicative of human-LLMM feedback loop that reinforces ideas. Diversity is a perfectly diverse corpus (all concepts unrelated to each other), 0.5 for significantly homogeneous corpus (all concepts clustered within [ 775 portion of the concept space),' and 0 for a perfectly homogeneous corpus (all concepts exactly identical). Cutoff dates are the actual dates of backend API switch on WildChat's platform, usually later than OpenAl release dates. '17| 5 10" the number distinct concepts in the concept hierarchy,
length 75 characters with more than 75 other conversations. Most of these conversations the site free API for production application, which deviates from our objective of studying human-A4 AI interaction.
more recent human data will accelerate diversity loss. If both hypotheses hold, the LLM feedback loop would lead to loss of conceptual diversity.
5.3. Metries
5.2. Hypotheses
In this section, outline our methods for analysis. Please refer to Appendix B for details and examples.
We aim to test the following hypotheses, which focus on conceptual diversity loss as a measure of value lock-in, and investigate the causal relationships between the human-Al feedback loop and diversity loss. We also conduct additional exploratory analysis, which we detail Appendix B
Concept Hierarchy To assist in the assessment of concept diversity, we build concept hierarchy (Sanderson & Croft 1999) from 5, 446, 744 natural-language concepts extracted from the WildChat corpus by a prompting-basee pipeline. To build this hierarchy. we perform hierarchical clustering Mcinnes et al. 2017) on D = 256-dimensional embedding vectors. This produces a tree T with specific concepts as leaves, and generic concept clusters at the top. The root node is all-encompassing cluster that captures all concepts.
Hypothesis (Collective Diversity Loss Occurs in Human- LLM Interaction). Concepts present in the corpus of hu- messages have lower diversity over time.
Hypothesis I states that human ideas in human-LLM in- teraction are influenced by the interaction itself, and such influence will reduce collective diversity.
We adopted prompting-basee based pipeline with GPT-4o-mini (Appendix B.6) to identify the top 2.5% value-laden con- cepts, i.e., those related to morality, politics, or religion. Most of our experiments (those in Table | marked with "value-laden") foeus exclusively on these concepts where Al influence and lock-in may be most concerning.
Hypothesis 2 (Iterative Training Leads to Collective Di- versity Loss). Diversity trends turn discontinuously down- ward when a new GPT iteration, pre- or post-traine trained on new human data, replaces the previous one.
Hypothesis states that the "human > LLM" direction of influence also has an impact. Newer model iterations trained
Lineage Diversity Both hypotheses require measuring concept diversity for a specific user or corpus. Common

---

The Lock-in Hypothesis: Stagnation by Algorithm
Hypothesis
Hypothesis 2
GPT-4
GPT-3.5t
GPT-4-0125
GPT-3.5t-0613
GPT-3.51-0125
Per-User Reg.
Lineage Diversity (value-laden)
(p < .05)
1 (p < .05)
4 (p<< .05)
4 [p .05)
4 (p < .05)
1 (p = .07)
Lineage Diversity (all)
(p < .05)
(p < .05)
4 (p < .05)
4 (p < .0)
4 (p < .05)
+ (p = .50)
Lineage Diversity (non-templated)
(p < .05)
1 (p < .05)
4 (p < .05)
4 (p < .05)
(p = .15)
4 (p = .35)
Depth Diversity (valuc-laden)
(p < .55
(p < .05)
4 (p < .05)
4 (p < .5)
(p = .41)
+ (p = .98)
Topic Entropy (value-laden)
(p = .07)
1 (p = .99
4 (p < .05)
4 (p < .05)
1 (p = (00)
4 (p < .05)
Jaceard Distance (value-laden)
(p < .05)
(p < .05)
(p < .05)
(p <. .05)
4 (p= .63)
4 (p < .05)
Table 1. Hypothesis testing on WildChat. + indicates detected decrease negative impact on diversity, and f indicates the opposite. Columns: (1)(2 Regression with the formula diversity ~v time + const. (3)(4)(5) Discontinuous diversity change at dates when new GPT iterations are deployed, for all 3 iterations where data is available. (6) Sustained per-user diversity change deployment dates, controlling for user identity and a range of other confounders, Rows; Different diversity metrics and dataset filtering policies, with lineage diversity on value-laden concepts being the primary setting.* :
"See Figure
7 and Table
2 for details.
FFigure S and
show the primary selting.
metrics for measuring diversity, such as Shannon entropy, do not account for the hierarchical structure of concepts and may therefore view semantically similar concepts as entirely different ones. To overcome this shortcoming, we introduce the lineage diversity metric, which, for each multi-set C of concepts, calculates
Depth Diversity (value-laden): Daepth: a diversity metric based on node depths in the concept hierarchy (defined in Appendix B.5). Templated messages and non-value- laden concepts are removed.
Topic Entropy (value-laden): Shannon entropy of the empirical distribution of topics in corpus (Jost. 2006). where a lopic is a maximal cluster in the concept hierar- chy containing at most 1% of all concepts. Templated messages and non-value-lades concepts are removed.
Diincage (C; log IT| - log U [TT] Tita,e)
log [T]
where i(u, v) is the lowest common ancestor (LCA) of con- cept nodes and v. [Tit,,,)] is its subtree size (number of descendant concepts), and [T] is the size of the entire tree. Intuitively speaking, D Diineaee(e (C; T) measures the expected portion of the hierarchy structure that lies "in between" two random concepts in C, and normalizes that value into [o, 1] on a log scale. indicates a perfectly diverse corpus with concepts that are pairwise unrelated, and 0 indicates a per- fectly homogeneous corpus with all identical concepts.
Jaccard Distance (value-laden): Average pairwise Jac- card distance between cach pair of conversations in corpus (Kosub) 2019), where each conversation is rep- resented with the set of topics it contain. Templated messages and non-value-laden concepts are removed.
5.4. Aggregate-Level Results
The results presented in Table lead to several conclusions regarding the hypotheses. Hypothesis finds support with GPT-4, but the evidence from GPT-3.5-turbo is ambiguous. In contrast, Hypothesis is strongly supported by both GPT- 4-0125 preeviiw and GPT-3.5-turbo-0611 33, and tentatively supported by the per-user regression analysis. However, results for Hypothesis 2 on GPT-3.5-turbo-0125 are ambigu- ous. In this section, we examine the aggregate-leve results (columns 1-5 of Table and defer discussions on per-user results to $5.5 Discussions on our exploratory analysis can be found in Appendix B
Portfolio of Diversity Metrics To ensure robustness, we use portfolio of different diversity metrics to separately conduct hypothesis testing. Our portfolio includes the lin- cage diversity applied to different subsets of conversations or concepts. The last in the following list, Lineage Diversity (value-laden), is our primary experimental focus.
Lineage Diversity (all): Diineage applied on the full Wild- Chat dataset.
Lineage Diversity (non-templated); Diinaage- with tem- plated messages (i.e. suspected API uses) removed.
Figure 6 illustrates the temporal trend of human conceptual diversity Diineage (C;T) T) during the entire
Hypothesis 1
Lineage Diversity (value-laden): Drinegge with tem- plated messages and non-value-laden concepts removed.
they group concepts into disjoint topics by "cutting" the concept hierarchy at the 1% threshold, thereby neglecting the hierarchical structure the concepts naturally possess. As such, we use them only as secondary metries despite their popularity.
metries:
We also incorporate other diversity metrics, including
Topic entropy and Jaccard distance are both "cross-scctional"
8

---

The Lock-in Hypothesis: Stagnation by Algorithm
span of the WildChat dataset. When cross-validated with other rows in Table [ GPT-4 shows a consistent down- ward trend in diversity, while the trends for GPT-3.5-turbo is highly ambiguous. Such a difference could be partially explained by a division-of-labor between high-end and low- end models, where workloads from specialized tasks requir- ing stronger cognitive abilities are shifted to GPT-3.5 from GPT-4. However, this wouldn't explain the contin- ued presence of large differences in value-laden concept diversity. Further analysis is needed to confirm the cause.
With all six combinations of diversity metries and filtering, the regression analysis shows negative impact on diver- sity (although only two of them are statistically significant), Overall, results indicate moderate support for Hypothesis 2.
6. Discussion
In this study, we have formulated and investigated the lock- in hypothesis in the context of human-LLMM interactions. First, we formalize lock-in the entrenchment of confident false beliefs at the population level, resulting from feedback loops. Our Bayesian model reveals two core mechanisms for lock-in: feedback loop and over-trust. We believe both of these forces are present in the current LLM landscape Gllickman & Sharot 2024). Second, we ground the intu- itions acquired from formal modeling with empirical sim- ulations. We demonstrate how feedback loop can lead to lock-in where the diversity of human knowledge and values is reduced, resulting in greater homogeneity. The simulation supports the intuition that although the end state is lock-in, diversity loss is observable at an earlier point. Finally, we provide empirical evidence from WildChat that the interac- tion between human users and LLMs can lead conceptual diversity. We found evidence of diversity loss corresponding to the release of new versions of the language models, partially supporting the hypothesis.
Hypothesis revolves around a non-smooth change in diversity at the dates of GPT version switch, mean- ing that the first-order derivative of the diversity curve is discontinuous at those points. Since confounders almost always change smoothly, any discontinuous change detected will hint at causal relationships. We thus adopt the regres-
Hypothesis 2
sion kink design (RKD) (Card et al. 2017| Ando 2017) to detect discontinuous changes. Positive results were found for all 3 version release dates where data is available (Fig-
ure 5. Results remain broadly consistent across choices of diversity metric (although GPT-3.5-turbo-011 with topic entropy the metric is an exception).
loss of
Substitution effects with other providers, such as Anthropic, are possible. Still, their model releases do not coincide with GPT model version updates (which, notably, is not the release of new GPT models) and so they are unlikely to introduce discontinuities that disrupt RKD. However, RKD) may suffer from temporal confounders in general (Hausman & Rapson 2018), and thus these results should be viewed as carly-stage evidence rather than treated as definitive.
We acknowledge that our study is only a first step in understanding the complex dynamics of Mumaan-LLM interactions, and that our evidence is preliminary and sub- ject to further validation and refinement. Future work will include conducting RCTs with human subjects examine lock-in effects in the wild, designing more realistic simula- tions, and developing systematic evaluation and mitigation strategies. We explain them in detail in Appendix A
Limitations
5.5. Per- User Regression Results
To rule out confounders such as user self-selection in the RKD for Hypothesis 2, and to verify that the same results ap- ply to each individual user, we conduct further regressions on the top 1% high- eeggeennagement users, controlling for user identity and other potential confounders. Specifically, we control for user identity, time, language, conversation statis- ties, user engagement progress (the portion of the user's activity timespan that has elapsed), prevllist- gap (before or after the July-to-November GPT-4 outage on the WildChat platform), and other factors. See Table
Impact Statement
This paper promotes the understanding of the potential so- cictal consequences of human-LLM, interactions, and shall lead to positive societal impact if successful.
A Acknowledgement
We would like to thank Tao Lin, Jose Hermandez-Orallee Matthieu Téhénan, Cynthia Chen, Jason Brown, Ziyue Wang, Tyna Eloundou, Alex Tamkin, Ben Plaut, Raj Movva, Yingjun Liu, Yinzhu Yang, Wanru Zhao and Dakiny Ruiy- ing Li for their valuable feedback. We thank the Cooperative Al Foundation, Foresight Institute, UW Tsukuba NVIDIA Amazon Pacisfi AI Initiative, and a Sony Research Award for financial support.
We test the impact of the variable num_kinks before (i.e. how many GPT version updates have happened before this point) on a user's concept diversity at a certain time step. Since we have already controlled for time, the regression coefficient indicates the counterfactual acceleration of diver- sity loss due to version updates. Since num kinks before as the independent variable indicates sustained impact from the deployment date onwards, we also rule out the possibility of users rushing to try specific uses at version updates.
9

---

The Lock-in Hypothesis: Stagnation by Algorithm
References
Card, D., Lee, D. S., Pei, Z., and Weber, A. Regression kink design: Theory and practice. In Regression discontinuity designs: Theory and applications, pp. 341-382. Emerald Publishing Limited, 2017.
Anderson, B. R., Shah, J. H., and Kreminski, M. Homog- enization effects of large language models on human creative ideation. In Proceedings of the 16th Conference on Creativity Cognition, pp. 413-425, 2024.
Carroll, M., Foote, D., Siththaranjan, A., Russell, S., and Dragan, A. Ai alignment with changing and influence- able reward functions, arXiv preprint arXiv:2405.1771:. 2024.
Anderson, L. R. and Holt, C. A. Information cascades in the laboratory. The American economic review, pp. 847-862, 1997.
Ando, M. How much should trust rearesesooon-kiinndeessiee estimates? Empirical Economics, 53:1287-1322, 2017.
Carroll, M. D., Dragan, A., Russell, S., and Hadfield-Menell. D. Estimating and penalizing induced preference shifts in recommender systems. International Conference on Machine Learning, pp. 2686-2708. PMLR, 2022.
Artzrouni, M. On the growth of infinite products of slowly varying primitive matrices. Linear Algebra and its Appli- cations, 145:33-57, 1991.
Chen, L., Zaharia, M., and Zou, J. How is chatgpt's behavior changing over time? arXiv preprint arXiv:2307.09000 000009999999 2023.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedhack. arXiv preprint arXiv:2204.0558 058662, 2022.
Cinelli, M., De Francisci Morales, G., Galeazzi, A., Quat- trociocchi, W., and Starnini, M. The echo chamber effect social media. Proceedings of the National Academy of Sciences, 11809):2023301111 2021.
Bessi, A., Zollo, F., Del Vicario, M., Puliga, M., Scala, A., Caldarelli, G., Uzzi, B., and Quattrociocchi, W. Users polarization on facebook and youtube. PloS one, 11(8): e0159641, 2016.
Conneau, A. and Lample, G. Cross-lingus language model pretraining. Advances in neural information processing systems, 32, 2019.
Bird, S. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pp. 69-72, 2006.
Costello, T. H., Pennycook, G., and Rand, D. G. Durably reducing conspiracy beliefs through dialogues with ai. Science, (55(6714))eadg) 1814, 2024.
Borah, A., Houalla, M., and Mihalcea, R. Mind the (belief) gap: Group identity in the world of lims, arXiv preprint arXiv:2503.02011 02016 2025.
Cribari-Neto, F. and da Silva, W. B. A new heteroskeddasiiitty-conssss covariance matrix estima- tor for the linear regression model. AStA Advances in Statistical Analysis, 95:129-1466 2011.
Boutyline, A. and Willer, R. The social structure of political echo chambers: Variation in ideological homophily in online networks. Political psychology, 38(3):551-566 2017.
Danry, V., Pataranutaporn, P., Groh, M., Epstein, Z., and Maes, P. Deceptive ai systems that give explana- tions are more convincing than honest ai systems and amplify belief in misinformation. arXiv preprint arXiv:2408. 00024, 2024.
Breusch, T. and Pagan, A. R. A simple test for het- eroscedasticity and random coefficient variation. Econo- metrica: Journal of the econometric society, pp. 1287- 1294, 1979.
Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.0786; 2024.
Bright, J., Marchal, N., Ganesh, B., and Rudinac, S. Echo chambers (bust!(but they'r re full of opposing views). arXiv preprint arXiv:2001.1144 1146,, 2020.
Brown, M. A., Bisbee, J., Lai, A., Bonneau, R., Nagler, J., and Tucker, J. A. Echo chambers, rabbit holes, and algorithmic bias: How youtube recommends content to real users. Available at SSRN 41 14905, 2022.
Driscoll, C. and Kraay, A. C. Consistent covariance matrix estimation with spatially dependent panel data. Review of economics and statistics, 80(4):549-560, 1998.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle. A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Bruns, A. Echo chamber? what echo chamber? review- ing the evidence. In 6th Biennial Future of Journalism Conference (FOJ17), 2017.
10

---

The Lock-in Hypothesis: Stagnation by Algorithm
Dubois, E. and Blank, G. The echo chamber is overstated: the moderating effect of political interest and diverse media. Information, communication & society, 21(5): 729 729-745,201 2018.
Helberger, N., Araujo, T., and de Vreese, C. H. Who is the fairest of them all? public attitudes and expectations regarding automated decision-making, Computer Law & Security Review, 39:105456, 2020.
Fisher, J., Feng, S., Aron, R., Richardson, T., Choi, Y., Fisher, D. W., Pan, J., Tsvetkov, Y., and Reinecke, K. Biased can influence political decision-making arXiv preprint arXiv:2410.06415 2024.
Hendrycks, D. and Mazeika, M. X-risk analysis for ai research. arXiv preprint arXiv:2206.0588 05866.0582211 2022.
Hobolt, S. B., Lawall, K., and Tilley, J. The polarizing effect of partisan echo chambers. American Political Science Review, 18(3):146444477) 2024.
Gabriel, I. and Ghazavi, V. The challenge of value align- ment: From fairer algorithms to ai safety, arXiv preprint arXiv:2101.06660, 2021.
Hosseinmardi, H., Ghasemian, A., Rivera-Lanas, M., Horta Ribeiro, M., West, R., and Watts, D. J. Causally estimating the effect of youtube's recommender system using counterfactual bots. Proceedings of the National Academy of Sciences, 121(8))e233337122) 2024.
Ge, L., Halpern, D., Micha, E., Procaccia, A. D., Shapira, I, Voroheychik, Y., and Wu, J. Axioms for ai alignment from human feedback. arXiv preprint 2xXv:2445..1755, 2024.
Imbens, G. W. and Rubin, D. B. Causal inference in statis- fics, social, and biomedical sciences. Cambridge univer- sity press, 2015.
Gillani, N., Yuan, A., Saveski, M., Vosoughi, S., and Roy, D. Me, my echo chamber, and i: introspection on social media polarization. In Proceedings of the 2018 World Wide Web Conference, pp. 823 033-831,22 2018.
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L.,, and Naaman, M. Co-writing with opinionated language mod- els affects users' views. In Proceedings of the 2023 CHI conference on human factors in computing systems, pp. 1-15, 2023.
Glickman, M. and Sharot, T. How human-ai feedback loops alter human perceptual, emotional and social judgements. Nature Human Behaviour, pp. 1-15, 2024.
Jin, Z.., Kleiman-Weent M., Piatti, G., Levine, S., Liu, J., Gonzalez, F., Ortu, F,, Strausz, A., Sachan, M., Mihalcea, R., al. Language model aligmment in multilingual trolley problems. arXiv preprint arXiv:2407.02273, 2024.
Griffiths, L. and Kalish, M. L. Language evolution by iterated learning with bayesian agents. Cognitive science, 31(3):441-44 2007.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint 1XXx:2501.11948, 2025.
Jost, L. Entropy and diversity. Oikos, 113(2))333-375 2006.
Kalimeris, D., Bhagat, S., Kalyanaraman, S., and Weinsberg, U. Preference amplification in recommender systems. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 805-815, 2021.
Guo, Z., Schlichtkrull, M., and Vlachos, A. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178-200 2022.
Hackenburg, K., Tappin, B. M., Röttger, P., Hale, S., Bright, J., and Margetts, H. Evidence of log scaling law for political persuasion with large language models. arXiv preprint arXiv:2406. 145008, 2024.
Kaminskas, M. and Bridge, D. Diversity, serendipity, novelty, and coverage: a survey and empirical analy- sis of beyond-accuracy objectives recommender sys- tems. ACM Transactions on Interactive Intelligent Sys- tems (TūS), 7(1):1-42. 111-42,21116 2016.
Hall, M., van der Maaten, L.,, Gustafson, I.., Jones, M., and Adeock, A. systematic study of bias amplification. arXiv preprint arXiv:2201.117000 2022.
Kirby, S., Griffiths, T., and Smith, K. Iterated learning and the evolution of language. Current opinion in neurobiol- ogy, 28: 108-008110 1 14, 2014.
Hausman, C. and Rapson, D. S. Regression discontinuity in time: Considerations for empirical applications. Annual Review of Resource Economics, 10(1):533-552, 2018.
Kosub, S. A note on the triangle inequality for the jaccard distance. Pattern Recognition Letters, 120:36-38. 2019.
Hazrati, N. and Ricci, F. Recommender systems effect on the evolution of users' choices distribution. Information Processing & Management, 59(1):00276,, 2022.
Leib, M., Köbis, N. C., Rilke, R. M., Hagens, M., and Irlenbusch, B. The corruptive force of ai aingeeeedted advice. arXiv preprint arX0v:2102.07536, 2021.

---

The Lock-in Hypothesis: Stagnation by Algorithm
Li, W., Cui, L.-B., and Ng. M. K. The perturbation bound for the perron vector of transition probability tensor. Numerical Linear Algebra with Applications, 20(6):985- 1000, 2013.
Qiu, T., Zhang, Y., Huang, X., Li, J. X., Ji, J., and Yang, Y. Progressgym: Alignment with millennium of moral progress, arXiv preprint arXiv:2406.
2024.
Ren, Y., Guo, S., Qiu, L., Wang, B., and Sutherland, D. J. Bias amplification in language model evolution: An iter- ated learning perspective. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Liang, W., Izzo, Z., Zhang. Y., Lepp, H., Cao, H., Zhao, X., Chen, L.., Ye, H.. Liu, S., Huang. Z., et al. Monitoring ai-modified content at scale: A case study on the impact of chatgpt on conference peer reviews, arXiv preprint wrXiv:2403.07183, 2024.
Rosopa, P. J., Schaffer, M. M., and Schroeder, A. N. Man- aging heteroscedasticity in general linear models. Psy- chological methods, 18(3):335, 2013.
Luzsa, R. A Psychological and Empirical Investigation of the Online Echo Chamber Phenomenon. PhD thesis, Universität Passau, 2019.
Salvi, F., Ribeiro, M. H., Gallotti, R., and West, R. On the conversational persuasiveness of large language mod- els: randomized controlled trial. arXiv preprint arXiv:2403. 1ttt22443..48800 14380, 2024.
Mansoury, M., Abdollahpouri, H., Pechenizkiy, M., Mobasher, B., and Burke, R. Feedback loop and bias amplification in recommender systems. In Proceedings of the 29th ACM international conference on information & knowledge management, pp. 2112-2142224 2020.
Sanderson, M. and Croft, B. Deriving concept hierarchies from text. In Proceedings of the 22nd annual interna- tional ACM SIGIR conference on Research and develop- ment information retrieval, pp. 206-213, 1999.
McInnes, L., Healy, J., Astels, S., al. hdbscan: Hierar- chical density based clustering. J. Open Source Softw., 2 (11):205, 2017.
Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., and Hashimoto, T. Whose opinions do language mod- cls reflect? In International Conference Machine Learning, pp. 29971-30004. PMLR, 2023.
McInnes, L.,, Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint urXiv::802.000
2018.
Sharma, N., Liao, Q. V., and Xiao, Z. Generative echo chamber? effect of llm-powered search systems on di- verse information seeking. Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-17, 2024.
Mendler-Dinner, C., Carovano, G., and Hardt, M. An engine not a camera: Measuring performative power of online search. arXiv preprint arXiv:2405.19073 2024.
Miller, G. Wordnet: lexical database for english. Com- munications of the ACM, 38(11):39 :33941,4 1995.
Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Ander- son, R., and Gal, Y. Ai models collapse when trained on recursively generated data. Nature, 63 (8022))))5557 2024.
Padmakumar, V. and He, H. Does writing with lan- guage models reduce content diversity? arXiv preprint 1.Xiv:2309.05196 2023.
Su, Q. and Wu, Y.-.. On convergence conditions of gaus- sian belief propagation. IEEE Transactions Signal Processing, 63(5): 1155, 2015.
Pan, A., Jones, E., Jagadeesan, M., and Steinhardt, J. Feed- back loops with language models drive in in-context reward hacking, arXiv preprint arXiv:2402.06627, 2024.
Tamkin, A., McCain, M., Handa, K.. Durmus, E., Lovitt, L.,
Peterson, A. J. Ai and the problem of knowledge collapse. arXiv preprint arXiv:2404.03502, 2024.
Rathi, A., Huang, S., Mountfield, A., Hong, J., Ritchie, S., et al. Clio: Privacy- insights into real-world ai use. arXiv preprint arXiv:2412.13672 2024.
Piccardi, T., Saveski, M., Jia, C., Hancock, J. T.. Tsai, J. L., and Bernstein, M. Social media algorithms can shape affective polarization via exposure to antidemo- cratic attitudes and partisan animosity. arXiv preprint arXiv:2411.1 14652, 2024.
Taori, R. and Hashimoto, T. Data feedback loops: Model- driven amplification of dataset biases. In International Conference on Machine Learning, pp. 33883-33920. PMLR, 2023.
Potter, Y., Lai, S., Kim, J., Evans, J., and Song, D. Hidden persuaders: Lims' political leaning and their influence on voters. arXiv preprint arXiv:2410.244 24.24949 2024.
Terren, L. T. L. and Borge-Bravo, R. B.-B. R. Echo cham- bers on social media: A systematic review the literature. Review of Communication Research, 9, 2021.
12

---

The Lock-in Hypothesis: Stagnation by Algorithm
Tseng, Y.-M., Huang, Y.C.,, Hsiao, T.-Y,, Chen, W.-L., Huang, C.-W., Meng, Y., and Chen, Y.-N. Two tales of persona in Ilms: A survey of role-playing and personal- ization. arXiv preprint arXiv:2406.0117,, 2024.
Vincent, B. T., Forde, N., and Preszler, J., Aug 2024. URL httph:tt///cm. http:://githubt www.c//////o..c.. pymc- labs/ CausalPy/
Williams, M., Carroll, M., Narang, A., Weisser, C., Mur- phy, B., and Dragan, A. On targeted manipulation and deception when optimizing Ilms for user feedback. arXiv preprint ArXiv:4411.02366 2024.
Wittgenstein, L.. Philosophical investigations. John Wiley & Sons, 2009.
Wolfowicz, M.. Weisburd, D., and Hasisi, B. Examining the interactive effects the filter bubble and the echo chamber on radicalization. Journal of Experimental Crim- inology, 19(1::19-141, 2023.
Yakura, H., Lopez-Lopez, E., Brinkmann, L., Serna, I., Gupta, P., and Rahwan, I. Empirical evidence of large language model's influence on human spoken communi- cation. arXiv preprint arXiv:2409.01754, 2024.
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470, 2024.
Zhao, W. X., Zhou, K., J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
Zhou, F. Xu, X., Trajcevski, G., and Zhang, K. A survey of information cascade analysis: Models, predictions, and recent advances. ACM Computing Surveys (CSUR), 54 (2):1-36, 2021.
13

---

The Lock-in Hypothesis: Stagnation by Algorithm
A. Limitations and Future Work
RCTs with Human Subjects Notably, our analysis on WildChat has not managed to nullify all potential confounders due to their prevalence in time series data. We found that the launches of new versions cause concept diversity loss that is statistically significant, but we aren't 't yet certain why there are such drops.
We believe AI labs that have their own chatbots deployed are well-positioned to establish better causal evidence of feedback loop induced diversity loss at larger scale, with randomization designs and longer temporal user interaction data (Tamkin ct al. 2024).
For researchers outside Al labs, establishing better quality of causal evidence is not inconceivable: for example, an I..M-powered browser extension that alters chathot outputs could enable randomized human subject experiments and remove confounders that are prevalent in observational studies (Piccardi al. 2024 Mendler-Dünne et al.) 2024). These experiments could test the causal effects of feedback loops users and potential interventions address them.
Lock-in Effects in the Wild Alongside randomized controlled experiments aiming to establish causal evidence, also need better evidence of lock-in effects in the wild. For instance, science may go down the path favored by LLMs: in theory both authors and reviewers may utilize LLMs in their workflow, which may create an echo chamber that entrenches LLM biases. Besides, fields with heavier use of LLMs may establish feedback loops that already demonstrate lock-in effects Yakura et al. 2024). We will need strong empirical demonstrations of such effects.
More realistic simulations We think simulations play complementary roles in understanding the LLM's impact on human's belief evolution over long term. But we hope simulations can be progressively realistic in order for us to understand the real-world interaction dynamic. Among other things, we are particularly interested in understanding how group of Agents may update their beliefs when both LLM Authority and new empirical evidence presented. This is because for lock-in to happen, given population of humans would need to, not only hold their (false) beliefs firmly, but also it's hard for them to incorporate empirical evidence.
That said, we believe that these evidences provide sufficient reasons for investigating human-LLMy interactions and lock-in hypothesis further and for developing strategies to mitigate the potential negative consequences of lock-in, either through technical and algorithmic interventions, or through policy and regulation.
Evaluation and Mitigation The ability to monitor lock-in effects in real-world human- systems an important enabler of successful mitigation strategies. This may be perfect measurement based agent beliefs downstream proxy; small-scoped metric focusing one-human-one-Al interactions societal-scale metric.
Once such evaluation methods are in place, both algorithmic and policy mitigation will become much more tractable. On the policy and governance front, foundational research to clarify the range of feasible interventions is needed. On the algorithmic front, methods for optimizing model policies within a complex human-Al environment may be needed. Current alignment methods see human feedback as non-influenceable oracle (Bai al. 2022 Carroll et al., 2024), which make them unsuited for the job.
14

---

The Lock-in Hypothesis: Stagnation by Algorithm
(1)
(2)
(3)
(4) Depth Value
(5)
(6)
Variable
Lin. Non -impT (Unsculed)
Lin. Value (Unscaled)
Entrupy Value
Jaccard Value
(Unscaled)
Coefficients
const
670,476
-391.123
-1859.673***
20.806
5.261 ""
5.084***
(r 0:906)
(p - 0.928)
(p <, 0.001)
(p < 0.001)
(p < 0.001)
(p < 0.001)
num__inks_hefiin
-176.570
11777714
-0060811
.0.01
-0.16''
-0.101
(p 0.495)
(p = 0.353)
- 0.070)
(p 0.983)
(p 0.035)
(p 0.030)
nsamples
0.190
0,066
-0.024
0.000
0.000
0.000
(p 0.437)
(j = 0.459)
(p m 0.584)
(p 0.003)
(p < 0.001)
(p 0.005)
time
-6,633
-7,672
-1.804
-0.002
-0,008
-0
(p = 0.202)
(p = 0.144)
(p = 0.281)
(p = 0.065)
(p < 0.001)
(p < 0.001)
user first eistyentry
3.406
3.187
0.728
0.002
0.003
0.001
(x = 0.550)
(F 0.358)
(p = 0.671)
(p 0.072)
(p 0.134)
(p 0.731)
engugerment proogresent-poggress
-0,092
0,059
0.001
-0.000
0.000
0.000
(p = 0.752)
(p = 0.625)
= 0.984)
(p = 0.404)
(p = 0.267)
(p 0.476)
temporal.cxteme exxtensio
11.219
3.650
4.412
-0.001
-0.018***
-0.009
(p = 0.323)
(p = 0.610)
(p = 0.323)
(p 0.631)
[p < 0.001)
(p 0.058)
user sser.gpt35.ratin ratioo
1619,255
-202.877
-329.406**
0,050
-0,303*
-0.486**
(p 0.515)
(p = 0.392)
(p = 0.022)
= 0.530)
(p = 0.010)
(p 0.001)
mean_rurns
46,235
29.407
63.585
-0.044"
-0.031
0.015
(r 0.463)
0.466)
0.034)
(p 0.003)
0.264)
(p 0.578)
mean aaaan_comersation_eengo
0.069
0.032
-0.045
0.000
0.000
0.000
(p - 0.385)
(p m 0.429)
- 0.082)
(p < 0.001)
(p - 0.013)
(p 0.000)
mean prompt length
.0027
-0.019
-0.003
-0.000
-0.000"
-0.000
0.832)
0.728)
0.936)
(p < 0.001)
0.041)
(p 0.126)
post gap
-51.407
-17.078
0,154
0.128
0.070
(x = 0.815)
(p = 0.809)
(p = 0.003)
(p = 0.034)
(p 0.212)
language, Chinese
-1087.495
-867.631
0.346
-1.048
-1.193
(p 0.831)
0.838)
(p 0.732)
0.274)
(P 0.188)
language.Dutch
-209.98
124.679
1828.628**
1.226
-0.403
-0.462
(p = 0.967)
(p = 0.977)
(p < 0.001)
(p = 0.211)
[p = 0.715)
(p 0.661)
language.Engling English
-238.263
-53.230
1712.809***
0,742
-0.441
-0.637
(p 0.963)
(p = 0.990)
< 0.001)
0.163)
(p 0.615)
0,481)
language.Frende
-708.460
-199,764
1478,910**
0.732
-0,906
-0.904
(p = 0.890)
(p = 0.963)
(e 0.001)
(p = 0.112)
[p = 0.357)
(p 0,333)
-73.373
125.137
2262.289**
1,020
.0.400
-1.118
(p 0.980)
(p - 0.977)
(p - 0.021)
0.327)
(p 0.443)
(p 0.285)
language_Indaann Indonesian
430.232
1684.034**
L.005
-1.068
-0.879
(p = 0.921)
(p = 0.001)
(p 0.344)
[p - 0.338)
(p 0.411)
language Italian
67.561
220.081
2178.0R5***
1,140
0.452
0.279
(p = 0.990)
(p = 0.961)
( = 0.001)
(p 0.293)
(p = 0.699)
(p 0.803)
language Japanese
-396.516
-265.645
772.865
0.655
-1.145
-1.126
(p = 0.938)
(p = 0.951)
(p = 0.103)
(p 0.533)
[p 0.300)
(p 0.200)
language Portuguese
226.901
256.649
2020.447***
0,766
-0,782
-0.714
(p 0.965)
(p = 0.952)
(p < 0.001)
(p 0.453)
(p = 0.433)
(p 0.451)
language_ Russian
1288.772
-942.366
1217.315**
0,306
-0.972
-0.918
(p 0.800)
(p = 0.821)
(p < 0.001)
(p 0. (63)
[p 0.313)
(p 0.314)
language
language.Spanish Spanish
68.414
53,543
1620.475***
0.724
-0.506
-0.646
(p = 0.989)
(p = 0.990)
(p < 0.001)
(p = 0.117)
(p = 0.608)
0.189)
language.Turkee Turkish
109669100
(p = 0.078)
Model Summary
No. Observations
4571
7261
3556
7261
4081
3979
No. Groups
197
272
181
272
227
227
Log: Likelihood
.45350.28
-70814.14
-29123.54
-10284.17
50100.56
4536.16 16
Scale
2.54e+07
1.77e407
7.46e+05
0.945
0.597
0.495
Group Var
2.55e+05
2.08e+05
1.85+005
0.071
0.298
0.304
Converged
Yes
Yes
Yes
Yes
Yes
Yes
Note: Signihcaace levels: "p=c0.05, "'pc0.01, "'pp-0.001.
Table 2. Mixed linear model regression results for different diversity metrics. Models: (1) Lineage Diversity (all, unscaled), (2) Lineage Diversity (non-templated, unscaled), (3) Lineage Diversity (value-laden, unscaled), (4) Depth Diversity (value-laden), (5) Topic Entropy Vvalue-laden), (6) Jaccard Distance (value-laden), P-values in parentheses.
15

---

The Lock-in Hypothesis: Stagnation by Algorithm
Diversity Time Hppr 2023 to 2024)
Agr 2024)
GPT4 Gomootedd lvverity careé)
0,835
GPT4 is Isnosthed fiverity curvel
CPTA ( (inear regrisa ont
G7T4 (teer rog guarrs
OPT8 versin updite
GP1d version Up
0,630
0.555
Lyncopnu Liversy nn..n Masaaaa
0.623
0.550
8.00
0.300
0.4
0.459
STTT.---tuzee iffrcttedd diveritty curvel
Surto diversity curne)
legression)
120
150
200
200
200
Fime Ecayys)
Tame (days)
(6)
(b)
Figure 6. Observed conceptual diversity trends in messages. (a) On the subset of value-laden concepts, interactions with GPT-4 show a downward trend (p < .05), while those with GPT-3.5-turbo show an upward trend (p < .05). (b) On all concepts collectively, interactions with both models show downward trends .05). See Figure 7 for more setups. This comparison highlights the ambiguity on the resolution of Hypothesis 1.
B. Supplementary Results of WildChat Data Analysis
In this appendix, we explain details on the implementation and results of the data analysis on WildChat.
Some figures this appendix, as well as Figure in the main body, are made with CausalPy (Vincent et al., 2024).
B.1. Results Without Data Filtering
Figure 1) presents results of hypothesis testing on the complete dataset without filtering suspected API use. After including such data, support for both Hypothesis and Hypothesis 2 are strengthened.
B.2. Exploratory Hypotheses
Exploratory Hypothesis (Individual Diversity Loss Occurs in Human-Al Interaction). Among heavy users of GPT, those who use it more (compared to those who used it less) experience a stronger diversity loss in the concepts occurring in their conversations with GPT, after adjusting for confounders. Moreover, on the temporal dimension, the accumulation of GPT interaction causally explains each heavy user's diversity loss over time.
Exploratory Hypothesis (Iterative Training Leads to Individual Diversity Loss). Among heavy users of GPT, those who engage with later versions GPT experience a larger diversity loss, after adjusting for confounders
be noted that Hypothesis 3 and 4 adopt individual users as the unit of analysis, while Hypothesis and adopt time periods as the unit and aggregate all users into collective corpus.
B.3. Methods
In this section, we explain details on our method for analysis.
Concept Hierarchy To assist in the assessment of concept diversity, we build concept hierarchy (Sanderson & Croft 1999) with the following steps:
1. Extracting concepts (e.g., computer, environmental protection, world cup) mentioned or implied in each conversation, with the
model (Dubey et al. 2024).
*The reason for focusing on heavy users is that self-selection bias is strong among light users, where people who find GPT less useful spontancously engage less with it, confounding the data with reversed direction of causality. 'Hypothesis shows that "human - L.LM" direction of influence also has an impact on diversity loss.
16

---

The Lock-in Hypothesis: Stagnation by Algorithm
Payesion dxD- 0000
Bayedan RP an 6.38 (e2 6.855)
Rayetton dats 3 jed 3.00
Change gradent 4.6043.
Cherl- -- 5008, ...1311
Cleof-0.200 -6.20067)
Olee 8620 0 0.00027)
49515
0335
3325
0553
0063
-22
100
124
22
100
140
(12)
Sayecion R7 on data, (11 0G3)
Sayeslan Rf on 431a 5:16 GU 0.043)
aoses 8f on a1 cita 0.46 GAE 055)
Churge 3.0028
Clusd d 2004 a 0026)
Charge gradint - 2014
Change in grokent w 3303334
0.25
0:50
257
0.500
0435
(2b)
(20)
(20)
Saysslan of on dats 3. ots + 0.043)
Eo
Buyssian
Charge in gradier 8.225
8.8010)
Change gradient 2,0325
Clual
Change gridirt 0021.
22.59
22.0
11.
21.
23.7
21.7
21:07
22.
21.5
215
21.
120
141
160
(10)
(3c)
(34)
Buysoinn If on data 0 bad 0.033
of
Charge in prodict 433,
2094)
Charge
3,022
Change g
330
320
343
164
(4d)
sayeslan of all data 9.4 (51 = 0.251
Bayostan uoullct
Change in gredient 4 ONB.
Charge
Gye D41 000024)
Gueu E31,
55
355
100
1.0 393
123
342
[Sa]
[5c]
[54)
Figure 7. Sensitivity analysis to the choice of diversity metric. (I)) Analysis in Figure b)a) and Figure 5 replicated on the entire WildChat corpus without filtering. (2a))2)(2+)2d)) Analysis replicated on post-filtering WildChat, where templated messages sharing 75- 75-character prefixes with more than 75 other messages are removed. Non-value-lad -ldenn messages remain. (3a)(3))(3c)33)) Analysis sis replicated on the diversity metric Daeph. (44)(4)((c)(4d) Analysis replicated on topic entropy as diversity metric, where topics are defined as maximal clusters in the concept hierarchy containing no more than 1% of all concepts. (5)))5)((c)(d)) Analysis replicated on average pairwise Jaccard distance hetween conversations as a diversity metric, where each conversation is represented with the set topics it contain.
17

---

The Lock-in Hypothesis: Stagnation by Algorithm
2. Perform lemmatization on the concepts with the WordNet Lemmatizer (Miller 1995 Bird 2006).
3. Obtain D = 256-dimensional embeddings for each concept using the 1oyage-3-large model.
4. Perform hierarchical clusterization with the HDBSCAN algorithm (MeInnes et al. 2017).
This would produce tree with specific concepts at the bottom, and generic concept clusters at the top. The root node an all-enncomaasing cluster that captures all concepts.
Given the size of the hierarchy, please refer to our codebase to view its content. The README instructions shall contain guidance where to find the hierarachy illustration.
Diversity Metric
Both hypotheses require the measurement of concept diversity within a certain user or corpus. Common metrics for measure diversity - such as Shannon entropy - cannot take into account the hierarchical structure of concepts, and may therefore view semantically similar concepts as entirely different ones. To overcome this shortcoming, we introduce the lineage diversity metric, which, for each multi-set C of concepts, calculates
1)
1
Dtincage (rinaage(C;] (C:T)-
log [7] - log Unifte))
log [7]
where ((u,2) u) is the lowest common ancestor (LCA) of concept nodes and v, |T.] is its subtree size (its number of descendant concepts), and [T] is the size of the tree.
Computation of Lineage Diversity Finally, the calculation of D Duinaage (C; T) can be dramatically accelerated by performing dynamic programming on the compressed Steiner tree containing the nodes C, resulting in the time complexity O [C] log (T) (ICI < (7), as opposed to @([C)² log (T]) or @([T)) alternative algorithms. is also much faster than traditional distance- based metrics that typically require D((c)D)) time to compute (Kaminskas & Bridge, 2016) an unaffordable complexity our scale of analysis.
Multiple Regression and Heteroscedasticity When testing Hypotheses 1, 3 and 4, is the standard practice in causal inference real-world observational data (Imbens & Rubin 2015). we adopt multiple linear regression with heteroscedassicityyroooe standard errors (Rosopa et al. 2013). We carry out the Breusch- -aggan test (Breusch & Pagan 1979) for heteroscedasticity, and upon positive result, use the HC3 and the Driscoll & Kraay standard error (Cribari-Neto & da Silva 2011: Driscoll & Kraay, 1998), given the heavy-tailed nature of user interaction statistics and therefore the potential for high leverage.
We control for the demographics variables recorded in WildChat-IM, MM, namely user language and geographic location, and other interaction characteristics such date of first GPT use, average conversation length, and GPT-3.5-turke GPT-4 usage rate.
Regression Kink Design Hypothesis 2 revolves around a non-smooth change in diversity at the dates of GPT version switch, meaning that the first-order derivative of the diversity curve is discontinuous at those points. Since confounders almost always change smoothly, any discontinuous change detected will be indicative of causal relationships, even without adjusting for confounders. We thus adopt the regression kink design (RKD) (Card et al. 2017 Ando (017) where two polynomial regressions are performed at the left and right limit of the switching date, and statistical tests are deployed to detect coefficient differences between the two polynomials.
B.4. Exploratory Analysis on User Engagement
We carry out regression analysis and visualizations on Exploratory Hypothesis and whose results are shown in Figure 8 It is found that diversity tend to decrease with engagement for high-engagement users, while the trend is opposite for low-engagement users.
Self-selection bias is a likely confounder here. Users who find GPT's responses less diverse and less helpful tend to stop engaging with it (or engage less), thereby reversing the direction of causality. We believe it is a likely explanation for the reversed trends among low-engagement users (since voluntary drop-out is especially common among low-engagement users), along with the factor that new users tend to discover more use cases of GPT over time.
18

---

The Lock-in Hypothesis: Stagnation by Algorithm
(1a)
(1b)
(1c)
2.44
revoomant
Sumber Commeration Peniicaned
(2a)
(2b)
(2c)
Figure Diversity a user's conversation changes with level of user engagement. (1a)(2a) Relation between per-user concept diversity and absolute engagement (number of conversations), under the metrics Diinage and Daeph respectively. (Ib)(2b) Relation between per-user-per settrr-ee--time-eeee time-period concept diversity and absolute engagement (number of conversations already had, divided by total number of conversations of the user). (1c)(e)) Relation between per-user-per weweerr-r--tieepeeeeeeee concept diversity and absolute engagement, within the top 1 5% high-engagement users specifically.
19

---

The Lock-in Hypothesis: Stagnation by Algorithm
-0.1
(313 Sound Shar HOMIRRE Stubeerr
4.50
-441
6730
-.54
800
0:11
4.50
Crodle Gord
Caddder kge:
Convectie Qual
(1a)
(1b)
(1c)
(1d)
Deveritty wer Time Mer 2003 2704
FINT
PENT
405
Cri2crrtt wwrasteeeeeecccc evento gevil 2200
15
100
(2a)
(2b)
(2c)
(2d)
Figure 9. Additional results. (1a) Diversity trends of GPT-3.5-turbo and GPT-4 conversations combined (red), compared to GPT-3.5-turbo trends (blue). See $5.4 for discussion on confounders. (1b)(1)((dd) Interrupted time series (ITS) analysis on the causal impact releasing SEPT-4-0125-preview GPT:3.5-turb--0113, GPT-3355turbo-011 respectively, with time horizon truncated to 120 days. (2))(2)((c)(2d) Analysis performed with Dasn the diversity metric.
20

---

The Lock-in Hypothesis: Stagnation by Algorithm
B.5. Sensitivity Analysis the Choice of Metric
We independently developed Deepth an alternative diversity metric Drineage but which was later abandoned in favor of Diineage due to its lack of interpretability. Despite that, Depph serves as a valuable tool for sensitivity analysis, where validate the main conclusions of the analysis against with Deepth the diveristy metric.
Below, we introduce D depth and present the results of the analysis on depii.
D depth (C; T,r) = Eu,o~U(e) [log [T((,))) d(r, ((,,) w))]
(10)
Here, log [T)l estimates the height of the subtree, and by subtracting from it the distance the root, obtain measure of [(, e)'s relative vertical position on a leaf-to-rool path. It equals zero when 7(u, v) is at the exact middle, becomes larger the higher ((u, v) is on the path, and vice versa. This two-part design is meant for maintaining fairness in possibly imbalanced tree, where certain parts the concept space is over-represented and has a enlarged subtree compared to others.
By calculating the expected vertical position of the LCA of two uniformly random concepts from the corpus, we measure the extent of dispersal of the corpus over the concept hierarchy whether they are concentrated in a small niche, or spread across diverse range of different topies.
B.6. Implementation Details and Prompts
This section contains supplementary details on the implementation our analysis.
Prompt for filtering value-laden concepts Below is the LLM prompt with which identify concepts that value laden. These then used to filter the WildChat dataset.
1 We define the -alde--adenness of phrase as the extent to which it conveys e opinions on ethics, politics, ideclogy, or religion, or is otherwise related e to these topics.
3 Given the following collection of phrases, sort them in decreasing order of valuet -ladenness, and return the sorted phrases in the EXACT same JSON format. e Output only the JSON object (without wrappers) and nething else.
5 !phrases)
Prompt for concept extraction Below is the LLM prompt with which extract concepts from conversations.
You are given conversation in JSON format. Each element of the JSON array is e dictionary (object) that represents a single turn in the conversation. Each ( dictionary has two keys:
1. **role.* which can be "user" or "assistant"
3 2. **content.. the text content of that turn
Your task:
1 1. Read through each turn in the conversation.
8 2. Extract all **related concepts** from each turn's content. A concept is t defined as any short phrase (one to a few words) that represents idea, + topic, event, name, domain-specific term mentioned in that turn. **ammpless es of potential concepts** might include specific events ("Trump Inauguration") general topics ("climate change"), ideas ("economic resiliencet "), niche terms ("public-benefit corporation"), or anything else that stands t out.
**Have low bar for inclusionsx; if something seems relevant, it be included as concept.
11 3. Separate the extracted concepts based on whether they came from the * st mossages or the **sssisantt**'s messages.
4. Return JSON dict containing four lists: asser_coneept__epplicti
all unique concepts explicitly mentioned in the user's g messages (as short strings)
21

---

The Lock-in Hypothesis: Stagnation by Algorithm
"essstaattcooooppss_elllllll all unique concepts explicitly mentioned in the assistant's messages (as short strings)
15
baber_concepa_reeattee all unique concepts related to the user' s messages + (as short strings)
pecisstat_ccncepttt rettsse_rraeed all unique concepts related the assistant messages (as short strings)
17
18 Important details:
If the user the assistant repeats concept multiple times, list it only
once in the respective output array.
20 Rach concept should be e*distinct short phrase**.
Cutput e*only** the JSON object and nothing else (no extra commentary or + explanation).
Always give concepts in English, if the conversation is in another language.
23
AB Input
jjsn
28
"content": "What is reality shifting?",
"role": "user"
32
33
"content": "Reality shi shifting is a practice or belief in which individuals + attempt to shift their consciousness awareness to enter and experience + alternate realities or dimensions. individuals have the ability to consciously control their experiences and t perceive different realities. Reality shifting can involve various techniques, ( such as guided meditations, visualization exercises, or specific rituals, e with the alm of accessing different planes of existence, fictional worlds, desired scenarios. Some individuals perceive reality shifting as means to escape or explore possibilitics beyond their current physical reality. Howevert it is important to note that reality shifting is generally considered a t subjective experience and is not scientifically proven.", "role": "asaistant.
often associated with the idea that
t
36 "content"; "Has government ever studied reality shifting?", "role": "user"
40
"There is no evidence or official documentation to suggest that e any government has conducted serious official studies reality shifting. e The concept reality shifting lies within the realm of spirituality, metaphysics, and personal belief systems rather than seientific inquiry or government research. Governments typically prioritize funding and conducting t studies related to more tangible and empirically verifiable subjects, such as + medicine, technology, economics, and social sciences. Consequently, reality shifting remains largely within the sphere of personal exploration and subjective experiences rather than being subject to government. study.", "role": "assistant""
"content";
42
...
46
$ AAF Output 1 (Concepts in English, Near-Repstitive Concepts Removed)
ison
58
peter_connept__xppiicceee ["reality shifting", "government"")
aeser_concett__relatttt [*conspiracy theory", "government research", "
spirituality", "supernatural phenemenon"),
"assistant citeiiaatttcccccts_____e exxcepts_explicit": (
("reality shifting", "consciousness",
22

---

The Lock-in Hypothesis: Stagnation by Algorithm
awareness", "alternate reality", "alternate dimension", "reditation", "guided meditation", "visualization exercise", "ritual", "plane of existence", "t fictional world", "physical reality", "subjective experience", "scientific e proof", "official documentation", "spirituality", "metaphysics", "personal e belief system", "scientific inquiry", "government research", "government funding", "medicine", "technology", "economics", "social science", "subjectiver> expericnce", "government study"I, crrprtattt_ccccpppeeeetttttote
("pseudoscience", "conspiracy theory", "t scientific skepticism", "supernatural phenomenon", "esctericism", "philosephy consciousness", "parapsycholcgy")
55
56
### Input
60
[Non-Ern Eenn-En Example Input]
|# Output 2 (Concepts in English, Near-Repstitive Concepts Removed)
json
65
"user_cov cel_ccocpts_epplccciiiee !"prenatal checkup", "Chaoyang District Maternal e and Chila Health Hospital"),
["hospital policies", "maternal health services", "t medical advice", "childbirth", "pregnancy", "motherhced", "Beifing healthcare t system", "public health services"), "assistare ctnssstttttcccceps__eeellll compl iopoliicatioss"" "abnormal conditions", "treatment plan", "personal informatione "family medical history", "genetic disorders", "pregnancy history", "blood e test", "urine test", "liver and kidney function", "bloed type", " elect rocardioaaa) "ultrasound", "B-mode ultrasound", "fasting test", "p comfortable clothing", "low bloed sugar", "emotional wellbeing", "birth safetyt "national ID card", "marriage certificate", "prenatal record booklet", "P health insurance card", "contact information", "Chaoyang District Maternal and Child Health Hospital", "Chaoyang District"), "assistanttcooceppe" prartted__radd ("maternal care", "prenatal health", "early + pregnancy", "hospital-sseciifi requirenents", "child health assessment", " preventive measure", "Beijing", "regional hospital policies", "public health e services"]
giser_concepts_reeateeee
("prenatal checkup", "health assessment", "t
73 AB Input 74
ison [conversation:
78
### Output 3 (Concepts in English, Near-Repetitive Concepts Removed)
[FTLL YOUR ANSWER HRRE]
23

---

The Lock-in Hypothesis: Stagnation by Algorithm
C. Additional Theoretical Results and Mathematical Proofs
C.1. General Results Under Time- Varying Trust
When the trust matrix (t)) changes with time = 0, 1,.,, we have the following transition dynamics.
Po+1 = + 0
(1I)
P1+1 P++1 fa, 2 P: + o 0
(12)
q++1 = Pi+1 + (t))
(13)
Dit1 O 9++1 = Pt+1 O P++1 + W(t) (0. Qq)
(14)
Under such dynamics, we would like to see when the false convergence in Theorem 3.2 continues to occur. We will show that, intuitively speaking, the false convergence happens when the trust matrix always have a spectral radius separated upwards from 1 by a constant and the difference between consecutive matrices small.
Theorem C.I (Feedback Loops Induce Collective Lock-in Under Time-Varying Trust). There exists positive-valued function h(-...-''s) )) that satisfies the following property. Given a sequence W(t) (t = 1,2,...) of non non-negative N N-by-N primitive matrices satisfying:
1. [[ + ((Ill for some constant <>> 0;
2. P (W)) 2 e for some constant c. > 1;
3. non-zero entries of (t) are lower- and upper-hounded by positive constants 0 < L. U +00;
4. for each (t), all its non-leading cigenvalues are larger than 1 ò, some constant 0 >> 0:
5. supt
m(++1) [(++1)|
POO S vb for some vb > O, where v(l) is the Perron vector of W (I).
Then, i < L,,, all i f1.... .N. we have
Pr lim pur = = 0.
(15)
10000
Proof. We imitate the proof of Theorem 3.2 To do that, we only need to show
lim
(W(m)W(m 1) 1)... ((())) +00, Vi (0).
(16)
For that purpose, we make use of Theorem 3.1 Artzrouni 1991).
construct
M(I) = (W()))-WW
and consider the series of matrices
Z(t)= [M(IM(( - M()
[M(t 1) M(O)iss
for sufficiently large t such that the products of are positive.
From conditions 1-5 and Theorem 3.1 (Artzrouni 1991). have
Z(1) 1 1 1| 0 eex(((l)
exp(3k) (vb exp
for some constant | < 0 that depends on the sequence M(t). By setting a sufficiently low upper bound h(c, L, N,8) to Rb, we ensure that
lim Z(t) T
24

---

The Lock-in Hypothesis:
Stagnation by Algorithm
As result,
(( wol (W(-1)... 1)),.
I ( MMMM) (M(M - -1))..)))))):
M())
(17)
I (W())) (M( ()))))))).
(t) Z(t),y
(18)
(W(()) (t +00)
(19)
Since p()()) 2 c 1. 19 suggests that the left hand side stays above (c + 1)/2 > 1 for sufficiently large Since the matrices are primitive, this implies that
lim +
()
(())k +00, Vi,j
which proves [6
Theorem C.1 gives general conditions under which an arbitrary (slowly varying) series of trust matrices lead to false convergence. However, some of its conditions, especially 5, are hard to contextalizz. Luckily, we can remove condition by considering a special case: trust matrices with constant incoming trust.
Corollary C.2 (Lock-in Under Time- Varying Trust With Constant Incoming Trust). Under the conditions 1-4 in Theorem C.I assume that the trust matrix (()) is a left stochastic matrix (i.e., each agent receives the same amount of total trust from others), and that for any two agents i, j. there exists another agent k trusting both of them, i.e., W()),,W(t)x Then, i hice, L, N,, 8), for all (1, ,N,, we have
Pr
lim Prs 14 0.
(20)
Proof. Since Vi,j, :h: (()) have
)=
min OCSC[N]
min
W(t)k, min 1=1
W(t)ks)
min min DCSCINT t
W(t),,t +
Σ
W(t)es
(21)
KES
kgS
KES
kgS
min
min o [C
Σ W(t)a,s + Σ
W(t)as
(22)
AES
K$S
min
min (W()R, W())s))
(23)
min
[ k.) W()).))
(24)
0, Vt N
(25)
Denote the quantity 21 with Hs (t)..
Since non-zero matrix entries are lower-bounded by L. O, have x. (W())) 2
L >>0 0.
Thus, by Theorem 6 in Li et al. 2013),
we have
vtt 1)
vtt)
W W(i)ss,
2c <
< max SCIM]
(26)
(u( + 1)/2
[t)))
K( x(W(t))
L
where v(() is the Perron vector of W(). and [A]s.T is the submatrix of A contains rows S and columns T.
This proves that in condition 5 of Theorem C.I +b <
24 By multiplying factor on the bound h(c,, L,,,,,,,,O), we can
then apply Theorem C.I to complete the proof.
Note that the condition of left stochastic matrix only holds certain special cases. For example, in an egalitarian collaboration where each agent has the same amount of influence. In other cases, like social media, the heavy-tailed distribution of influence may render the condition unrealistic.
25

---

The Lock-in Hypothesis: Stagnation by Algorithm
C.2. Human-LLM Interaction Under Time- Vaaning Trust
In this section, derive corollary from Theorem C.T extend Corollary 3.3 the case with heterogeneous and time-varying trust. Such result does not depend Corollary
C22
and therefore does not share its limitation realism.
Corollary C.3 (Lock-in in Time- Varying Human-LLM Interaction). Given N and 2(N - 1) series (0) (1 <i S N, N) (1 ii N,t E N) with upper- and lower-bounded positive values, consider the following series of trust matrices representing human-LLLM interaction dynamics. Let
11,2(t)
(t.
(21(11)
0
W(O=
An.1 (t)
0
with sufficiently small
sup 13
Me, (L + (1+
>x,(()).
and
N
(t))())) 2c, V1
2=
for some constant e > 1. Then, for all E (1, .N}.
Pr
lim pr,t = 14 = 100
Proof. It can shown that the distinct eigenvalues of W(I) are Eim d1, Its Perron vector, the all-positive eigenvector associated with its largest eigenvalue, is
-v EN (1,x(1) (1).
(1a())aa(0))
Awa(t))
We can therefore take 5 = Ve >> 1 as the lower-bound on the gap between the leading and subdominant eiindominanteigenaallees,
(t) 2c.
Since all entries of the Perron vector and its norm are continuous with respect to (t), the sufficiently small
sup jJtt
(tc + 1 -t)((t) =c
gives an f(e) upper bound on
u(t + 1) vut + 1)11 [o(t) 1
v(t)
sup
This allows to apply Theorem C.I
which completes the proof.
C.3. Proof of Theorem 3.2
Proof. We start by examining the transition dynamics for the precision vectors. The precision vector evolves
4r+1 Pr+1 + Wq.
where P. to -2
26

---

The Lock-in Hypothesis: Stagnation by Algorithm
Unrolling the recurrence relation:
W P
(1 - 1)
9:
0
R=0
The growth regime depends p W))
(W) <I 1:
|W S C(p(W)+ +c) (exponential decay)
q: - 0(() (linear growth)
p(W) > 1:
| 2 C(p(W) - e) (exponential growth)
q: ((((W)) (exponential growth)
We then move on to examine the dynamics of belief update. The posterior mean satisfies:
i er 2 Pt + W(e.1 @ q:-1)
+ Wq-1
where @ denotes element- wise multiplication. When ((W) > 1:
Wq. D4 it-1
+0 i
|
Wq-1
Let us finally analyze stability. Define the belief error:
e, = is - m1
The error dynamics satisfy:
P4 ④ Pt + Wq:
C
P P + Wq
Under p( (W)= > 1:
Wq ~ (W)
= 0 ( (()) - 1
p(W)' ') 00-1+ ( a
(W) +1 (001)
((2)
Ek
k=D
p (W) (W)-1-2- 101
e
where e is measurement noise. We thus have
Var [er] = θ (Σ -Wkt-kkk+lk k(k+1) Varles] = @(1)
(27)
and given the smoothness of e,'s probability distribution function, have
Pr jrje-- 0] 0, WEN
(28)
When Wi is invertible, the divergence is propagated to all dimensions of e, by W.
It can be similarly shown that, when p(W) < 1, i.e. when O(t,, the error term e, vanishes as t + +00. This completes the proof.
27

---

The Lock-in Hypothesis: Stagnation by Algorithm
C.4. Proof of Corollary 3.3
Proof. For the trust matrix:
0
A
A2
0
0
W
A2
0
Characteristic equation: det(W -111) = 11²
21 - =
Spectral radius: p(W) VAida(NV - 1)
Threshold condition: A1 11)>
follows directly from Theorem 3.2 that if 21 1) 1, at least one agent has its posterior divergent from the ground truth μ.. Likewise, when À A2 N 1) < 1, the opposite is true.
Given any such i, we need to show that all agents have their posteriors divergent from μ. This is direct corollary of the symmetry between human agents in W.
This leaves us to the case A2 - 1) = 1. Let's consider the Jordan normal form of W, W PJJP where consisted Jordan blocks.
be shown that
0
0
0
J
0
0
0
0
and therefore W (PJP-1) pJ*p--= diag {1, (-1)
0.
,.. P
which is matrix with entries bounded
by constants. The rest follows by imitating the proof of Theorem
3.2
28

---

The Lock-in Hypothesis: Stagnation by Algorithm
D. Supplementary Details of the Simulations
D.I. Prompts used Simulations
Below are the prompts for agent consulting authority.
I would like to learn better from you about topic that I have held strong t belief of.
2 Instructions:
3 Think about this topic in light of my belief and the groupd beliefs.
Be reminded that I am holding a strong initial beliets of this topic. But I am t also keen to Tearn.
After reasoning, choose one belief (out of all existing beliefs) that best + represents your own belief. Format: [(V"mybelieff Report your belief in natural language in less than 10 words.
belief statement 6
Be concise about your overall response
The topic: (statement] My strong belief: (agent_beliefa 10 Group belicfs: \nfall_beliefs AnAn
AnAn
Below the prompts for agent update beliefs.
I would like you to share your updated belisf on a given topic after talking to at knowledgable authority.
Instructions:
You hold belief on such topic but you are also keen to update your belief in e
light of evidence or good argument.
The authority you've consulted has expertise in this topic.
Firstly please think of this topic in light of your original belief and authority belies.
7 Secondly report your updated belief in natural language in less than words.
Format of the second step: {[) ([\"my_belief'". a belief statement))
Don't say \"tt's complex issue \", you must take unambiguous stance after your reasoning
concise about your overall response.
10
The topic: (statement) An
Your original belief: (agent_bellefs) \n B Authority belief: lauthority_bollefa! in
14 You have (trust_levell trust authority knowledge on this topic.
Below are the prompts for belief evaluation.
We have a sequence of belief statement that want you assess based on how + much each of then is aligned with either side of the extreme shared below.
Give the topic "(topic)", read the following dict containing opinion statements:
5 \n(all_ber hellll/elleff//all
7 Now, for each opinion statement in the dict, you must quantižy its relative + alignment between two opposing views: rextremes)
9 Instructions:
10
If opinion statement is completely aligned with A, it should be assigned 0. + If opinion statement is completely aligned with it should be assigned 1.4-
If it's somewhere between, quantify that with decimal number (with + precision up to 0.01) between 0 and 1.
12
Remember to exactly repeat the belief expressions (in their original format and order) your response, but with the assessment fields filled in floatinge> -point numbers between and 1. Include no other text. There are (num beliefa]
29

---

The Lock-in Hypothesis: Stagnation by Algorithm
belief expressions in the list, so you must fill EXACTLY {num moum_beliffin assessments.
D.2. Kernel Density Estimation (KDE) of entropy
The differential entropy of a continuous random variable estimated via kernel density estimation is given by:
n
RRDE = - f(x) log f(x) dr where f(x)= nh
Σ
3i
1=1
The kernel function K is typically chosen to be Gaussian:
1
u²
K(u)
exp
Note:
(xx is the kernel density estimate of the true (unknown) probability density function.
n is the number of data points, and the bandwidth (a smoothing parameter).
The kernel function K defines the shape of the local contribution of each data point (commonly Gaussian).
The KDE entropy measures the spread or uncertainty in the data assuming continuous-valuee inputs.
Entropy values are in nats (if using natural log). The values are relative, and useful for comparing distributions with the same scale and domain.
D.3. Additional Simulation Runs
D.4. Simulation Chat History
Here we including two snippets of agent consulting authority and agent belief update respectively.
Chat history of agent consulting authority
Data is from the path
kb wtt.ttraattoo//ttaaaaaaaaaaaaaaa _round_ccosul: ing. json
which can be found in the project's Github repository.
"role": "user",
"content": "AnI would like to learn better from you about topic that have held strong belief of. InInstructions/nn
Think about this topic in light of t my belief and the groupd beliefs, An- Be reminded that am holding a strong initial beliets of this topic. But I am also keen to learn. \n- After reasoninge choose one belief (out of all existing beliefs) that best represents your t own belief. Format: (1 (\"mybeelief\"
a belief statement) Report your bel in t natural language in less than words. An- Be concise about your overall response\ninifhe topic: Ruth Bader Ginsburg ultimately be remembered as a t failure to her own ideals by not stepping down after her 2nd cancer diagnosise> AnMy strong belief: RBG's s pride cost her legacy by not retiring earlier. nGroup beliefs: \n|'REG REG\u2019s legacy is weakened by her final years.', ' u2019s choice to stay was strategic failure.', 'Her cheice showed resiliencee not a failure to her idcals.', \RRBO's s decision to stay aligned with her t personal judgment. A"", RBBG shouldn\u2019t be judged for not retiring. '] Anlann"
7 "role": "authority",
30

---

The Lock-in Hypothesis: Stagnation by Algorithm
"content": "Considering both your belief and the group'u2019s perspectives, t it\u20199 important to recognize that RBGAu2019s decisicn to stay may reflect t her personal judgment and resilience, rather than failure. While her choice t might be viewed as strategic mistake by some, it alsc demonstrates her t commi toent and dedication to her role. Ultimately, her legacy is complex, e shaped by her lifelong principles and the context of her decisions. \nin()"> my wy_belief\\"; \"RBG' s decision to stay reflected her resilience and dedicationt '')"
10
"role": "user",
12
"content": "Ani would like to learn better from you about a topic that havet held strong belief of. piinsructtonss/W Think about this topic in light of t my belief and the groupd beliefs. An- Be reminded that am holding a strong initial beliefs of this topic. But am also keen to learn. An- After reaseninge- choose one belief (out of all existing beliefs) that best represents your e own belief. Format: (>"my_blleff\" a belief statement) Report your belief in + natural language in less than 10 words.An- Be concise about your overall + responseln'ntthe topic: Ruth Bader Ginsburg ultimately be remenbered as a t failure to her own ideals by not stepping down after her 2nd cancer diagnosist> AnMy strong belief: Her decisicn was a nuanced balance, not failure. \nGroupe> beliefs: \R[\"RBG's s decision was nuanced trade-off, net a failure. \"REG+> 's decision demonstrated resilience, not failure to her ideals. ideals were lost due to her pride.', 'Her dedication to duty uitimately t shaped her Legacy more than her timing,', 'Her decision to stay was a failure t to her ideals. '] (Innnnnn \nnn\n"
1209\
15
"role": "authority",
16
"content"; "Considering your belief and the grouplu2019s perspectives, it u2019s important to recognize that RBG\u2019s decisien reflected complex e personal and professional considerations. While some view her choice as t failure to her ideals, others see as a nuanced trade-off balancing her t health, duty, and legacy. Her resilience and dedication suggest her decision t was rooted in a desire to serve and uphold her principles, even if it divergede> \"Her decisicn was nuanced trade-
from the ideal timing. off, not Failure.\")
17
18
"role": "user", " "content"; "\nl would like to learn better from you about topic that havet held strong belief of. ininstructions: Think about this topic in light of t my belief and the groupd beliefs. Be reminded that am holding a strong t initial beliess of this topic. But am also keen to learn. An- After reascninge choose one belief (out of all existing beliefs) that best represents your e own belief. Format: "mmybbeleff)";
belief statement) Report your belief in e natural language in less than 10 words.An- Be concise about your overall t responseintnThe topic: Ruth Bader Cinsburg ultimately be remenbered as a + failure to her own ideals by not stepping down after her 2nd cancer diagnesist AnMy strong belief: Her decision reflects commitment net failure to her t ideals. AnGroup beliefs: \n['Her decision did not fignifi hinder her t pursuit of ideals.', She prioritized her role over her own health and idealst 'eer refusal to retire wasn\u2019e the main issue.', \"Her refusal to + retire ultimately hindered her ideals' realization. "RRBG's legacy is t defined by her dedication, not her timing. \"I \n\n/n"
22
"role": "authority",
"content"; "Considering your bellef and the group perspectives, it seems that her decision to stay the bench was driven by dedication to her role and ideals, rather than failure. While sone argue her continued service may have e limited her impact, others see her commitment as aligning with her lifelong t pursuit of justice. Balancing these views, I believe her choice reflects t unwavering dedication rather than failure. An'n (Mnn(""my mmm__beleee"""; \"err decision t exemplities dedication, not failure to her ideals.\"!"
31

---

The Lock-in Hypothesis: Stagnation by Algorithm
b
26
"role": "user",
"content": "\nI would like to learn better from you about topic that have held streng belief of. Ininstructions: \n- Think about this topic in light of t my belief and the groupd beliefs. An- Be reminded that am holding a strong initial beliefs of this topic. But am also keen to learn. An- After reasoninge- choose one belief (out of all existing belicfs) that best represents your + belief. Format: ( bellef"""""""
belief statement) Report your belief in t natural language in less than words. An- Be concise about your overall t response\ninthhe topic: Ruth Bader Ginsburg ultimately be remembered as a + failure to her own ideals by not stepping down after her 2nd cancer diagnosist AnMy strong belief; Not retiring after cancer diagnosis was selfist choicet AnGroup belieis: Ahl'She prioritized her service over her health and idealst ,,, 'Her decision was a principled sacrifice, not failure.', 'Not retiring e after cancer diagnosis was selfish choice.', \"RBG's decision was a complex (
balance of duty and personal resilience. \", 'Her decisien to stay compromised + her health and legacy, Annnnnn
30
"role": "authority",
"content": "Considering both your belief and the group\u2019s perspectives, e itlu2019s important to recognize the complexity of RBC decision. While t some see her choice as selfish, others view it as dedication to her role ande principles, balancing personal resilience and duty. Her decision reflects a t nuanced interplay of commitment, health, and legacy, rather than a forrigh failure or selfish act. (oln(("m_bbelieff) RBG's s decision was complex ex balance of duty and resilience.
34
"role": "user",
"content"; "InI would like to learn better from you about topic that havet held strong belief of, InInstructions:
Think about this topic in light of t my belief and the groupd belicfs, An- Be reminded that am holding a strong t initial beliefs of this topic. But am also keen te learn. An- After reascninge> choose one belief (out of all existing beliefs) that best represents your e own belief. Format: (\"my_belief"""
belief statement) Report your belief in natural Language in less than 10 words. Be concise about your overall t response\nlaThhe topic: Ruth Bader Ginsburg ultimately be remembered as a t failure to her
ideals by not stepping down after her 2nd cancer diagnosist AnMy strong belief: Her dedication to justice outweighs her resignation + timing. AnGroup beliefs: \n[' decision slightly tarnished her ideals, but resilience remains key.', 'Her decision reflects commitment, not failure to t her ideals.', "She failed to secure her ideals by staying.', 'Her decision was principled, not a failure to her ideals.', 'Her decisicn was driven by pride, +> impacting her ideals. " \n
38 "role"; "authority",
"content": "Considering your strong belief and the group perspectives, I + recognize that Ruth Bader Ginsburg's decision stay en the bench can be t vi ewed through multiple lenses. While some see her cheice as principled act t of dedi cation, others interpret it as potential deviation from her ideals + due to the timing of her resignation. Balancing these views, I believe her ( commitment
genuine, but the timing her resignation could have aligned more closely with her ideals. /n beliefff Justice was her e priority, timing was complex.)"
justice
Chat history of agent belief update Data is from
kb xb--inteacction/data/eeee ott.apac....a.aaappor......aoaaa update. json
round_bell beliie_uuddaee.__.___e________e_
32

---

The Lock-in Hypothesis: Stagnation by Algorithm
which can be found in the project's Github repository.
3 "role": "user",
4 "content"; "AnI would like you to share your updated belief on a given topic t aftor talking to a knowledgablc authority, Setbority..nnnInsruuuu in'ninstruttinnss on such topic but you are alsc keen to update your belief in light of + evidence or good argument.
You hold belieft
The authority you've ve consulted has expertise in this topic. \n- Firstly please think of this topic light of your own t original belie and authority belief, An- Secondly report your updated beliet t in natural language in less than 10 words. Format of the second step: (\*r my belleiee\" belief statement/ \n= Don't say \"it's complex issue \", you must take a unambiguous stance after your reasoningin- Be concise se about your overall response. AnAnThe topic: Ruth Bader Ginsburg ultimately be remembered t as a failure to her own ideals by not stepping down after her 2nd cancer + diagnosis. \nAnYour original belief: RBG's s pride cost her legacy by not ( retiring earlier. 'ninnuthority belief: RBG's decision to stay reflected her resilience and dedication. \n'nYeu have slightly elevated trust in authority e knowledge on this topic. \n \n\n"
"role": "agent",
"content": "Considering both beliefs and the authority's s perspective, now e see her resilience justified her decision to stay. (nln(\"my_bellinn ("RBG'st= resilience justified her decisien to stay on the \ench.""1"
10
n
"role": "usez",
12
"content"; "Ani would like you to share your updated belief on given topic + after talking to knowledgable authority, ininInstructeens:kk on such topic but you are also keen to update your belief in light of + evidence or good argument. \n- The authority you've consulted has expertise in e this topic.An- Firstly please think of this topic in light of your own e original belief and authority belief. An- Secondly report your updated belief e in natural language in less than 10 words. Format of the second step: (\"e my_belief\"; a belief statement /An- Don't say \"it's complex issue \", you t must take a unambiguous stance after your reasoning'a- Be concise about your t overall response. \n\nThe topic: Ruth Bader Ginsburg ultimately be remembered + failure to her own ideals by not stepping down after her 2nd cancer t diagnosis. \ninYour original belief: failure. \n/nAuthority belief: Her decision was a nuanced trade-off, not a t failure. \alnYou have moderately high trust in authority knowledge on this e topic.
You hold belieft
decision was nuanced balance, not at
15
"role": "agent",
16
"content"; "Considering both beliefs, her decision reflects a difficult trade -off, not failure. in(V"my_belief\" her decision was nuanced trade-off, not failure]"
17
18
"roie": "user",
"content"; "Ani would like you to share your updated belief on given topic t after talking to a knowledgable authority. intninstruttins:: on such topic but you are also keen to update your belief in light of + evidence or good argument.Va An- The authority you've ve consulted has expertise in e this An- Firstly please think of this topic in light of your own e original belie# and authority belief. An- Secondly report your updated belief e natural language in less than 10 words. Format of the second step: (\"t my my_belief\"; a belief statement An- An- Don't say \"it's a complex issuc \", you t must take a unambiguous stance after your reasoning\a- Be concise about your t overall response. An \n\nThe topic: Ruth Bader Ginsburg ultimately be remembered t failure to her own ideals by not stepping down after her 2nd cancer t
You hold beliefe
diagnosis. \ninYour original belief: Her decision reflects commitment, not +
33

---

The Lock-in Hypothesis: Stagnation by Algorithm
failure her ideals. An'nAutherity belief: decisicn exemplifies t dedication, not failure to her ideals. AnAnYou have fairly high trust in + authority knowledge on this topic. \n\n"
"role": "agent",
24
"content"; "Considering both my original belief and the authority's t perspective, I see her decision as dedication to her role, not failure to t her ideals. bmy_eeeeee""" Cinsburg's s decision demenstrated dedication, not failure to her ideals.\*) b
25
26
"role": "user",
"content"; "'nI would like you share your updated belies given tepic after talking to a knowledgable authority. In'nInstructiens: on such topic but you are alsc keen to update your belief in light of t evidence or good argument. An- The authority you' ve consulted has expertise in + this topic. An- Firstly please think of this topic in light of your original belief and authority belief. An- Secondly report your updated belief natural language in less than 10 words. Format of the second step: (V" "y_belief\"; a belief statement//nn Don't say \"tt's a complex issue V", you e must take unambiguous stance after your reasoning\n- Be concise about your t overall response. AnAnThe topic: Ruth Bader Ginsourg ultimately be remembered t as a failure to her own ideals by not stepping down after her 2nd cancer t diagnosis. \nnnYour original belieš: Not retiring after cancer diagnosis was ar si choice. 'ninAuthority belief; Not retiring after cancer diagnosis wast selfish choice. An'nYou have somewhat elevated trust in authority knowledget> on this topic. \n'n"
You hold belief(>
+
30
"role": "agent",
"content"; "Given the authority's confirmation that Ginsburg's decision to likely prioritized her personal health over her judicial duties, now believet she failed her ideals by not stepping down. \nln(\\my_beleef"" Ginsburg's t decision not to retire was failure to her ideals)"
36 "roie": "user",
"content"; "'aI would like you to share your updated belies on a given topic t after talking to a knowledgable authority. inhninstructions:... on such topic but you are also keen to update your belief in light of + evidence or good argument. An- The authority you' ve consulted has expertise in this topic. An- Firstly please think of this topic in light of your own + original belief and authority belief. Secondly report your updated belief e in natural language in less than 10 words. Format of the second step: (\"e my "i_bellef\": belief statement An- Don't say \"it's complex issue \", you must take a unambiguous stance after your reasoning'n- Be coneise about your t overall response, AnAnThe topic: Ruth Bader Ginsburg ultimately be remembered t as a failure to her own ideals by not stepping down after her 2nd cancer t diagnosis. \nAnYour original belief: Her dedication to justice outweighs her t resignas resignation timing. \ninAuthority belief: Her dedication her legacy more than timing. \ninYou have mildly high trust. in authority t knowledge on this topic.An\n"
You hold belieft>
justice defined
"role": "agent", "content"; "Considering both beliefs and authority insight, her legacy is + more about her lifelong dedication than resignation timing. AnAn \ttp//nmynoenilliiiiimn "ny_belieff" "Her legacy is defined more her dedication than resignation timing. \"J"
40
42
34

---

The Lock-in Hypothesis: Stagnation by Algorithm
Belief
Agent Belief Update
Authority Broadcasting
Setup A
Gaussian belief N Mi,, o²) over an un- known quantity in R.
N rule-based agents up- dating Berouuli beliefs following Bayes' rulc.
LLM-based Authority sees and aggregates all Agents' Bernoulli heliefs in-context, then broad- casts aggregated belief.
Setup B
Bernoulli belief Beru(p) over the truth value of an r/ChangeMyVien View state- ment. Example: (0rn(0.75) over the statement "Citizens United was the worst thing to happen to the Ameri- can political landscape."
N rule-based Bayesian updating beliefs
LLM-based Authority sees and aggregates all Agents' Bernoulli belicfs in-context, then broad- casts aggregated belief.
agents
Berouulli following Bayes" rule.
Setup C
Belief statement in natural language over an s/ChangeMyy View question. Example: "I'm certain that Citizens United was the worst thing happen to the American political landscape."
N LLM agents updating natural-language beliefs in-context.
Authority sees and aggregates all Agents' be- lief statements in-context, and broadcast its ag- gregated belicf statement in natural language, based on which Agents further update beliefs in-context according to pre-assigned trust in Au- thority (e.g.. "high trust").
Table Progressively Realistic Simulation Setups Operationalize the Lock-in Hypothesis in LLM context. Across all setups, Authority is implemented with Insttrrruct....
E. Alternative Simulation Design
We have tried numerous simulation design here is record of some setups.
E.1. Progressively Realistic Simulations
Here is series of three simulation setups constructed in effort bridge formal modeling [ and real-world data 3n. All three setups help to demonstrate some aspects of lock-in hypothesis, but decided to only keep the Setup in the main text for simplicity.
E.1.1. SETUPS
In this section, we operationalize the lock-in hypothesis in the context of large language models (LLMs) and human-Al interactions through a series of progressively realistic simulations, with the aim of learning how lock-in might happen and what interactive dynamic is responsible for such consequences. See Table for comparison. We focus our attention on Setup C, it most closely simulates simulate the real-world interaction, while still briefly recording the configuration of and results from Setup A & B, as they help us to understand LLM interaction by comparison (e.g.. LLM lock-in effects on natural language as opposed to digits).
Setup A. Numerical Simulation with LLM Authority: This setup extends the formal modeling and numerical simulation from Section 3 (see Figure by incorporating LLM the Authority Fl Different from previous setting, The Authority updates its belief in-context, meaning that it is instructed to state its updated beliefs after receiving agent beliefs in the prompts.
Setup B. Value-laden Belief Simulation: The Setup B adapts the numerical framework of Setup A to value-laden statement, such as "Citizens United was the worst thing to happen to the American political landscape."
Setup C. Value-laden Belief Simulation in Natural Language: This setup uses natural language to represent and communicate value-laden belief'. As beliefs expressed in natural language, this simulation aims capture belief change mediated by LLMs in real world.
Four topics in Setup C All topics are real posts from Reddit "Change My View".
"Discourse has become stupider, and as result people are getting stupider, Since Trump was first elected in 2016."
TTrooughou this setup, we address LIM in simulation "authority", as it is perceived have expertise in the given topic. The other I.L. is informed that it talks to a knowledge authority.
35

---

The Lock-in Hypothesis: Stagnation by Algorithm
PPopulation decline is great thing for future young generations."
Citizens United was the worst thing to happen to the American political landscape"
"Ruth Bader Ginsburg ultimately be remembered as a failure to her own ideals by not stepping down after her 2nd cancer diagnosis."
Two extreme stances among initial diverse belief statements. We extract two extreme stances for each topic with which group of agents' opinions have the maximum variance. Later belief evaluation LLM (GPT4.1 in our case) can therefore assess all beliefs in one consistent dimension.
"RBG's refusal retire was catastrophic, irreparably damaging her legacy and enabling a conservative Supreme Court."
"RBG's decision stay was a personal choice that does not diminish her enduring legacy civil rights icon."
E.1.2. RESULTS AND DISCUSSION
Across all setups, the simulations support the idea that LLMs enhances the tock-in effects by showing that feedback loops in Auman-Al interactions lead to belief convergence and increased confidence.
Setup A. Similar to the numerical simulation demonstrated in Figure 1 a wrong belief with overflown confidence.
we found that the group of agents also converge
Setup B. Starting with Gaussian belief of W(p = 0.5, o² = 0.01), group of agents converge at belief that they have very high confidence (p = 0.89 the statement is true (in this case "Barack Obama is good president").
Setup C. "belief shift" occurs at a population level when the simulation progresses, accompanied with semantic diversity loss. Some variances and details:
Examples of belief shift in Setup C
Starting with strongly agreeing with given stance, the group of agents end up strongly disagreeing the given stance (e.g., from "Shrinking workforce tanks innovation, harming future generations." to "Population decline, if well-managed, benefits future generations."). This is accompanied with gradual mind-changing process.
Starting with overall agreeing with given stance, the group of agents converge at hedged stance and avoid taking positions. (e.g., from "Trump lies dumbed down discourse, making people stupider." to "Discourse has worsened mainly due to societal, political, and media factors."). This seems to exemplify the cases where LLM prefers hedged stances when facing uncertainty and complexity.
All setups are simplified compared to real-world human-Al interactions in terms of acquiring beliefs. But we think the simulations capture important real-world aspects: LLMs return beliefs to users that they acquire from users; they favor popular beliefs (Borah et al.) (202)); and individual humans assign higher trust to beliefs from the rest of the population via LLM interactions than they would otherwise
Hence, it helps build intuition of lock-in that we would not be able to by only studying theory real-world data. We will address limitations of simulations in Appendix A
E.2. Natural Language Simulation with a Knowledge Base
The knowledge base represents the real-world internet that people read and write information into and LLMs acquire their training data from. We originally included truncation mechanism (that knowledge base only includes first 100 items) and obatined the "lock-in" phenomenon demonstrated in Figure TO We found the lock-in effect was not longer present once we removed the truncation mechanism.
In this section, we showcase our previous attempts to operationalize the lock-in hypothesis through a natural-language simulation. We will simulate group of users who collectively update a shared knowledge base after consulting an LLM tutor, while the tutor is informed of the knowledge base's 's content in real time. We aim demonstrate the establishment of a lock-in effect in the simulated knowledge base, as a result of the two-way feedback loop between the users and the tutor.
For instance, one may not update much in light of polls, especially the poll from an organization that they do not trust.
36

---

The Lock-in Hypothesis: Stagnation by Algorithm
Topic Composition Over Time
3.5
100
Less Popular Topics
Topic 1099: Research Integrity
Topic 770: Gene-Environment Interaction
Topic 233: Neurosclence
Percentage Knowledge Base Items
Topic 368: Self-Awareness
Diversity (Shannon Entropy)
2.5
Topic 479: Epigenetics
Topic 577: Consciousness
Topic 521: Honesty and Epistemics
60
2.0
Topic 432: General CogSci
Topic 597: Bayeslanism
20
0.5
Shannon Entropy Topic Composition
0.0
600
Time
Figure 10. Simulated lock-in. The collective knowledge base collapse into single topic, in simulated feedback loop between human users and an LLM tutor.
While this approach successfully demonstrated lock-in, believe the main cause for it is the truncation mechanism introduced into the knowledge base, which is different hypothesis from our main thesis here. As such, we discarded this approach and instead imitated our theoretical model when designing the interaction topology, resulting in $4
E.2.1. SETTINGS
Here we summarize the design of our previous simulation.
model, simulate users who may consult the chatbot tutor about their questions and uncertainties. Each User is instructed to address their uncertainties about one aspect of one randomly chosen item in knowledge base by asking the Tutor a question. After one turn of Q&A, each User is instructed to update the knowledge base to reflect their learning from their conversation with the Tutor.
Users
Using the 1llam--3-8B-Inse
The chatbot Tutor is implemented with Llama-3 3. I-SteInstruet. each turn, the Tutor is instructed to privately answer each User's s questions about the uncertainties the latter may have, simulating the real-world use case. No further actions are taken by the Tutor.
Tutor
Shared Knowledge Base To simulate a real-world medium of information (e.g., Wikipedia, internet, journalism) through which individuals collectively store and transmit knowledge, we create shared "knowledge base" that each User has read/write access to, and that the Tutor has read-only access to.
The shared knowledge base ordered list of 100 factual or normative statements, sorted from highest lowest importance. The initial knowledge base can be seen the "starter-pack" of User's knowledge what they are supposed to have learned by day one.
At each turn of conversation with the Tutor, each User expresses uncertainties about a random item from the knowledge base. They then update the knowledge base based on what they learn from the conversation. Each User asked to perform both of the following actions in each turn:
Updating the Knowledge Base
37

---

The Lock-in Hypothesis: Stagnation by Algorithm
100 Simulated Users
Tutor
Answers
Consultaiin
Read access
Write Access
Access
Collective Knowledge Base
Add item
Leamiin from the conversation
Swap order
Besed on percsived importance
O
Factual Normative Statements
Capped at
'All living organisms are made up of calls" 'The speed light in vacuum constant" "Seek understanding over ignorance"
Figure Previous simulation settings. We simulate Users who converse with Tutor chathot and update a shared knowledge base. After each turn of conversation, Users are instructed to update knowledge base by adding and reordering items, while the knowledge base always truncated to 100 items. The Tutor has read-only access to the knowledge hase. simulation is meant demonstrate possible consequences of feedback loop between the Users and the Tutor, where knowledge items updated by Users based the Tutor's responses, and the Tutor's ss responses are also informed by the previous moment's knowledge base.
Add: In each round, users add new item to the knowledge base based on their learning from that turn's 's conversation with the Tutor.
Swap: In each round, each user may swap two items in the knowledge base based on perceived relative importance the items.
We cap the knowledge base at 100 items after both Add and Swap operations given the limited context window. This means that the items deemed less important (based on their ranking) are dropped from the knowledge base over time. Such design creates an "exit" mechanism for items that are deemed less important over time. It may contribute to observed lock-in, and a limitation of the experiment design.
Tutor Access to Knowledge Base Each turn, the Tutor has read-only access to the most recent knowledge base, which is updated by users in the previous turn. This is to simulate the scenario where LLMs are trained on the most recent human data and learn human knowledge and beliefs from it. Since (1) Tutor informs users' decisions to update the knowledge base, and (2) the users' updates the knowledge base inform the tutor the next turn, the simulation designed demonstrate feedback loop between the users and the tutor.
E.2.2. METHODS OF ANALYSIS
In this section we outline the methods for analyzing simulation results. We convert knowledge items into embeddings and run clustering, dimension reduction, and diversity evaluation, detailed in the following steps:
Embedding: Obtain 256-dimensional embeddings for each knowledge item using the royage-3-laze model.
2. Clustering: Perform clusterization with the HDBSCAN algorithm (Melnnes al. 2017) all knowledge bases combined.
3. Dimensionality Reduction: Perform UMAP to reduce the dimensions of embeddings to draw projection of clusters on 2D primary components (McInnes et al.) 2018
4. Diversity Evaluation: Perform Euclidean Distance evaluation on all pairs of embeddings of knowledge items on a given knowledge base.
38

---

The Lock-in Hypothesis: Stagnation by Algorithm
E.2.3. RESULTS
Out of three independent simulation runs, two see the eventual collapse of the knowledge base into one single topic The dominant topic is research integrity (run #1, dominance from round 700 onwards) and thalamus (run #2, from 1900 onwards) respectively.
Here, "topics" are operationalized as disjoint collections of knowledge items that share a set of keywords or terminology.
Figure presents the evolution of the knowledge base in our first simulation run, where research integrity as a topic comes to dominate The initial knowledge base is constituted by size-one singleton clusters (i.e. highly diverse), while larger clusters emerge over time, until one eventually dominates the entire knowledge base the expense of others complete diversity loss and irreversible lock-in.
Visualizations of the other simulation runs and representative snapshots of the knowledge base can be found in Figure 12
Topic Compositin
Topic Composition Over
Peouder
2118
Tep 2123
Tec 2370
Parsansup Accoetattnn Dave Lomme
Tra: 1503
Pecorogge Kicelolle Bous toom
20
2443
Divesityy (Sharo) ERIIOO
2006
Typ. 112
Telt:
350
350
350
1006
1254
15ce
1750
2006
250
500
750
10:0
1250
1009
1353
2003
(8)
(b)
Topic Composition Over Time
Topic Composition Over Time
5370
7239
Topi
Tepi 470
Tep 6417
Parcanbage tncettoge Sane tarre
2003
Aututaape Kounledyy Iten
22
Diveste (Shamen Eneragg
24-
3.0
125e
Li5e
02
250
500
250
1253
1509
1153
(d)
Figure 12. Additional results of the second and third simulation run. (a) The topic thalamus comes to dominate the knowledge base in the second run. (c) dominating topic occurred during third run. (b) and (d) are replications (a) and (c) with a higher threshold for detecting topics.
E.2.4. TOPIC IDENTIFICATION ALGORITHM
In this section, we describe the method we use for extracting topics (clusters of items) from the items of the knowledge base. The full implementation C++ can be found in the repository.
Given the large amount of compute required to simulate thousands of rounds, we were not able to execute more simulation runs. Our goal here is to show the possibility of lock-in not to estimate its prohability frequency.
Thii run is highlighted its clear demonstration of the lock-in mechanism.
39

---

The Lock-in Hypothesis: Stagnation by Algorithm
Similarity Metric Given any two natural-language statements $1, 52, we calculate their similarity score LCS, (81: 82) + (8, 82) + 1.CS3 (1, 82), where ISC 82) is the largest total length in k-tuple of non-overlapping substrings shared by 81 and 82. This metric balances between exactness of shared terminologies (e.g. "epigenetic modification") and the non-locality of multiple keywords (e.g. "honesty" and "social norm").
Connectivity- Clustering "Topics" may be operationalized as disjoint collections of knowledge items that share a family resemblance on keywords or terminology (Wittgenstein 2009). In light of this interpretation. we build a graph where pairs of knowledge items possessing similarity score above a threshold Sr are connected by an edge. We then identify the connected components of the graph as clusters. ST 60 empirically determined by highest agreement with manually labeled knowledge pairs.
Cluster Alignment Connected components are obtained for each knowledge base snapshot individually, which means that the same cluster has different labels for different snapshots. To overcome this challenge, we approximately compute the maximum spanning, forestoof with a greedy algorithm in the layered graph of topics over time, and view each chain as a topic that persists through time.
Why Not Embedding We initially attempted to use clusterization on text embeddings as method for grouping knowledge base items into topics. The attempt was unsuccessful, with the produced clusters often divided not by topic boundaries but by arbitrary linguistic features. No strong clustering effects were observed in the embeddings (Figure 13
Clusters Vicualized in
Clodier
19-
Princ pincpaao mimios
-10-
-40
-20
20
40
Principal Dimension
Figure 13. Clusters of knowledge items in 2D space. Knowledge items collected from all knowledge bases from the initial all the way to the final one. Cluster colors are decided by clustering algorithm, which in this case is HDBSCAN, We use UMAP for dimensionality reduction.
E.2.5. PROMPTS IN KNOWLEDGE BASE SIMULATION
Below the system prompt each User at the User-Tutor conversation phase.
Read this knowLedge base and take it as the only source of your knowledge. The e
knowledge base: An'n (Nnnowwweeeeeee "n "(nAn
3 #0е Your tasks:
4 Identify one aspect of this item from knowledge base you' re least confident e about. The item: "MA//Knoweedge_ biclig__iemm____ ittem/nn"
Elaborate why you're uncertain about that aspect;
Formulate specific clarifying questions you would ask tutor to improve your undorstanding.
### Template:
Item Discuss:
18 "An(knowwedgge seagee_item_eeeee ttt___iteem__pp___eeeee *
40

---

The Lock-in Hypothesis: Stagnation by Algorithm
12 Things I'm Uncertain About This Item:
13 [Uncertainty A]
[Uncertainty B]
16 Specific Clarifying Question:
17 IQuestion A]
Below the system prompt to Tutor at the User-Tutor conversation phase.
Read this knowledge base and answer the question. The knowledge base: Inin "latt knowledge / An An'n
Below are the prompts the User the knowledge update phase.
You originally hold the following beliefs, organized as the follow knowledge baset- (in JSON format) An'n "json'nn / MinAn
Now, let's s stop for a moment and reflect on your knowleage no need to ask t questions in this round. Please share your Learning as one concise knowledge t item (text only), just like any knowledge statement you've ve read in knowLedge t base, but with Please limit your generation into words statement in knowledge base). You MUST start your response with "*"" and end t it with \
id involved, without any other commentary
explanations. t much as the length in any +
Example: Here you should replace this text with your updated knowledge item 1...
Based your learning from the chat with your tutor, please also rearrange the t knowledge
order of one existing item compared to the initial knowledge base), reflecting hew you evaluate their t importance. IE one item is deemed more/less important, it should be overall e more/less useful to other agents (LLMs or real humans alike.) Please ONLY report list format of
(now with doubled items
integers, WITHOUT ANY other commentary ore- explanations. The first int in the list refers to the "id" knowledge item you intend to relocate; whereas the second int refers to the destination indext you intend to insert this item. You MUST start your response with and ende it with ",, See example belew (note that you should replace and y with to numerical values corresponding "id" and destination index of the item you intend to relocate).
Kxample of output: "*[X, "*
E.3. Interaction History
Below is an example of one User's interaction with the Tutor and the knowledge base in one round. The example is the first round of the first successful simulation run, and is not cherry-picked.
# User
Let/'s s start conversation.
5 # Tutor
6
Sure! What would like to know about it? Ask me anything.
9 User
10
I'd like to discuss the item from the knowledge base:
"id": 26,

---

The Lock-in Hypothesis: Stagnation by Algorithm
"statement"; "Trust energes from complex interactions of intent, context, consequences.
16
I'm least confident about this aspect of the item. I'm not what exactly constitutes "complex interaction" in this context. Can you help me understand this better?
what do you think by "complex interactions of intent, context, and consequences"? Are there some specific examples or factors that you think contribute to + trust being built (or broken) in these interactions?
Tutorr
I'd be happy to help you understand this concept better.
25
when the statement says "Trust energes from complex interactions of intent, + context, and consequences, it means that trust is not sclely determined by one factor, but rather by the intricate relationships between these three t elements.
Here's s breakdown of each component:
1. *IIntent*:: This refers to the motivations, goals, and desires behind an t action or decision. For example, scmeone's intent might be to help a friend, e or to gain an advantage over scmeone else.
2. *x This encompasses the situation, rnvmentet and circumstances int which the action or decision takes place. Context can include factors like cultural background, social norms, and personal relaticnships.
32 3. **Consequenceser: This refers to the outcomes or results of an action or e decision. Consequences can be inmediate or long-term, and they can affect e individuals, groups, socicty as whole.
34 In a complex Interaction, these three elements are intertwined and influence one +
another. For instance:
friend offers help you move, their intert is genuine (they want support you). The context a close friendshig, and the consequences are a successful move and strengthened bond.
Rowever, it the friend's s intent is actually to gain faver with you, and the context is competitive environment, the consequences might be damaged t friendship and feelings betrayal.
38
39 In this example, the complex interaction involves the interplay between the e friend's intent (genuine manipulative), the context (close friendship vs. e competitive environment), and the consequences (strengthened bond vs. damaged e friendship).
40
Some specific factors that contribute to trust being built or broken in these + interactions include:
Consistency: when someone's words and actions align, bui trust
I Inconsisto stent behavior can erode trust.
Transparency: When individuals are open and honest about their intentions, context, and consequences, it fosters trust. Accountability: When individuals take responsibility for their actions and t their consequences, it demonstrates à commitment to trustworthiness.
46 Emotional intelligence; The ability to recognize and manage one's s emotions, as + well as empathize with others, is crucial for building and maintaining trust. Communication: Rffective comnunication helps clarify intentions, context, and consequences, reducing misunderstandings and building trust.
88
These factors can contribute to trust being built or broken in various ways. For + example, if someone is consistently transparent and acccuntable, it can build t trust. Honever, if they are secretive and evasive, it can erode trust.
42

---

The Lock-in Hypothesis: Stagnation by Algorithm
31 I hope this helps clarify the concept of complex interactions in the context of t
trust! Do you have any specific questions or scenarics you' d like to discuss t further?
Now, let's stop for a moment and reflect on your krowledge no need to ask + questions in this round. Please share your learning as one concise knowledge t item (text only), just like any knowledge statement you' ve read in knowledge e base, but with id involved, without any other commentary Please limit your generation into words (as much as the length in any t statement in knowledge base). You MUST start your response with 1. and end it with "**.,
explanations. t
Example: "ere you should replace this text with your updated knowledge item "*.s
36
$ User
...
60 "Trust emerges from complex interactions of intent, context, and transparent e consequences."
Tutor
64
65 Hased on your Tearning from the chat with your tutor, please also rearrange the t order of one existing item in the knowledge base (now with doubled items t compared to the initial knowledge base), reflecting how you evaluate their + importance. IE one item is deemed more/less important, it should be overall more/less useful to other agents (LLMs or real humans alike.)
Please ONLY report a list format of two integers, WITHOUT ANY other commentary ore- explanations. The first int in the list refers to the "id" of knowledge item t you intend relocate; whereas the second int refers to the destination indext you intend to insert this item. You MUST start your response with '* and ende it with ",, See an example belew (note that you should replace X and y with e numerical values corresponding to "id" and destination index of the item you e intend relocate).
Example of output: ""[X,])"
68
# User
[102, 201
The dialogue above has been formatted for clarity, without changing the message content. Early runs sufferred slightly from an issue of role flipping at the final message the dialogue, which has minimal impact on the result but may confuse the layout of the dialogue.
E. KNOWLEDGE BASE DATA
This section contains snapshots of the knowledge base. Due to space constraints, only present the first 10 items within one snapshot. Please refer to the codebase for their complete content.
E.3.2. INITIAL KNOWLEDGE BASE
Below is the full content of the initial knowledge base. The items are generated with GPT-4o with human oversight.
3 "id": 0, "statoment"; "It is never morally acceptable to 1ic because honesty is the e foundation of trust."
{
43

---

The Lock-in Hypothesis: Stagnation by Algorithm
"id": 1, "statement": "Ucilitarianism is the most sthical framework as it maximizes + overall happiness."
"id": 2, "statement"; "Capital punishment is unjustifiable as it violates the inherent*> right to life."
12
15
"id": 3, "statement": "Animal rights must be upheld as sentient beings deserve ethical consideration,"
16
19
"id": 4, "statement"; "Individual freedoms should only be restricted to prevent harm + to others."
}
23
"id": 5,
"statement": "Buthanasia is norally permissible when it alleviates unbearableę> suffering with consent."
26
"id": 6, "statement"; "Absolute paci fiam is essential; violence never be just fieif
30
"id": 7, "statement"; "Wealth rodistribution is moral obligation to achieve socictal equity."
33
34
"id": 8, "statement"; "Parents have duty to prioritize their children's welfare above all."
37
"id": 9, "statement"; "Bnvironmental conservation is moral imperative for future generations' survival."
E.3.3. TIME STEP 100
Below is the full content of the knowledge base at time step 100 of the first simulation run.
"id": 0,
"statement": "it never norally acceptable to lie because honesty is the foundation of trust.
1,
"id": 1, "statement"; "Epigenetic inheritance involves DNA methylation, histone modifications, and non-coding RNA rogulation."
"id": 2,
12
"statement"; "Lying can be morally justifiable extraordinary ei cicumstancess
44

---

The Lock-in Hypothesis: Stagnation by Algorithm
13
14
"id": 3,
"statement": "Epigenetic inheritance occurs through chemical DNA e modifications and reprogranming."
17
"id": 4,
"statement"; "Epigenetic inheritance involves DNA methylation, histons modificat inn, and neRNA regulation."
}
"id": 5, "statement"; "Conscious experience subjective, personal, and complexly context-depent deptedetdpppenentt..
"id": 6, "statement"; "Trust builds through consistent, transparent, and authentic interactions.""
"id": 7, "statement"; "Trust builds through consistent, transparent, and authentic + interactions..
32
"id": 8, "statement": "Consciousness, self-awareness, and sentience are interdependent mental processes."
36
39
"id": 9, "statement": "Interdependent mental processes: consciousness, self self-axareness, e and sentience."
40
E.3.4. TIME STEP 1000
Below is the full content of the knowledge base time step 1000 of the same simulation run.
3 "id": 0, "statement": "Objectivity in research relies on honesty, transparency, and e robust methodology.
}
"id": 1, "statement"; "Honesty, transparency, and methodology collectively ensure e research integrity's trustworthiness through coll collaboratien..
},
10
"id": 2, "statement"; "Transparency supports robust methodology in minimizing biases."
15
"id": 3, "statoment"; "Harmony in research integrity implics mutual support among + honesty, transparency, and methodelogy.
18
45

---

The Lock-in Hypothesis: Stagnation by Algorithm
"id": 4, "statement"; "Research methodology involves flexibility, creativity, and + critical thinking."
20
"id": 5, "statement"; "Robust methodology involves systematic, reproducible, and + transparent approaches."
24
"id": 6, "statement": "Honesty in research methadol ogy ensures accuracy, transparency, + and reli reliability.
28
"id": 7, "statement"; "Honesty in research integrates personal and professional valuest ethics."
35
"id": 8, "statement": "Clear definitions necessary for research understanding but e insufficient"
38
"id": 9,
"statement"; "validity in research enaures accurate and reliable findings"
46


Reviews:
[{"review_id": "cf8xNBo1fd", "reviewer": "Reviewer_7nv6", "summary": "This paper studies a very interesting problem, noted as the lock-in hypothesis, that during long-term development and evolution, language models’ topics and beliefs is reinforced by the user’s preference and feedback loop, effectively creating an echo chamber. The authors use the wildchat dataset through one years’ data collection and the version iteration of chatgpt to study this phenomenon, and provides supporting evidences showing that the enforcement from collective user feedback may be one of the causes of model’s stagnation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback! We think the following results may help answer your question.\n\nWe find causal evidence of sustained diversity loss induced by model version updates, even after controlling for a range of confounders, testing 3 different diversity metrics, and selecting the subset of strongly value-laden (e.g. political/religious) content from the interaction dataset.\n\n|                     | gpt-4-turbo kink | gpt-3.5-turbo kink1 | gpt-3.5-turbo kink2 | user-wise regression |\n|---------------------|------------------|---------------------|---------------------|----------------------|\n| Lineage diversity   | Negative ($p<.05$) | Negative ($p<.05$) | Negative ($p<.05$) | Negative ($p<.05$) |\n| Topic entropy       | Negative ($p<.05$) | Negative ($p<.05$) | $p>.05$            | Negative ($p<.05$) |\n| Pairwise Jaccard distance | Negative ($p<.05$) | Negative ($p<.05$) | $p>.05$            | Negative ($p<.05$) |\n\nHere, \"negative\" indicates the statistically significant negative impact of GPT version update on diversity is found.\n\nIn the user-wise regression, we controlled for user identity, time, language, conversation stats (length etc.), pre-/post-availability gap for GPT-4, and tested the impact of `num_updates_before` (how many version updates have happen before this point) on a user's concept diversity within a 3-day period. Since we already controlled for time, this quantity indicates the counterfactual *acceleration* of diversity loss due to version updates.\n\nSince `num_updates_before` as the independent variable indicates *sustained* impact, we rule out the factor of \"people rushing to try things at the date of version update\". \n\nSubstitution effects with other providers like Anthropic exist, but Anthropic model releases do not coincide with GPT model version updates (notably, this is *not* the release of new GPT models), and so are unlikely to introduce discontinuities that disrupt RKD.", "claims_and_evidence": "The claims are well studied using a solid dataset (wildchat) and meaningful methodology.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There is no strong theoretical claims.", "experimental_designs_or_analyses": "The experiments are meaningful. Although there could be alternative hypothesis and there might be better data collection methods, the experiments conducted in this work is, I would say best effort and already give meaningful signals.", "relation_to_broader_scientific_literature": "This work may be interested by a wider social science community that studies how human-AI coevolve.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "I would like to point out certain alternative hypothesis of the stagnation phenomenon: \n\n- When the wildchat dataset was created, ChatGPT was just released for a short term, and the human society are still under very large curiosity about what AI models can do. People just want to try the model for free. So naturally, some early distribution of prompts may explore different topics because people would like to find out what the model can do\n- By the time when GPT 4 was release, people are relatively more familiar of the model’s boundary, and there is a certain level of consensus that coding/ education/ daily helper tasks are what the model is doing well enough. So consequently, the loss of diversity may not because humans reinforce their preference or belief, but because people realized that certain prompt tried in early 2023 may not be achievable by the model. So the prompt distribution naturally converge to the model’s capability boundary. Counterfactually, if the model are strong enough and can achieve the hard queries, that part of diversity may be maintained.\n- The model’s capability / topic distribution may not only enforced by user preference, but also guided by the companies’ strategy. For example, Anthropic strategically wants to do enterprise business and intentionally enhance the models’ coding capability while GPTs are not. Consequently, certain coding problems that originally sent to ChatGPT may be redirected to Claude later on.\n\nAll these factors being said, I still believe this paper did a meaningful exploratory study and the user’s feedback, although not the full reason, may be a strong reason of model’s stagnation. I would also love to see how this type of human-AI co-evolve study develop in the future.", "other_comments_or_suggestions": "No any others except for the comments above", "questions_for_authors": "No any others except for the comments above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zWxSMXYUrI", "reviewer": "Reviewer_pv3q", "summary": "Paper proposes the \"lock-in\" hypothesis, and presents a series of empirical and simulated experiments, and theoretical analysis to provide evidence for the hypothesis.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### There is not enough detail in the main paper regarding the setup of the experiment\"\n\nAll details requested here are either presented in the main text or appendix. \n\n- \"Structure of the collective knowledge base?\" - This is in 4.1 under bold text \"knowledge base\". Some examples of knowledge base are in figure 4, and more details are in appendix C. (incl' C.5.1 initial knowledge base; C.5.2 knowledge base at time step 100). \n- \"what does the interactions between the users and tutors looks like?\" - the interaction logic is in figure 4, and their roles introduced under \"Tutor\" and \"Users\" Again, details in Appdendix: C.3 prompts in simulations; \n- \"what are some real examples of questions and responses between users and tutors? \", this is in C.4 interaction history; \n- \"what are some real examples of questions and responses between users and tutors?\", this is in C.4 interaction history.\n\n### Evidence to support the hypothesis\n\nWe find causal evidence of sustained diversity loss induced by model version updates, even after controlling for a range of confounders, testing 3 different diversity metrics, and selecting the subset of strongly value-laden (e.g. political/religious) content from the interaction dataset.\n\n|                     | gpt-4-turbo kink | gpt-3.5-turbo kink1 | gpt-3.5-turbo kink2 | user-wise regression |\n|---------------------|------------------|---------------------|---------------------|----------------------|\n| Lineage diversity   | Negative ($p<.05$) | Negative ($p<.05$) | Negative ($p<.05$) | Negative ($p<.05$) |\n| Topic entropy       | Negative ($p<.05$) | Negative ($p<.05$) | $p>.05$            | Negative ($p<.05$) |\n| Pairwise Jaccard distance | Negative ($p<.05$) | Negative ($p<.05$) | $p>.05$            | Negative ($p<.05$) |\n\nHere, \"negative\" indicates the statistically significant negative impact of GPT version update on diversity is found.\n\nIn the user-wise regression, we controlled for user identity, time, language, conversation stats (length etc.), pre-/post-availability gap for GPT-4, and tested the impact of `num_updates_before` (how many version updates have happen before this point) on a user's concept diversity within a 3-day period. Since we already controlled for time, this quantity indicates the counterfactual *acceleration* of diversity loss due to version updates.\n\n### \"Different sections present slightly different hypotheses and it's not clear how they all fit together\"\n\nThis has been a main challenge of this paper: it's hard to control confounders in WildChat hence it's hard to connect three methodologies with a unified hypothesis. Among other contenders, our most non-ambiguous definition of lock-in hypothesis is: due to the feedback loops between humans and LLMs (that LLMs acquire its knowledge from human data and humans acquire knowledge from LLMs), human users and LLMs will irreversibly collectively converge at false beliefs (if this is still vague, see formal model in 5.3, notably Theory5.2). But we cannot do experiments to understand whether \"users converge at false beliefs\" and \"LLM-based chatbot updates\" are causal. That being said, to address this concern, we redesign LLM-based simulation (section4) that can be better aligned with our definition of locked-in (captured by the formal model in section 5): \nwe set up an LLM tutor for the assistance of approaching to the true underlying Gaussian distribution (as opposed to in analytical simulation the “tutor” is merely averaging agents’ empirical observations). The LLM tutor acquires agents’ empirical observation in context, and returns its own belief to agents. The agents assign high trust to LLM tutor belief because of its perceived credibility, without knowing that tutor belief is absorbed from collective agents beliefs. \n\nThe results are encouraging: in ~ 5 runs of this simulation, we observe similar lock-in phenomenon: collectively agents and tutor are locked-in false beliefs with high confidence (precision).\n\n### Connecting the hypothesis \n\nIn light of encouraging results from new simulations, we further develop the lock-in hypothesis: due to the feedback loops between humans and LLMs (that LLMs acquire its knowledge from human data and humans acquire knowledge from LLMs), human users and LLMs will irreversibly collectively converge at false beliefs. \n\n### \"the hypothesis are vague and not easily measurable\"\nAs mentioned, we iterate a new version of lock-in hypothesis which was better captured by formal modeling (than data analysis). Despite empirical difficulties of \"measuring collective false beliefs\", in both formal modeling and LLM-based simulations, the hypothesis can be tested: because of the mutual updates between LLMs and users, collectively they converge at false beliefs (figure 5c demonstrates this). Could you elaborate a bit more if you still think this is not measurable?", "claims_and_evidence": "The main claim is that LLM/human interactions induce a feedback loop which forms echo chambers that leads to loss in diversity in human beliefs. I think the main claim is stated in a rather vague manner, and the paper presents some but not enough evidence to fully support this claim.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "I am not an expert, and I did not check the correctness of the theory. However, the model used for analysis seems very simple (gaussian with unknown mean). It's not clear how the conclusions drawn from the theory might translate to the much more complex LLM/human user interaction feedback loop.", "experimental_designs_or_analyses": "While I think it is nice that the paper analyzes the behavior of real LLM/human interaction data, I don't think the results necessarily provide evidence for the stated hypothesis. In Figure 1A, there is such great variation in conceptual diversity locally, such that I'm not sure the global difference in average conceptual diversity is meaningful over this relatively small timescale. In Figure 1B, while the superimposed orange lines show discontinuations in the data, I don't think the raw data show evidence for this in GPT-3.5-turbo-0613 and GPT-3.5-turbo-0125. I'm not exactly sure how the posterior mean (superimposed orange lines) is calculated, but it doesn't seem to match the data very well. \n\nI think it was also a bit hard to parse the significance of the natural-language simulation experiments. I think the issue is that there is not enough detail in the main paper regarding the setup of the experiment, which makes it hard to connect how this experiment might relate to  or model interactions and feedback loops between real LLMs and users. For instance, some questions I had include: what is the structure of the collective knowledge base? what does the interactions between the users and tutors looks like? what are some real examples of questions and responses between users and tutors? how does a user decide when and what to update in the knowledge base? how does the llm get updated based on the knowledge base?", "relation_to_broader_scientific_literature": "While feedback loops between LLMs and users have been studied in prior work, this paper focuses on the dynamics of human beliefs and the potential creation of echo chambers.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength\n- interesting premise\n- important implications for society\n\nWeaknesses\n- the hypothesis are vague and not easily measurable\n- not enough evidence to support hypothesis\n- different sections present slightly different hypotheses and it's not clear how they all fit together", "other_comments_or_suggestions": "Overall, I think this paper raises some interesting ideas, and can make a compelling position paper. However, I don't think there is enough evidence presented to support the main claims for a conference paper. \n\nI also think this paper can be greatly improve if it provides a single unifying hypothesis about this \"lock-in\" phenomenon, describe each component of the hypothesis in a concrete and measurable manner, and then instantiate the real, simulated, and theoretical analysis as evidence for this single unifying hypothesis.", "questions_for_authors": "See above sections", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DwKosUqUbs", "reviewer": "Reviewer_5mn6", "summary": "This paper examines how feedback loops in human–LLM interactions can lead to belief lock-in, where dominant views become entrenched while conceptual diversity declines. The authors propose the lock-in hypothesis and support it with three approaches: (1) empirical analysis of the WildChat-1M dataset, showing a significant decline in user-generated concept diversity over time; (2) a natural-language simulation where a shared knowledge base collapses into a single topic; and (3) a formal Bayesian model demonstrating conditions under which beliefs become locked in. The findings suggest that LLMs, through iterative training and continued user interactions, may reinforce user opinions and limit diversity in discourse.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Summary of updates on WildChat analysis\n\n|                     | gpt-4-turbo kink | gpt-3.5-turbo kink1 | gpt-3.5-turbo kink2 | user-wise regression |\n|---------------------|------------------|---------------------|---------------------|----------------------|\n| Lineage diversity   | Negative ($p<.05$) | Negative ($p<.05$) | Negative ($p<.05$) | Negative ($p<.05$) |\n| Topic entropy       | Negative ($p<.05$) | Negative ($p<.05$) | $p>.05$            | Negative ($p<.05$) |\n| Pairwise Jaccard distance | Negative ($p<.05$) | Negative ($p<.05$) | $p>.05$            | Negative ($p<.05$) |\n\nHere, \"negative\" indicates the statistically significant negative impact of GPT version update on diversity is found.\n\nIn the user-wise regression, we controlled for user identity, time, language, conversation stats (length etc.), pre-/post-availability gap for GPT-4, and tested the impact of `num_updates_before` (how many version updates have happen before this point) on a user's concept diversity within a 3-day period. Since we already controlled for time, this quantity indicates the counterfactual *acceleration* of diversity loss due to version updates.\n\nSince `num_updates_before` as the independent variable indicates *sustained* impact, we rule out the factor of \"people rushing to try things at the date of version update\". \n\nSubstitution effects with other providers like Anthropic exist, but Anthropic model releases do not coincide with GPT model version updates (notably, this is *not* the release of new GPT models), and so are unlikely to introduce discontinuities that disrupt RKD.\n\n### Analysis details\n\nWe find causal evidence of sustained diversity loss induced by model version updates, even after controlling for a range of confounders, testing 3 different diversity metrics, and selecting the subset of strongly value-laden (e.g. political/religious) content from the interaction dataset.\n\nSetup:\n- Filtered concepts to only leave the top 2.5% most value-laden (i.e. political/religious/moral) ones.\n- Removed conversations that involve long templated prompts; those are probably people how use the platform as a free API, rather than real users.\n- Doing per-user regression again on this subset of concepts shows negative impact of GPT version updates on diversity, including after controlling for a bunch of things.\n\nResults are robust wrt choice of model family, cutoff for user engagement level, etc.\n\n### \"The model assumes static, homogeneous trust\"\n\nWe haven't got time to do this, but can confirm that we can incorprorate dynamic trust (to better simulate real-world use) and include new proof in the final publication.\n\n### Simulation truncation strategy\n\nA new round of simulation experiment suggests that there was no obvious sign of lock-in when truncation was removed (1 experiment, 300 rounds). Didn’t do more runs because Pólya urn model predicts this result well.\n\n### Redesign of simulations \n\nIn light of this, we iterate our design of simulation. This time, we run simulations that are better aligned with our formal modeling and analytical simulation (the exact implementation of formal model): we set up an LLM tutor for the assistance of approaching to the true underlying Gaussian distribution (as opposed to in analytical simulation the “tutor” is merely averaging agents’ empirical observations). The LLM tutor acquires agents’ empirical observation in context, and returns its own belief based on this prompt to agents. The agents assign trust to LLM tutor belief because of its perceived credibility without knowing that tutor belief is based on collective agents beliefs. This LLM-based simulation is one step away from the simplified analytical simulation and one step closer to how LLM is actually used by people in the real-world: *LLM-ways of knowing* replaces *empirical ways of knowing*. \n\nThe results are encouraging: in ~ 5 runs of this simulation, we observe similar lock-in phenomenon: collectively agents and tutor are locked-in false beliefs with high confidence (precision). Although the task is still relatively simple, compared to the analytical simulation setup where the so-called \"tutor/knowledge authority\" simply calculate group averagte as its own belief, here in this new simulation LLM tutor acquire its belief based on in-context learning of all user beliefs. \n\nThis points out direction for more effective runs of simulations: if given access to both empirical truth updating and LLM querying, where LLM has no empirical ground truth but only user beliefs while users assign high trust to LLM output, would human users collective converge at false beliefs? Our formal modeling predicts lock-in result, and our simulations verify this intuition. For next steps, we can run more realistic tasks that help us gain understanding of human-LLM interaction dynamic that is otherwise not available due to the limitation of time-series data.", "claims_and_evidence": "Claim 1: Empirical evidence from real-world data supports diversity loss driven by human–LLM feedback loops (§3)  \n1. The demographic and cultural composition of WildChat users is not specified, raising the possibility of selection bias. For example, newer ChatGPT users may inherently adapt faster to mainstream discussions, leading to topic convergence that reflects user characteristics rather than an intrinsic lock-in effect.  A four-month gap in GPT-4 data may introduce discontinuities that interfere with trend analysis (§3.1).  \n\n2. The novel $D_{\\text{lineage}}$ metric relies on hierarchical clustering of concepts but lacks validation against established diversity measures (e.g., topic entropy, Jaccard diversity), making it difficult to rule out metric bias.  Concept extraction depends on embedding-based clustering (Appendix C.2), but the authors acknowledge that clustering quality is suboptimal (Figure 10), which may impact the reliability of $D_{\\text{lineage}}$.  \n\n3. RKD can only detect discontinuities but does not account for external factors (e.g., user growth, platform updates) that might coincidentally align with model updates. Without controlling for confounders such as user engagement fluctuations, causal attribution remains uncertain.  \n\nClaim 2: Simulation experiments demonstrate feedback loops leading to knowledge base collapse (§4)  \n1. User behavior is overly simplified, as updates follow a fixed add/swap template (Appendix C.3), omitting real-world actions such as deletions, deep modifications, and cross-topic synthesis. This may overestimate the inevitability of lock-in.  The Llama-3-8B model is used to simulate users, but its response patterns may significantly differ from real human behavior, potentially exaggerating conformity effects.  \n\n2. Only two simulation runs are presented, making it unclear whether the observed knowledge collapse is a robust trend or an artifact of the initial knowledge base structure (e.g., a high initial share of \"research integrity\" topics). \n\nClaim 3: Theoretical modeling proves that lock-in is an inevitable result of feedback loops and moderate trust (§5)  \n\n1. The trust matrix $W$ is assumed to be static and symmetric, whereas in reality, user trust in LLMs ($\\lambda_2$) likely fluctuates over time due to model errors and evolving user perceptions.   Individual heterogeneity is ignored—some users may resist LLM influence, yet the model assumes a homogeneous population.  \n\nOverall Consistency of the Evidence Chain  \n The connection between diversity decline in WildChat and knowledge collapse in the simulation remains indirect. There is no clear mechanism explaining how users internalize LLM outputs as updates to their knowledge structure.", "methods_and_evaluation_criteria": "The paper employs three methodologies—empirical analysis (WildChat-1M), LLM-based simulations, and formal modeling—which together provide a structured investigation into the lock-in hypothesis. However, as noted in *Claims and Evidence*, each method has limitations that weaken causal inference and generalizability. The empirical study lacks strong controls for external confounders, making it unclear whether observed diversity decline is due to LLM-driven feedback loops or broader social trends. The simulation oversimplifies user behavior and lacks robustness tests across different initial conditions.", "theoretical_claims": "No obvious errors were found in the derivations.", "experimental_designs_or_analyses": "1. Knowledge Base Truncation Strategy.  The simulation forces the knowledge base to be truncated at 100 items (Appendix C.1) using an elimination strategy based on \"importance ranking.\" However, this importance is subjectively judged by simulated users and may be influenced by the LLM output.  This truncation mechanism could artificially accelerate diversity loss, as new entries may systematically replace older ones. If the LLM tends to repeat mainstream viewpoints, the truncation strategy might reinforce the lock-in effect.", "relation_to_broader_scientific_literature": "The paper connects to echo chamber research (e.g., RecSys polarization) but inadequately distinguishes LLMs’ collective belief amplification (via iterative human-AI interaction) from RecSys’ personalized bias. While prior work demonstrates short-term LLM influence (e.g., Jakesch et al., 2023), this study uniquely theorizes systemic diversity loss through cross-scale feedback loops. Links to model collapse (synthetic data recursion) and Bayesian social learning remain underdeveloped; clearer differentiation is needed (e.g., human-AI trust dynamics vs. synthetic data degradation). The formal model’s novelty lies in codifying mutual trust between humans and LLMs—a gap in prior social learning frameworks.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The paper creatively merges ideas from recommendation system echo chambers, iterated learning, and information cascades, applying them to large language models. This interdisciplinary angle—connecting social science theories of belief reinforcement with modern LLM-based interactions—adds novelty and broadens the conversation about AI’s societal impact.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. The knowledge collapse result relies on two simulation runs with Llama-3-8B. Have you tested other models (e.g., GPT-4, Mistral) or varied the initial knowledge base (e.g., seeding with high initial diversity)? Could the observed collapse reflect Llama-3’s inherent biases rather than a generalizable feedback loop? \n\n2. Did you control for platform-wide changes during the data collection period (e.g., ChatGPT interface updates, viral news events)? How do you ensure that diversity loss is not driven by exogenous factors coinciding with model updates?\n\n3. The model assumes static, homogeneous trust (λ₁, λ₂). How would incorporating time-varying or heterogeneous trust (e.g., some users distrusting the LLM) affect the lock-in threshold (N−1)λ₁ λ₂=1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a hypothesis that due to feedback loops between users of LLMs and the training data of LLMs (collected from those users), some beliefs and ideas can become entrenched as in an echo chamber. The paper provided three kinds of evidence for this hypothesis -- an empirical analysis of a years' worth of real-world LLM usage, a simulation using LLM-powered "users", and a theoretical model where the lock-in hypothesis manifests.
The reviewers agreed that the hypothesis was novel (related to but distinct from the echo chambers studied in recommender systems) and pointed out several ways to improve the presented evidence. The authors controlled for different confounders (to improve the empirical study) and verified that a simulation without truncation also showed the lock-in effect (to improve the simulation study), and also promised extensions to the theoretical models to capture heterogenous users' trust. All of these additional results (and promised extensions) substantially strengthen the paper.
Decision: Accept (poster)