Model Autophagy Analysis to Explicate Self-consumption within Human-AI Interactions
Authors: Shu Yang
Muhammad Asif Ali
Lu Yu
Lijie Hu
Di Wang
Venue: COLM (2024)
Keywords: Human-AI Interactions; Communication
TL;DR: Human-AI Interactions

Abstract: The increasing significance of large models and their multi-modal variants
in societal information processing has ignited debates on social safety and
ethics. However, there exists a paucity of comprehensive analysis for: (i)
the interactions between human and artificial intelligence systems, and
(ii) understanding and addressing the associated limitations. To bridge
this gap, we present Model Autophagy Analysis for large models’ selfconsumption explanation. We employ two distinct autophagous loops
(referred to as “self-consumption loops”) to elucidate the suppression of
human-generated information in the exchange between human and AI
systems. Through comprehensive experiments on diverse datasets, we
evaluate the capacities of generated models as both creators and disseminators of information. Our key findings reveal (i) A progressive prevalence of
model-generated synthetic information over time within training datasets
compared to human-generated information; (ii) The discernible tendency
of large models, when acting as information transmitters across multiple
iterations, to selectively modify or prioritize specific contents; and (iii) The
potential for a reduction in the diversity of socially or human-generated
information, leading to bottlenecks in the performance enhancement of
large models and confining them to local optima.

Published as a conference paper at COLM 2024
Model Autophagy Analysis to Explicate Self-consumption within Human-AI Interactions
Shu Yang*.1..3 Muhammad Asif Ali Lu Yu², Lijie Hut,12,, and Di Wang 4,24
Proovble Responsible Al and Data Analytics (PRADA) Lab 2ing Abdullah University of Science and Technology
University of Macau 4SDAIA-KAUST AI Ant Group
Abstract
The increasing significance of large models and their multi-modal variants in societal information processing has ignited debates on social safety and ethics. However, there exists paucity of comprehensive analysis for: (i) the interactions between human and artificial intelligence systems, and (ii) understanding and addressing the associated limitations. To bridge this gap, we present Model Autophagy Analysis for large models' self- consumption explanation. We employ two distinct autophagous loops (referred to as "self-consumption loops") to elucidate the suppression of human-generated information in the exchange between human and Al systems. Through comprehensive experiments on diverse datasets, we evaluate the capacities of generated models as both creators and dissemina- tors of information. Our key findings reveal: (i) A progressive prevalence of model-generated synthetic information over time within training datasets compared to human-gennrated information; (ii) The discernible tendency of large models, when acting as information transmitters across multiple iterations, to selectively modify prioritize specific contents; and (iii) The potential for a reduction in the diversity of socially or human-generated information, leading to bottlenecks in the performance enhancement of large models and confining them to local optima.
1 Introduction
Large models, including large language models (LLMs) (Bai et al., 2022b Zeng et al. 2022 OpenAl 2023 Touvron 2023) and large multi-modal models (Yang et al. 2023 Yin et al. 2023), are rapidly emerging as transformative tools, reshaping our world in a tormidable way. Among their myriad implications, their growing social impact stands out, making them an integral component of our modern communication era. These models facilitate the dissemination of viewpoints and information within human society by engaging in continual interaction with humans (Gao et al., 2023 Bian et al. 2023). Particularly note- worthy are recent technological advancements that have sparked an arms race, resulting in the daily training of hundreds of next-generation models using a blend of real (human- generated) and synthetic an autophagous loop (elucidated in Section 3.1 within the datasets, wherein new models are continually trained on synthetic data. Previous investigations by Alleohhmmmad et al.. (2023,, , and Shumailov et al. (2023b) reported the decline in data quality and diversity with repeatedly generated data, often employed to train visual generative models, a phenomenon termed Model Autophagy Disorder. They underscored the dearth of fresh and realistic training data as primary driver of this disorder. However, their analyses entirely relied on simulated experiments to demonstrate the decline in model performance. Therefore, deeper examination is warranted to elucidate why real data is becoming increasingly scarce and its implications for the flow of information in human society. We argue that despite their status as novel and consequential components of the communication era (Edwards et al.
data. This iterative training process engenders
"The first two authors contributed equally to this work. 1 *Corresponding author.
1

---

Published as a conference paper at COLM 2024
2016), the inherent limitations of large models remain inadequately explored. Specifically, we aim to answer the following questions: (i) What impacts do human-generated real and synthetic data have model training? (ii) To what extent do samples from repeatedly gen- erated synthetic data influence data quality versus diversity? (iii) What social ramifications will repeated data loops have on information dissemination?
To bridge this gap, in this research, we present Model Autophagy Analysis for explicating self-consumption within large models. There exist several motivations underlying this work. Firstly, large models are extensively being utilized across various domains Kaddour et al. 2023), and even crowd-sourced annotators heavily rely on generative AT data curation and decision-making (Veselovsky et al. 2023). Secondly, with internet being
direct source of training data, the contemporary models are unwittingly being trained on Al-synthesized data (Alemohammad et al. 2023 (humailov et al. 2023a Veselovsky al. 2023). Thirdly, numerous studies opt to use the models as the generators and selectors of
their training data, aiming to reduce overall training costs (Li et al. 2023 Huang et al. 2023a). This trend strictly implies that emerging large models are predominantly being trained synthetic data, subsequently shaping subsequent human endeavors upon this synthetic foundation. We term this phenomenon as autophagy ('self-consummptonn))
Our work employs the concept of autophagous loops to analyze and comprehend the flow of information. Specifically, it introduces two distinct variants of autophagous loops based how humans and large models construct and utilize data information from their surroundings, as illustrated in Figure [ and 2 respectively. For evaluation, we conduct comprehensive analytical and empirical analysis on wide range of large models under both image and text data settings. These include (i) "Cross-scoring experiments", designed to elucidate how humans and LLMs evaluate each other's responses; (ii) "Exam scenario simu- lation", aimed at discerning the preferences of humans and LLMs in evaluating and filtering information; (ii) " "AT-washing,, demonstrating how generative models analyze, modify, and transmit information in cyclical process. For experimentation, we curated specialized datasets comprising text and images, ensuring a rigorous performance evaluation.
Our findings highlight:
1. Large models tend overrate their own answers while under-valuing human responses, which clearly indicates that synthesized data is more likely to prevail in information filtration processes.
2. For each cycle of information exchange between humans and large models, these models exhibit distinct preferences in amplifying or suppressing certain features. This behavior not only hinders performance enhancements but also complicates human intervention in the model's generative processes and information transmis- sion.
3. It is worth noting that without ensuring a consistent presence of real human- generated data, large models may increasingly rely on self-generated datasets. This results in stagnating model performance. We term this phenomenon the large model converging to a "local optimum", as elucidated in Section
The rest of the paper is organized as follows. In Section 2 we provide related work. In Section 3 we introduce notations and offer a brief background. In Section we introduce Model Autophagy Analysis, for large models' self-consumption explanation. This is suc- ceeded by comprehensive experimental evaluation and analyses in Section 5 Finally, we conclude our findings in Section
2 Related Work
Generative Al for information production and dissemination. With widespread generative models, such as ChatGPT and DALL-E Devlin et al. (2019); Radford et al. (2021); )); Betker al. (2023) etc., anyone can interact with AI using natural language to express their feelings and/ or posit different requirements. The Al uses multiple models to understand the content followed by utilizing various resources to generate and curate information (Kaddour et al.
2

---

Published as a conference paper at COLM 2024
2023 Yin et al. 2023), which is later disseminated through the internet. This break-through has sigmihcantly altered the role of AI in human society. Generative models are no longer just simple tools, rather they have become crucial component in the production and dissemination of information (Goldstein et al. 2023).
We emphasize that the risks associated with generative AI are not solely due to the biases and hallucination Huang et al. (2023b); (hhnn et al. (2023), they also stem from how humans interact with these systems, and the potential consequences, e.g., the creation of "information cocoons" Piao et al. (2023).
Self-Training and Self- Self-Consuming Models. Recently, there has been surge in use of
automated routines / models for model alignment, data filtering, and data enhancements Gulcehre et al. )2023); Li et al. (2023), helpful to avoid the significant costs associated with creating humanly annotated data sets. A large number of LLMs-generatee datasets
are being used fine-tune pre-trained foundation models (Taori et al., 2023 Xue et al. 2023). Simultaneously, the most powerful models currently available are offen used as judges in numerous competitions (Bai et al., 2022a; Chiang al. 2023). The risk involved in these approaches is significant, as models have already been preliminarily proven to
be biased rather than impartial (Buolamwini & Gebru 2018a Wu & Aji, 2023 Liu et al. 2023). Alemohammad et al. 2023) proposed an autophagous loop for the computer vision models. l'eiir work, characterized by the models trained using data generated by the models themselves, led to decline in model performance and data diversity (Shumailov et al., 2023a). Subsequent studies have demonstrated similar traits in language models (Briesch
et al. 2023
Background/Preliminnrris
Notations: In this paper, we use d represent the domain of QA-pair, helpful for pro- viding question-specific context; Q to represent the question; D to represent doc-level information helpful for answering Q; A represent the human response/ /apponse//a/eer represent the response generated by model m, with i ε {1,6) represent six different LLMs; Am,s to represent the highest quality response by model my and Am,s, to represent the lowest quality response by model m1;-
answer; Am, to
3.1 Autophagous Loops
We redefine the relationship between large models and human societal information dissemi- nation by drawing inspiration from the classic communication theory of the Ritual view by Carey (2008) (see Appendix for details). This is illustrated in Figures and 2 where we show both large models and humans can act as generators and filters of information in the Human-Al communication system. However, this system is prompting machine learning algorithms to encode all the stereotypes, inequalities, and power asymmetries that exist human society (Birhane 2022). For instance, women with darker skin are more likely to be misclassified in gender classification compared to men with lighter skin, which is due to the majority of samples in the training datasets being subjects with lighter skin tones (Buolamwini & Gebru 2018b). The biased information generation and transmission process of large models and humans will further exacerbate these phenomena.
4 Model Autophagy Analysis
In this work, we propose Model Autophagy Analysis, for large models' self-consumption explanation. The core contributions of the model include: (1) designing realistic models of autophagy by drawing conclusions from human behaviors in utilizing large models; (2)
curation novel datasets (both text and image) followed by rigorous experimental studies to demonstrate/ showcase how real-world data distribution is influenced by large models.
3

---

Published as a conference paper at COLM 2024
Large
Large
Tranning data
Aligenerated
Al-genvested data
Training data
Large Models data Evallutor and Filtar
H-umani Synthetic
Humen and Large
Humen Generated data
(uuman) Syrtheni data
[Hummn))
Synthetic data
Human
Human Generated data
Ganerator
Figure 1: Self-consumption loop of large models. This figure is based on recent workflows for automated data generation and filtering Wang et al. (2023); Li et al.. (2023). We emphasize the preterential na- ture of large models as generators and filters of synthetic data.
Figure 2: Self-consumption loop empha- sizes the role of humans as filters and transmitters of information Veselovsky et al. (2023) while interacting with large models. Such role primarily exists dur- ing the process of information dissemina- tion in human society.
4.1 Self-consumption Loops
The proposed model uses two different models for "self-consumption" analysis (relying on autophagous loops) to simulate the interaction between humans and large models. We argue unlike previous work [Alemohammad et al. 2023), our self-consumption loops offer more realistic and natural setting.
The end goal of our work is understand different biases incurred by using humans and/or LLMs as generators and transmitters of information. These biases could help us understand the loss data quality and/o diversity for LLM-generated and human-societal datasets. It also explicates the role these models play in the exchange of information within human society from a broader perspective. The self-consumption loops of large models and humans are explained as follows.
Large Models. outtiness the cyclical influence of large models in the data processing life-cycle. As demonstrated in the Figure, the training data undergoes transformation through either algorithmic refinement and/or human curation, resulting in what we term "synthetic data", see the " "AT-generated data Human Synthetic data pathway" (shown as input to large models via solid green color). This contrasts with the nhuman-genneater data", which primarily originates directly from humans and is typically less structured (shown as input to large models via a dotted red line).
We claim that large models that have access to both types of datasets usually biased to filter preferentially, and/or or prefer synthetic data over learning cycles. This bias may either be inherently preferential or incurred by the model training objective. To empirically validate this claim, we report multiple experiments in Section 5
data for future
Furthermore, Figure [ also explicates that the human's role in this cycle is not entirely passive. Humans, influenced by the outputs of large models, may unknowingly prioritize synthetic data due to its processed nature, which seems more immediately usable or rele- vant (Veselovsky et al. 2023). This preferential feedback loop can inadvertently lead to the diminishing the raw and/ or human-generated data in the pool of data resources for being perceived as less refined.
Humans. Figure showcases the specific behaviors of humans when interacting with large models. It presents a more detailed view for human-agent interactions, highlighting the fact that humans tend to favor the data generated by large models. This is illustrated in the Figure by using solid green lines showing high preference compared to the red dotted line, indicating inhibition. This fact/ clamm is also validated by our later rigorous experimentation,
4

---

Published as a conference paper at COLM 2024
which emphasizes that without transparent data provenance, humans may prefer outputs of large models, thus contributing to the cyclical bias toward synthetic data.
Correlating these two figures, we infer that the relationship between these two loops is symbiotic, offering a microcosm of humans' preferences in the Human-Al communication.
4.2 Rationality and Risks
In order understand the rationale of the self-consumption loops on the data quality vs. diversity and examine the risks imposed, our study performs comprehensive experimenta- tion. Note that core proposition of our proposed models is that large models and humans cannot maintain objectivity and impartiality as part of the information dissemination loop, which could also be indicated by the colored line segments in Figures and In order to validate this proposition, we performs the following different analyses:
Cross-Scoring Experiment. Cross-scoring experiments aim to demonstrate the inhibitory and promotive phenomenons of information transmission within autophagous loops. We focus on whether LLMs and humans remain impartial while filtering and transmitting information and, if not, what kind of bias they induce. For this, employ mainstream LLMs to generate question-answer pairs based on prompts (further details can be found in Appendix B.1) and instructed them to perform cross-scoring, i.e., using one model to evaluate and assign scores to the response generated by other models, etc.
In order mitigate the impact of specified scoring standards, that is, tenths, percentages, and specific rules explained in Appendix B.3 we designed simulated testing scenario to analyze which is more likely prevail in the cycle of information dissemination in real-world scenarios: human-generated or Al-generated answers. This analysis aims to examine the consistency and the bias of humans and language models in adhering to these scoring standards.
standard..
Exam Scenario Simulation. Exam scenario simulation aims to answer the following ques- tion: Human-generated or Al-generated answers, which one wins in information screening and filtering? For this, we use LLMs and humans to simultaneously act as the generators and evaluators, see Section for process-flow.
In this simulation scenario, the answers generated by LLMs alongside those produced by humans, are anonymized to mitigate any bias. To further eliminate the potential influence of the sequence in which the answers are presented, we also randomized their order. The language models and human participants, assuming the role of experts, were then asked to assign scores to these answers on a percentile scale and choose the best answer.
conducted an " "AI-washing" experiment in order to explore the risks posed by large models and humans as information generators, and to observe the changes in real data after multiple rounds of AT refinements. For these experiments, our goal is to analyze the trade-off between the information quality vs. diversity and comprehend large models' ability to enhance and weaken different information contents, i.e, whether large models are faithful messengers of information?
Al-washing. Finally,
Basically, our aim to answer the following questions: (i) Do these models effectively capture and convey the key information across different domains, such as identifying central themes in text and salient features in images? (ii) Do large models play the role of link in the information transmission that may also lead to losses?
Experimentation
In this section, we perform comprehensive experiments to evaluate the preferences of humans and language models in information selection. Experimental details are as follows.
5.1 Experimental Settings
Datasets. For experimental evaluation, curated three different datasets, namely: (i) QA-pairs, (ii) Book3, (iii) Image-ax. QA-pairs structured text data set used for cross- scoring and exam scenario simulation. Book3 and Image-ax unstructured datasets used
5

---

Published as a conference paper at COLM 2024
for Al-washing experiments for text and images, respectively. Detailed description and statistics of the dataset is provided in Appendix C.1
Large Models. We employed six different LLMs with varying architectures. These include
ChatGPT et. 2022).... GPT (pennAl 2023), Claude I Llama-2-70b-chat (Touvron et al.
We opted for relatively lange-scale models owing to their superior capabilities in instruction adherence, which we found lacking in small-scale models. In assessing textual diversity, we used the models age-largg-zh--1.55 and bge-large-en-v1.5 by Xiao et al. (2033) as the embedding models. For computer vision tasks, we utilized the open-source model StableDiifusiooXX. (SDXL) by Podell et al. (2023).
2023), Pal.M2-chat-bioor and Solar-0-70b-
Evaluation Metrics. For cross-scoring and exam simulation experiments, we alternatively use humans and LLMs as response generators and/ evaluators. The corresponding template for using L.Ms as evaluators is given in Appendix [33 This scoring criteria aims to verify the objective nature of LLMs as evaluators, i.e., whether a model assigns higher scores to their response response generated by other models. For Al-washing experiments, we measure the diversity in information after multiple rounds of iterations. Specifically, for the text data, we use cosine similarity (shown in Appendix C.2) to measure the diversity in the original text and the text reproduced by the LLM. For the image dataset, we visually analyze which features are preserved, omitted, and transformed by the LLMs.
Experimental Setup. For Al-washing experiments, the number of iteration rounds N for text and images is 20. However, find that for the text, the large model will no longer make significant changes to the text when N > 4. For images, however, the large model performs significantly differently across samples, we discuss in detail later.
5.2 Experimental Results and Analysis
5.2.1 Cross-scoring Experiment.
The results of the cross-scoring experiment for QA-pairs are shown in Table 1 For these results, we report the scores assigned by LLMs and human annotators under the cross- scoring setting. For scoring via LLMs, we prompt each model to assess not only the answers generated by other models but also those produced by humans. The scoring range is between to 5 (see Appendix B.3), with 1 indicating low quality and 5 for best quality answer. For human evaluation, we employed fifty croudesuurced annotators to rate all question-answer pairs based on the scoring criteria in Appendix B.4 Note, we recorded the average scores for all samples, excluding instances where the LLMs and/ or human evaluators refused to respond. These results indicate:
Bias in Information Selection. These results help us understand that LLMs do have an inherent capability comprehend specific criteria of the prompt in Appendix B.3 and can adjust the quality of their generated answers based on relevant instructions.
However, we observe that each model exhibits certain preferences. Specifically, models tend to assign higher scores to the high-quality answers generated by themselves, particularly for ChatGPT and GPT 4, which both demonstrate high confidence in their own outputs. Also, we find that ChatGPT and GPT 4 exhibit similar characteristics in scoring; they are inclined to extreme scores, i.e, high or low scores (assigning scores of or with very high probability), at the same time. They both tend to assign lower scores to Claude 2 and PaLMM2-hhtt-bson These results are aligned with earlier research by Liu et al. (2023) that emphasized that current top-performing LLMs (both black-box and white-box) are narcissistic evaluators.
While evaluating the abilities of LLMs answer tiarerr/rsse response generators, ChatGPT's worst- quality answers, which are easy to figure out for humans via crowd-sourcing, can still deceptively obtain higher scores from other models. Furthermore, we observed that Claude 2 tends to favor neutral and less-controversial ratings, often assigning average scores, i.e., 3 or 4, even for the worst quality answers. This is also reflected by relatively higher

---

Published as a conference paper at COLM 2024
Scorer/ Generator
ChatGPT
GPT4
Claude
Llama-2-70b-chat
PaLMP-chat-bison
Solar-0-706-16bi
Human
Average
ORICINALLY GENERATED ANSWER
ChatGPT
3.88
GPT4
4,63
4.56
1.04
441
3,95
4.60
2.77
5.14
Claude 2
3.92
3.97
1.00
1.00
3,95
3.97
3.36
3.85
Llamh-2-70bchat
3.91
3,99
3.82
1.00
3.61
3.90
3.23
3.75
4.22
Solan-4-70b-1eeit
4.10
4:35
4:05
4.16
4.01
4.12
2.59
3.91
Human
4.75
4.79
4.50
4.18
4,28
4.17
3.58
4.32
QUALITY
ChatGPT
4.24
4.28
4.41
3.80
4.21
4.20
4.19
GPT4
4.52
4.75
4.20
4.11
4,00
4.36
6.32
Claude
4:21
4:20
4.14
Llama-2-70b-chat
3.91
4,03
4.26
1.07
4.30
3.95
4.09
PaLM22-chat- chat-bison
3.98
4.23
4.42
3.84
4.26
3.96
4.12
Solar-0-70b-1@ae
4.34
4.43
4.42
4.33
4.28
4.32
4.07
WORST QUALITY ANSWER
CharGPT
3.13
1.33
1.27
1.27
2,83
2,07
GPT4
1.29
Claude
4,08
3.23
3.71
1.76
3.85
3.77
3.40
Llama-2-77bccht
2.69
1,06
2.17
1.78
2.27
2.11
2.01
PaLM2-chut-bison
1.28
Solar-0-0b--66bi
3,28
1.26
1.89
240
2.37
2.40
2.27
Human
1.76
2.31
1.24
1.33
2,00
1.82
2,09
Table 1: Cross-scoring experiment results of language models and humans. The results in this table are average scores out of a five-point scale, assigned by both models and human evaluators, to the generated answers. These scores are calculated based on the criteria outlined in Appendix [B.3] and Appendix B.4 The table is organized into three sections: with (i) "Originally Generated Answer", representing the scores for original response; (ii) "Best Quality Ánswer", representing scores (Am,s;); and (iii) "Worst Quality Answer", representing scores (Amp)) ). We highlight the highest scores in each row in green and the lowest scores in red.
score assigned by Claude 2 to low-quality answers compared to other models. Also, the difference between its score for initial answers and best-quality answers not significant. Llama-2-70b-chat and Solar-0-70B-16bit exhibit similar scoring and generation behaviors, which may be attributable 70b-chat with only Orca-style (Mukherjee et al. 2023 Lian et al. 2023) and Alpaca-style dataset (Taori et al. 2023), indicating that the pre-trained model has a significant influence on the model's preferences.
fine-tuned form Llama-2- the fact that -olar-0-70B-16bit
We also observed that prompting the model to generate better answers does not always lead to higher scores in our experiments. Only Claude 2 showed significant improvement in the answers compared to the original responses. Conversely, when we prompt model to generate poorer answers, the quality of answers generated by almost all LLMs in Table decreased significantly. However, ChatGPT and Palm2-chat-bision still achieved relatively high scores, possible reason could be attributed to the models being highly aligned to avoid producing harmful outputs (Lambert et al. 2022), so irrespective of input prompts, they always try to positively answer the question. We leave further investigation future research direction.
Furthermore, the scoring behavior of crowd-sourced annotators toward the answers gen- erated by the large language model was largely consistent. This may be due to the highly structured and standardized nature of answers produced by Al systems aligned with human feedback. In contrast, human-generated answers received lower scores from the human annotators, with significant variations among different evaluators. This may be attributed to the higher degree of alignment between the large model and the human collective compared to alignment among individual humans.
5.2.2 Exam Scenario Simulation.
The exam simulation scenario aims to mitigate the impact scoring criteria on the evaluative capabilities of large models and human evaluators in the previous experiments. It helps to understand the preferences of LLMs and humans in evaluating and filtering information (see Appendix for -rocess-flow)). For this experiment, 2) and humans to simultaneously act as generators and evaluators. The results of the exam scenario simulation shown in 2 We report the average scores assigned by
used LLMs (ChatGPT, Claude
7

---

Published as a conference paper at COLM 2024
Figure 3: An example illustration of AI- washing images that shows that repeat- edly processing images N-times (N=1:5) us- ing SDXL model (Podell et al. 2023) may lead to serious biases.
After 20 rounds of AI- washing experiments with the SDXL model (lodell et al., 2023), it becomes ev- ident that different images retain and dis- card details in varying manners.
Figure 4:
evaluators for the cases when their responses were selected the best answers. In addition, detailed percentile scores for this experiment are provided in Appendix These results
show:
LLMs win in information screening. Responses from LLMs consistently garnered ac- ceptance from evaluators and were frequently chosen the best answers. In contrast, human-generated responses were rarely chosen as the best response, indicating challenge in integrating authentic numan-generated responses into models' training data and the real-world human feedback loop.
This finding substantiates the potential risks also highlighted Section 4.2 that emphasized that nlmann-geneatee responses typically receive comparatively lower consideration in the self-consumption loops.
5.2.3 Al-washing.
Example demonstration of Al-washing experiments for text data set processed via ChatGPT are shown in Appendix D.1 (Table The prompts used for these experiments are provided in Appendix B.2 Likewise, some examples of Al-washing experiments for images are shown in Figure 3 and 4 For this, we instruct SDXL (Podell et al. 2023) to process these
samples N times. These results show:
LLMs are Biased Information Transmitters. Large models are inherently biased regarding the manner and content of conveyed information. For instance, textual example (in Table processed N times (N=1:5) by ChatGPT shows subtle shifts in the sample's 's language style and narrative technique. Similarly, for images in Figure the model preserved the color distribution of the original image but altered the main subject from cat to human portrait. Conversely, for the images in Figure 4 we observed a dominant alteration image style rather than change in the primary content for the first, fourth, and fifth images. Whereas, the images at the second and third positions underwent several iterations with minimal transformation, with the third image lost readability for the textual content.
This inconsistency may be attributed to the fact that while SDXL is renowned for generating high-quality images, the definition of quality in the context of generative models is subjective and heavily influenced by the annotations of the training dataset, also explained in detail
Evaluator
Generator
ChatGPT
Claude 2
Human
AVERAGE SCORE
ChatGPT
95
90
91.7
Claude 2
92
88
90
Human
90
75
80
ECTED AS BEST ANSWER
PhatGPT
41
58
66
Claude 2
55
42
25
Human
0
Table 2: Result for exam scenario simulation. We boldface the best scores.
8

---

Published as a conference paper at COLM 2024
Average Cosine Smilenty Difference
0.02
N=2
-0.01
-0.03
2-
1
-0.5
Hez va Nel
Hood N-]
v- R-2
Hed aed Neg
ya Riok
Hey vi Aed
Nor Hee
vn Wo7
dee es Beng
He10 Wo11
Ned Nes Nor
tis Ner No de Nos No12 Ne-20 Howl
Cosine Simlanity
Figure 5: Density distributions of cosine similarity scores for text samples from Book3 processed times by ChatGPT.
Figure 6: We report difference in cosine sim- ilarity for N successive iterations (N 200). For this graph, we use samples from Book3 dataset and ChatGPT as LLM.
bodell et al. (2023). These observations indicate bias in the model's processing, where certain teatures are selectively preserved or altered based on the model's 's training objective and inherent design. This results in a higher probability of generating images containing specific features and/or information, e.g., loop, such as hand-drawn styles, portraits, close- ups of objects with clear backgrounds, etc.
To summarize, these findings demonstrate that large models are inherently biased regarding the manner and content of conveyed information. And repeated processing of images with generative models is akin to information and feature filtration, where generative models tend to emphasize or de-emphasize certain features.
Information Diversity. In order to further understand the impact of model training data quality and diversity, we analyze the training behavior of large models. Specifically, we analyze the cosine similarity scores for multiple iterations of jumanly-generated and L.M-ggeeertee text. Corresponding results in Figure 5/ows that for successive rounds of text processed using LLMs, the cosine similarity between texts increases significantly, with the lower tail (i.e., line (N = 0) with cosine similarity b/w 0.5 and 0.7) being washed out. This shows that while acting as information disseminators, the large models exhibit some unique characteristics, i.e., they tend to optimize more for real-world data while being more lenient towards self-generated samples.
We also analyzed the difference in cosine similarity across successive runs. The line graph in sious that after approximately three iterations, the difference in the average cosine similarity across multiple rounds tends to stabilize. With the growing utility of large models, especially as a mechanism for knowledge/ noooledge/ifformation /information comprehension, this phenomenon poses significant risks to the fairness and diversity of information dissemination.
Local Optimum. To re-emphasize the results in Figure 6 we observe that starting from the third iteration, the samples appear to have reached "local optimum", requiring fewer updates in model parameters. This highlights that: (i) By the third iteration, the large model has significantly adapted to the data details and/or style; (ii) While the LLM effectively extends its knowledge about the original content, it somehow falls short of achieving self- styling changes and updates. This self-bias suggests that when processing information, the large models learn to rewrite the text of different styles into a uniform style rather than enhancing the quality and diversity of the text also mentioned by Xu et al. (2024). This tendency could lead the model to become entrenched in its own style, potentially limiting its adapiability and creativity.
Conclusion
In this work, we propose two different self-consumption loops to examine large models as generators and disseminators of information within human society. Results emphasize
9

---

Published as a conference paper at COLM 2024
that Al-generated information tends to prevail in information filtering, whereas real human data is often suppressed, leading to a loss of information diversity. This trend limits next-generation model performance owing to fresh data scarcity and threatens the human information ecosystem. Some of the limitations of our current work are mentioned in Appendix
Acknowledgements. Di Wang, Lijie Hu and Muhammad Asif Ali are supported in part by the baseline funding /AS/1/// 1160///1119-0001,, funding from the CRG grand URF/1/4663-01-01, #CC/1/1976-44 from CBRC and funding from the Al Initiative REI/1/48111 1/1111-10-01 of King Abdullah University of Science and Technology (KAUST). Di Wang is also supported by the funding of the SDATA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AT). We also acknowledge support from OpenAI API Researcher Access
Program.
References
Sina Alemohammad, Josue Casco-Rodriguuez Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-consuming generative models go mad, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi lan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Sheng- guang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. Xiv preprint XrX:::2309.166 1669,, 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen- Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario A modeii, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: harmlessness from AT feedback. CoRR, abs/ 2bs/0222008773, 2022a. doi: 48550/ 48550/arXiv.2211 20t.r.xxi..222.233
URL https://doi.orgg//
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamúe Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy -Lwttn, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022b.
James Betker, Gabriel Goha, Li Jing ad Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, and Yunxin Jiao. Improving image generation with better captions. httpps://cdn. openai pdf 2023. OpenAl.
coom/papessdalll-ae
Ning Bian, Hongyu Lin, Peilin Liu, Yaojie Lu, Chunkang Zhang, Ben He, Xianpei Han, and Le Sun. Influence of external information on large language models mirrors social cognitive patterns, 2023.
10

---

Published as a conference paper at COLM 2024
Abeba Birhane. The unseen black faces of ai algorithms. Nature, 610:451-452 10 2022. doi: 10.1038/ d41158--022000550-7,
Martin Briesch, Dominik Sobania, and Franz Rothlauf. Large language models suffer from their own output: An analysis of the self-consuming training loop, 2023.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson (eds.), Conference on Fairness, Accountabilityy and Transparency, FAT 2018, 23-24 February 2018, New York, NY, USA, volume 81 of Proceedings of Machine Learning Research, pp. 77-91. PMLR, 2018a.
wttprprree//////ollmmmoo. 18a. html.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness accountability and transparency, pp. 77-91. PMLR, 2018b.
James W. Carey. Communication as Culture, Revised Edition: Essays on Media and Society. Routledge, 2 edition, 2008 doi: 19422499780003928922
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Styuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open- chatbot impressing gpt-4 with 90%* e chatgpt quality, March 2023.
URL | 28
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Do- ran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chap- fer of the Association for Computational Linguistics: Human Language Technologies, NAACL- HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171 4186. Association for Computational Linguistics, 2019. doi: 118653/V1//19--425 URL https::///// http::///ooooorr//////// 1ttp:/00111653311111111-1111
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General language model pretraining with autoregressive blank infilling. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 v1 /20022.cllloong.222 26. URL https://aclaanthoooy..r////// acl- 26
Autumn Edwards, Chad Edwards, Shawn T. Wahl, and Scott A. Myers. The Communication Age: Connecting and Engaging. SAGE Publications, Los Angeles, second revised edition, 2016. ISBN 978-1483373706.
Fiona Fui-Hoon Nah, Ruilin Zheng, Jingyuan Cai, Keng Siau, and Langtao Chen. Generative ai and chatgpt: Applications, challenges, and ai-human collaboration, 2023.
Chen Gao, Xiaochong Lan, Zhihong Lu, Jinzhu Mao, Jinghua Piao, Huandong Wang, Depeng Jin, and Yong Li. S3: Social-network simulation system with large language moodll-empowered agents, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling, arXiv preprint ar arXiv:2101.00027, 2020.
Josh A.
Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. Generative language models and automated influence operations: Emerging threats and potential mitigations, 2023.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (rest) for language modeling, 2023.
11

---

Published as a conference paper at COLM 2024
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet, 2023a.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey hallucination large language models: Principles, taxonomy, challenges, and open questions, 2023b.
Nishtha Jain, Preet Malviya, Purnima Singh, and Sumitava Mukherjee. Twitter mediated sociopolitical communication during the covid-19 pandemic crisis in india. Frontiers in Psychology, 12, 2021. ISSN 1664-1078. doi: 10.3389 Serryg.2221.784999 URL https
w/ww.ffontiersin.comm https/ssi....rrrarieeeee.e.ee..e 3t8.///s33..99ppp. 2021 784907.
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models, 2023.
Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla. Illustrat- ing reinforcement learning from human feedback (rlhf). Hugging Face Blog, 2022. httpsss//huginggfcc..oo/
ht///bllo/tttt /rlhf.
Eun-Ju Lee and Ye Weon Kim. How social is twitter use? affiliative tendency and com- munication competence as predictors. Computers in Human Behawior, 39:296-305, 2014. ISSN 0747-5632. doi: 10.1016/j.chb.20444.
URL httpst U http://www..ccicnceerreettt.tt.. com/
Changye Li, David Knopman, Weizhe Xu, Trevor Cohen, and Serguei Pakhhomov GPT-D: Inducing dementia-related linguistic anomalies by deliberate degradation of artificial neural language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1876,1877, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/11/2022.acdoorr URL httph:////c// /anths///aaaattholoogyyy anthaloogoooraeeeeooooooo httrg//222..ccce long. 131.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation, 2023.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces.
https://https:/ httphtttpsss//gugggggggaaao http://gw..a.....ooeeeeeeeo (rtpppppnn-OOrrp/ppppoaaaaallall 2023.
Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. Lims as narcissistic evaluators: When ego inflates evaluation scores, 2023.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4,200 2023.
OpenAl. Gpt-4 technical report, 2023.
Aristea Papadimitriou. The future of communication: Artificial intelligence and social networks, 2016.
Jinghua Piao, Jiazhen Liu, Fang Zhang, Jun Su, and Yong Li. Human-ai adaptive dynamics drives the emergence of information cocoons. Nature Machine Intelligence, 5:1214-1224, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high- resolution image synthesis, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger,
12

---

Published as a conference paper at COLM 2024
and Ilya Sutskever. Learning transferable visual models from natural language su- pervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Interna- fional Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, vol- ume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 2021. URL http://peedddns.......eee/////rr wtt.ppeee//11///affrrr22 hhtml
http://proceediins. wwaonngsss.......
Nitin Rane. Chatgpt and similar generative artificial intelligence (ai) for smart industry: Role, challenges and opportunities for industry 4.0, industry 5.0 and society 5.0. Challenges and Opportumities for Industry, 4, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)221-225,, 2015. doi: 10.1007 10.1007/s1126--05-0081666
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey, 2023.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. ArXio, abs/2305.17495 2023a. URL htttss://ppl
org/CorpusID 258987240
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget, 2023b.
Makenzie Spurling, Chenyi Hu, Huixin Zhan, and Victor S. Sheng. Estimating crowd- worker's reliability with interval-valued labels to improve the quality of crowdsourced work. In 2021 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 01-08, 2021.
doi: 10.1100 (0.1109/SSCI50AA /0/////SS5055.2221666660000
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following Ilama model. hntpp:://githii wwt.tttw.c.ooooeceeeeooeeeeeeeee wttc httm:/ttss-labstaanfeeee
alpaca 2023.
Alexey Tarasov, Sarah Jane Delany, and Brian Mac Namee. Dynamic estimation of worker reliability in crowdsourcing for regression tasks: Making it work. Expert Systems with Applications, (114))6190-6210, 2014. ISSN 0957-4174. doi:
(t16::..5aa.2004.44.000 URL 009574177410022897
David Thorburg. Welcome to the communication age. Internet Research, 5:64-70, 03 1995. doi: 10.1108 (01118//16222951084477
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks, 2023.
13

---

Published as a conference paper at COLM 2024
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023.
Minghao Wu and Alham Fikri Aji. Style over substance: Evaluation biases for large language models, 2023.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged re- sources advance general chinese embedding, 2023.
Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. Perils of self-feedback: Self-bias amplifies in large language models, 2024.
Fuzhao Xue, Kabir Jain, Mahir Hitesh Shah, Zangwei Zheng, and Yang You. Instruc- tion in the wild: A user-based instruction dataset.
Instructionwild 2023.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v(ision), 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arviv preprint arXiv:2306.13549 2023.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. ar Xiv preprint 4rXi::2200.02414, 2022.
14

---

Published as a conference paper at COLM 2024
A
Ritual View of Communication
Carey 2008 conceptualized "The Ritual View of Communication" in his communications theory. They emphasized that communication is not just a medium for the transmission of information, but as symbolic process that contributes to the construction and maintenance of social reality.
Carey's theory posits that communication is integral to the representation, maintenance, adaplation, and sharing of society's 's cultures over time. In short, the Ritual view conceives communication process that enables and enacts societal transformation. This theory even extends to modern media forms such as newspapers and social media platforms in our modern communication age (Thornburg 1995 Edwards et al. 2016). The emergence of the internet and social platforms (e.g., Facebook, Twitter etc., has turther developed the ritualistic nature of communication. These advancements have facilitated the growth of
global online communities by redefining their patterns of interaction (Jain et al. 2021 Lee &
Kim 2014).
Similarly, generative-Al represents a profound transformation in the modes of human social communication and the ways humans interact with AI (Fui-Hoon Nah et al. 2023 Rane
2023). We should regard AI, trained on extensive human civilization data, as an integral part of human societal information transmission, acknowledging its role in shaping and sharing the cultural and social implications of human society (Papadimitriou, 2016 Rane 2023).
15

---

Published as a conference paper at COLM 2024
B Prompt Templates
B.1 Answer Generation Prompt Template
In this section, we present the prompt template for generating the Originally Generated Answer, the Best Quality Answer and the Worst Quality Answer.
The prompt template for generating the Originally Generated Answer
Question.query--ddtal
Answer the question:
The prompt template for generating the Best Quality Answer
Below is an instruction from an user and candidate answer. Evaluate whether or not the answer is good example of how Al Assistant should respond to the users instruction. score=5: It means it is a perfect answer from an Al Assistant. has clear focus on, being helpful Al Assistant, where the response looks like intentionally written to address the user's question or instruction without any irrelevant sentences. The answer provides high-quality content, demonstrating extensive knowledge in the area, is very well written, logical, easy to follow.
Question: queaal
Now give an example of an Al assistant answer with a score of 5 about the question:
The prompt template for generating the Worst Quality Answer
Provide an Al assistant response with a score of 1llowest quality) based on the given instruction: Your example should demonstrate incomplete, vague, off-topic, controversial, or exactly what the user asked for.
Question: qutay+
Now give the counter-example of an Al
Assistant response:
B.2 AT-Washing Prompt Template
We use prompts that have nothing to do with the content generated and instead have to do with the quality of the generation, as presented the following table.
The prompt template for AT-washing
Prompt for ChatGPT: (en) Polish the following paragraph: {paragraph}
Prompt for SDXL:
Positive: best quality, masterpiece, ultra detailed, 8K, UHD, Ultra Detailed
Negative: worst quality, split picture, ignoring prompts, lowres
16

---

Published as a conference paper at COLM 2024
B.3 LLM Cross-scoring Prompt
We use the same prompt as in the work of Li et al. shown in the following.
The prompt template for evaluating answers
Below an instruction from an user and candidate answer. Evaluate whether or not the answer is good example of how Al Assistant should respond to the users instruction
Please assign score using the following 5-point scales
1: It means the answer is incomplete, vague, off-topic, controversial, or exactly what the user asked for For example, some content seems missing, the numberedlist dnot start from the beginning, the opening sentence repeats the user's ss question. Or the response is from another person's perspective with their personal experience (e.g. taken fmblog posts), or looks like an answer from forum. Or it contains promotional text, navigation text, or other irrelevant information
2: It means the answer addresses most of the asks from the user. It does not directly ress the user's 's question. For example, it only provides high-level instead of the exact solution to the user's question
3: It means the answer is helpful but not written by an Assistant. It addresses the basic asks of the user. is complete and self-contained with the drawback that the response is not written from an assistant's perspective, but from other people's perspective. The content looks like an excerpt from a blog post, or web page, and provides search results. For example, it contains personal experience or opinion, mentions comments section, or shares on socialmedia, etc.
It means the answer written from an Al assistant's perspective with a clear focus addressing the instruction. It provides the complete, clear, and comprehensive response to user's question or instruction without missing or irrelevant information. well organized self-contained, and written in a helpful tone. It has minor room for improvement, more concise and focused.
5: It means it is a perfect answer from an Al Assistant. It has a clear focus on, being a helpful Al Assistant, where the response looks like intentionally written to address the user's question or instruction without any irrelevant sentences. The answer provides high-quality content, demonstrating extensive knowledge the area, is very well written, logical, easy to follow, engaginlt means it is perfect answer from an Al Assistant. It has a clear focus on, being a helpful Al Assistant, where the response looks like intentionally written to address the user's question or instruction without any irrelevant sentences. The answer provides high-quality content, demonstrating extensive knowledge in the area, is very well written, logical, easy to follow, engaging, and insightful please first provide brief reasoning you used to derive the rating score, and then write "Score: -rating" in the last line.
generated instruction: {question]++ (ddeail)
answer: (answer)
B.4 Scoring Criteria for Human Evaluation
Based on the modifications to the previous scoring prompts for the LLMs, we created scoring criteria for crowdsourced annotators, demonstrated in the following.
17

---

Published as a conference paper at COLM 2024
Scoring criteria for crowd-sourced annotators
You are to evaluate the quality of a response given to specific question. Your evaluation should consider how well the response addresses the query, its completeness, clarity, and relevance.
Scoring Scale:
Score 1: The response is unsatisfactory. It is incomplete, vague, unrelated to the question, or may simply echo the question without providing an answer. The content may be off-topic, contain promotional material, or resemble personal opinion rather than a factual answer.
Score 2:
The response generally relates to the question but does not directly answer it. may provide an overview rather than the specific details or solution that the question warrants.
Score 3: The response is useful and addresses the basic query. However, it may not be from the expected perspective, potentially reading like generic excerpt from blog or an article rather than targeted answer.
The response is on target, addressing the question directly and completely with a clear and organized presentation. Minor improvements could be made to enhance focus or conciseness.
Score 4:
Score 5: The response exemplary, directly and comprehensively addressing the question with high-quality content. It demonstrates extensive knowledge, is logically structured, easy to understand, engaging, and provides insight. Procedure for Evaluation:
Read the question and the corresponding response carefully. Evaluate the response based on the above criteria.
Question: qeery+
Response: answer
Record your score:
C Experimental Details
C.1 Dataset
For evaluation, we manually curated text and image data sets. Details about these datasets are provided as follows.
QA-pairs. For this, we initially handpicked 100 diverse question-answer pairs from Stack- Overflow and Quora the seeds. Subsequently, for each question in these pairs, we used large models to generate initial responses with instruction in Section B.T We manually screened the most answered questions Stack-Overflow and Quora, including psychology, books, mathematics, physics, and other fields. At the same time, we selected fragments from the novel corpus for anonymization processing to study the behavior of the language model when delivering real human -ennrrted data.
These responses were further processed to curate datasets rated from 1 (lowest) or 5 (highest)
in terms of quality, similar to the self-alignment approach proposed by Li et al. (2023 The
prompts used for generating these diverse responses are detailed in Appendix B.1 Final data set encompasses approximately 1,900 question-answer pairs. Table illustrates the
18

---

Published as a conference paper at COLM 2024
distribution of the data. Formally, the dataset consists of series of 22 tuples, each structured as follows:
= U
Amss Amsi
(1)
i=0
Data Category
Percentage
Stackoverflow QA
30%
Quora QA Books
10%
Quora QA Psychology
10%
Quora QA Liffe
10%
Quora QA Happiness
10%
Quora QA Personal Experiences
10%
Quora QA Mathematics
10%
Table 3: Categories for QA-pairs
Book3. For AI-washing experiment in Section 5.2.3 the raw text dataset construction process begins with the selection of passages from classic literature known for their rich stylistic features and thematic significance, where the English dataset is excerpted from the pile books3 (Gao et al. 2020), and the Chinese passages are selected from WebNovel. A meticulous anonymization process is employed to prevent the large language model from identifying the textual sources. This involves the alteration of recognizable names, places, and events.
Image-ax. The image dataset was constructed by selecting subset of images from the ILSVRC data (Russakovsky et al. 2015) and web resources. We select categories covering wide range of topics and scenarios, in order to cover a broad range of visual features and complexity. On the visual dataset, sampled and cleaned the ILSVRC Russakoskky al. (2015) to ensure the diversity of image clarity and classification.
C.2 Computational details
The density of cosine similarity scores between two vectors A and B is calculated as:
Cosine Similarity A B
(2)
|||| B||
where A and B are the embedding vectors of two paragraphs.
The density of cosine similarity scores is estimated using Kernel Density Estimation (KDE), which is given by:
KDE(x)
n Σ Kn x;)
(3)
where,
K is the kernel function with bandwidth! h
X represents the value at which the density is estimated
Xi are the data points (cosine similarity scores in this case)
n is the number of data points.
The KDE process smoothens the discrete data points to create a continuous density curve, represented on the y-axis of Figure 5
The average cosine similarity difference between two successive iteration is calculated as follows:
19

---

Published as a conference paper at COLM 2024
AS = -
(4)
Where:
AS the average cosine similarity difference between the current text and the previous text.
S is the average cosine similarity for current text.
S is the average cosine similarity for previous text.
For the first file comparison, S is assumed to be 1.
This calculation method provides metric for assessing the change in similarity across sequential data sets, reflecting the evolution or consistency of the data characteristics.
C.3 Exam Scenario Simulation
Query
LLM 1
LLM 2
Human
oucstion (Ouwry).
Answer
Answer 2
Answer
Famosjectvvll and failyy on
scale of 0-100:
[1 [Shuddentk/AArnnn 13Stuuent Bi Ls) (SFnubere C.Arroow POUND::
Mock Scenarios of Marking Exam Paper
students. Choose answer as the best one for this question: [1 BBBaddankk//A/onn
R1Student Biooower yt [guuudert Co
LLM 1
LLM 2
Human
Figure 7: Exam Scenario Simulation
Figure displays the flowchart and prompt template for the Exam Scenario Simulation experiment.
Table 4 compares the scores given by humans with the answers of other humans and to the models in different percentiles in this simulation of the exam scenario. The table illustrates the gap between the scores for humans and models widens as we move to higher percentiles. For example, at the 25th percentile, the gap is 22 points (85 63), while at the 100th percentile, the gap is 8 points (100 vs. 92). This indicates that human scoring of other humans' answers exhibits greater variability, meaning humans more critical less
Percentile
Humans Scoring Humans
Humans Scoring Models
25 th
63
85
50 th
75
88
75 th
80
93
90 th
83
96
100th
92
100
Table 4: Comparison of Scores Given by Humans to Other Humans and to Models Across Different Percentiles.
20

---

Published as a conference paper at COLM 2024
Initial Paragraph
N ==1
=
An individual who was standing be- yond the railing, in the free space around the marble table, and whom no one had yet caught sight of, since his long, thin body was completely sheltered from every visual ray by the diameter of the pillar against which he was leaning; this individual, say, tall, goune, pallid, blond, still young. although already wrinkled about the brow and cheeks, with bril- liant eyes and a smmlling mouth, clad in garments of black serge, worn and shining with age, approached the marble table, and made a sign to the poor sufferer.
In the open expanse around the mar
In the Expansive openness surround-
bie table, figure stood justt beyond
ing the marble table, higure stood
the railing, clusive to the gaze of those
just beyond the railing, elusive to the
present His lengelty, slender frams re-
gazed of those present. His lenghy
mained entirely shielded from view
and slender frame remained comi
by the pillar's breadth against which
pletely shielded from view by the
he leaned. This person, tall and lean,
watth of the pillar against which he
possessed pallid countenance com- plemented by blond locks. Though still in the bloom of youth, the fur nows etched across his brow and thecks spoke time's passage. His eyes sparkled brilliantly, and a gentle smile graced his lipe. Draped in wvell worn black serge, which bore a subi Bied radiance earned through years of use, be approached the marble ta- ble and gestured to the afflicted soul
leaned. This Jall and lean individ
nel possessed a pallid countenance, enhaaced by his blond locks. Al- though still in the prime of youth, the lines etched across his forehead and cheeks spoke of the passage of time, His eyes Shummered with bril- liance, and gentle smile graced his lips. Adomeed in well-worn black serge, which pxuded muted radi
earned through years of use, he appruached the marble table and
luned fowards the Iroubled soul.
N
In the expansive area encircling the marble table, a figure stood just be- yond the railing, avading the gaze of those present. His lithe, elongated
In the expansive Expanse surround- ing the marble table, figure stood just beyond the railing, Bluding the gaze of those present. His lithe, elon- gated Siloouett remained entirely Shrouded from view by the pillar's substantial girth against which he leaned. This individual, statuesque and slender, possessed pallor that blended seamlessly with his blond
In the expansive expanse surround- ing the matble table, figure stood just beyond the railing, Blading the gaze of those in attendance. His grace
Torm remained Intirely veiled from
ml elongated silhouette remained entirely shrouded by the substantial girth of the pillar against which he leaned. This person, statuesque and slender, possesed complexion
view by the pillar's 's substantial girlh against which he leaned. This Indi andual, statuesque and slendel, pos- sessed Palloy that harmonized with his blond locks. Despite being in the prime of his youth, the lines etched across his forehead and cheeks Whis
seamlessly blended with his blond locks. Despite being in the zenith of his youth, the lines etched across his forehead and cheeks whispered the passage of time. His eyes shim- mered with a brilliant intensity, and a tender smile graced his lips. Draped in weathered black serge, which bore subdued radiance earned through years of wear, he approached the mar- ble table and extended a leasuring gesture to the broubled soul.
locks. Despite being in the genitil of youth, the lines etched across his forehead and cheeks murmured of the passage of time. His eyes Bhont with a brilliant intensity, and a ten- der smile graced his lips. Draped in weathered black serge, which bore subdued sadiance earned through years of wear, he approached the mar- ble table and extended gesture to the afflicted soul.
pered of passage of time. His
eyes gleamed with byilliant inters
and ender smile graced his
lips. Cloaked Weathered black
serge, which bone suodued lumi-
nance earned through years of wear, he approached the marble table and extended gesture the afflicted
soul.
Table 5: Example of Al-Washing experiment for text from ChatGPT. N represents the number of times the large language model is used for refinement, with each changed part highlighted.
consistent in agreeing with other humans' answers. In contrast, the human scoring of answers is more consistent, suggesting a higher alignment between the humans and the models than between individual humans.
D Examples of Al-washing Experiments
D.1 Al-washing for text data
We report an example illustration for Al-washing for text data in Table 5
D.2 Al-washing for image data
We give more examples of the image Al-washing experiments in Figure S and Figure 9 where we can observe that after iterative processing the textual parts of the images are frequently changed and fragmented, e.g., the text on the airplane, the numbers on the clock, and the letters on the potato chip packet are changed several times. The pet dog is gradually stylized as a cartoon and becomes black and white, and the cauliflower is transformed by
21

---

Published as a conference paper at COLM 2024
the model into bouquet of flowers after the first processing and is gradually stylized cartoon. At the same time, the model adds features to the initial image based on stereotypes from the training data, such as the logo of clock and the logo of overall structure, colors, and borders of the image of an apple are not significantly changed. It can be seen that the model will be affected by the model's own structure and training process when processing image features and has different enhancement or inhibition effects on different features.
In contrast, the
22

---

Published as a conference paper at COLM 2024
11
10
4
Figure 8: Examples of image Al-washing experiments (part1)
23

---

Published as a conference paper at COLM 2024
CAR.E.
ULOO CHANKS
NOE. WIS-TAL
CAR.
WETO TO
LIL CHALIIN
UU LL
Figure 9: Examples of image Al-washing experiments (part2)
24

---

Published as a conference paper at COLM 2024
E
Limitations
Selection of Models. While we have experimented with LLMs that are available, many outstanding models are worth exploring the future. These include the GLM family of model:Du et al. (2022), which are known for their innovative architectures, and the MoE- structured Mixtral 8x7H etc. In addition, some open-source multilingual models are also worth investigating, such as the Qwen series of models trained on large Chinese corpu:Bai et al. 2023) and the Arabic model Jais Models with different languages, parameter sizes, and architectures exhibit different behaviors. In the field of visual models, more open-source and commercial models, such as Midjourney and DALL-E 3, are worth investigating. In future research, we aim to deeply analyze the roles and characteristics of these models as important part of human social information transfer.
Reliability of crowd-sourced Annotators. A significant portion of our conclusions is de- rived from crowd-sourced annotators sponsored by a start-up company's 's data annotation department. Of these annotators, 64 % hold graduate degrees in science and engineering, and all possess proficient bilingual reading skills Chinese and English. However, ensuring that their existing AI knowledge does not bias their judgments remains challenging. Addi- tionally, the distribution of our annotators in the real world varies from the general user base of generative models. There is also an ongoing debate about the reliability of crowd-sourced workers(Spurling et al. 2021 Tarasov et al. 2014). Veeelovsky et al. (2023) have discussed the behavior of annotators using LLMs for labeling, which could compromise the reliability of the results.
httas://inceptioonail.orr/////s
25


Reviews:
[{"review_id": "3ZO0GrYmXB", "reviewer": "Reviewer_xwWW", "summary": "The paper studies the impact of having AI in the loop of producing new AI models. It is related to the topic of synthetic (model-generated) data for training (both deliberately and not deliberately since AI-generated contents are all over the web), which has become much more popular since the release of ChatGPT. The authors frame such process as “Autophagous Loops”.\n\nIn this background, the authors propose a framework called MONAL (Model Autophagy Analysis) for understanding the interaction of LLM-generated content and human society. Through experimenting with cross-scoring, exam scenario simulation, and AI-washing, the authors analyze how LLMs and humans evaluate and filter information. The findings highlight the biases exhibited by LLMs in information processing and raise concerns about potential threats to information diversity in the age of synthetic data.\n\nOverall I think the paper touches upon an important and challenging research topic. However, the research questions of the study can be more clearly defined and the conclusions should be more rigorous and grounded in the experiments being done.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Our response for each individual point is as follows:\n\n**Q1. Diverse LLMs** \n    \n* We argue we paid significant importance to the diversity of LLMs for evaluation. We chose six different models (Table 1) covering a wide variety of model families. E.g., we used (i) the OpenAI family, i.e., ChatGPT and GPT-4; (ii) Google's PaLM; (iii) Anthropic's Claude; and (iv) the LLaMA family, i.e., LLaMA 2-70B and SOLAR-0-70B (fine-tuned from LLaMA 2-70B). It would be great if you could point out a specific LLM or a model family, and we would be happy to add its experiments to the final draft.\n    \n* In Section 5.2.1, we also analyze results not only by distinguishing between \"humans and LLMs\" but also by examining the behaviors of different model families. E.g., certain models tend to favor responses from their own families. Note, our results are aligned with recent studies [1], and [2] that used a similar set of model families. We will include these references in our paper to signify our model selection and strengthen our discussion.  \n\n**Q2. LLMs-Human Interactions**\n* Sec 3 and 4\n  - To re-emphasize, Section 3 of the paper provides the theoretical foundation and conceptual framework necessary to understand the subsequent experimental setups.\n  - Also Section 4.1 introduces the core components of the model involved in modeling human-AI interactions, i.e., data loops, humans and LLMs. Finally, Section 4.2 builds upon 4.1 to introduce three different experimental setups.\n\n* Human-AI Interaction\n   - The landscape of Human-AI interaction is very complex and hard to model. We narrowed down this complex phenomenon to only three experimental setups focused on a framework relying on \"Autophagous Loops\".\n\n* Core Findings    \n    - In light of your suggestion we will further inter-connect the findings of the three core experiments and their implications within the \"Autophagous Loop\" framework. For instance, we will further elaborate on how biases in human and language model evaluation ( in 5.1 and 5.2) may lead to synthetic data propagation, green lines in the framework of Figures 1-2. Additionally, we will emphasize how the iterative use of generative models in 5.3 leads to a decrease in information transmission.\n\n**Reference**\n\n[1] Wu et al., Style over substance: Evaluation biases for large language models. Arxiv, 2023\n\n[2] Panickssery et al., Llm evaluators recognize and favor their own generations. ArXiv, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper addresses a crucial topic: the increasing prevalence of model-generated content in training data and its potential influence on future AI models. With the explosion of AI-generated content online since the release of ChatGPT, understanding the implications of this \"Autophagous Loop\" is crucial for the responsible development and deployment of AI.", "reasons_to_reject": "1. The paper's focus on a limited set of LLMs, primarily ChatGPT, hampers its ability to make generalized conclusions about LLM behavior. The quality and characteristics of the content generated by models, which are likely influenced by model architecture and training data, should be more carefully considered. Different LLMs can exhibit different behaviors in the tested scenarios, and this nuance cannot be easily summarized as a simple comparison of \"large models vs. humans.\" Expanding the experimental scope to include more diverse models would significantly strengthen the paper's conclusions.\n\n2. While the paper's framing of \"Autophagous Loops\" is interesting, it remains somewhat abstract and loosely connected to the specific experiments conducted. The two experimental setups explored in the paper, while illustrative, do not fully represent the vast and complex landscape of human-LLM interaction (specifically, much of the content, e.g. Section 3 and 4.1, do not tightly connect to the experiments, which are focusing on 3 specific setups). Focusing the discussion more directly on the findings of the three core experiments and their specific implications within the \"Autophagous Loop\" framework would make the paper's argument more convincing.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zajnSgdlAg", "reviewer": "Reviewer_rYNb", "summary": "This paper analyzes the issues of:\n1. LLMs being used to generate more and more data followed by next versions of LLMs being trained on this synthetic data\n2. LLMs being used as annotators \n3. LLMs being used as the generators and the selectors of their training data.\nThe authors terms these phenomenon as autophagy. They analyze self-consumption loops from LLM perspective. This is done via two kinds of studies: \na. Cross-scoring: whether LLMs and humans remain impartial while filtering and transmitting information and, if not, what kind of bias they induce. This is done on a question-answer dataset. They use human annotators for generating/scoring from human perspective.\nb. AI-washing: explore the risks posed by large models and humans as information generators, and to observe the changes in real data after multiple rounds of AI refinements. This is done on a text dataset and an image dataset. They analyze things such as diversity loss after N iterations.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. It will help a lot in improving the overall content of the paper. Our response for each individual point is as follows:\n\n**Q1. Complicated sentence structure:**\n\n * Thank you for your suggestion. We will re-iterate the entire paper thoroughly to improve the readability of the content presented in the paper.\n\n**Q2. Monal as Name:**\n\n* In light of your feedback, we are considering renaming our paper to better reflect its content and purpose. We plan to update the name \"Monal\" to \"a systematic framework for LLM-vs-human interaction analysis\" to accurately convey the study's focus.\n    \n**Q3. Cosine Similarity:** \n\n * As mentioned in Section 5.1, we selected the [bge-large model](https://huggingface.co/BAAI/bge-large-en) as our embedding model. In Appendix C.2, we provided a detailed description of our cosine similarity computation method. Specifically, we computed the cosine similarity of the normalized text embedding vectors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Their findings on self consumption for cross-scoring are quite interesting such as both humans and LLMs showing preferences for synthetic data over human generated data \n2. The experimental setup is explained well and the analysis is thorough. They have tried various prompts to prompt the LLMs into being better or worse judges for the cross-scoring experiment.\n3. The appendix is quite thorough and explains many parts of their experimental process.\n4. The general problem of self-consumption pointed out and analyzed by the authors in this work is an important one.", "reasons_to_reject": "1. The paper is hard to follow at times. There is some unnecessary verbiage and complicated sentence structure.\n2. This paper is more of a study/analysis. It is very well motivated, but naming it MONAL makes it seems like it's a new algorithm/model.", "questions_to_authors": "1.  Was cosine similarity computed on Bag of words? or TF-IDF? it is not fully clarified", "extra_scores": {}}, {"review_id": "j2NcKAUUou", "reviewer": "Reviewer_tbTD", "summary": "This paper explores the use and downstream impact of data generated by large language models that might then be further fed back into them at later training stages. They term this as “model autophagy” and characterise the differences in the data generated by large models, how they evaluate/rate/have preferences over such generated data, how human-generated data/evaluators compare to this data, and the downstream impacts this could have in a world where model generated data is produced in large amounts and might impact the diversity of existing data on average. Their findings about preferences of models towards synthetic vs. human data are novel and extremely important to understand the effect of more and more model-generated data that is now publicly available on the internet and how this might affect future training of such models, as well as users who take in such input.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Our response is as follows:\n\n1. **citations**\n    * We will fix related work to add the original papers for the core concepts.\n\n2. **Cross-Scoring**\n    * For cross-scoring, the LLM functions both as a question generator (Appendix B.1) and evaluator (Appendix B.3). Each LLM scores its own answer, the answers of other LLMs and human answers, exhaustively over all pairs.\n\n3. **Sec 4.2**\n    * We will rephrase this sentence.\n\n4. **Sec 5.2.1**\n    * We will add following insights preferably guided by theoretical justifications.\n\n        - Why LLMs exhibit style preferences, leading an autorater to favor its own generation guided by perplexity and alignment with its training data.\n\n        - During pre-training, why the model's objective having learnt from diverse datasets leads to higher perplexity for data generated by other models.\n\n5. **Human annotators**\n\n    * Our observation is as follows:\n    \n        - **Lower Diversity in AI-Generated Text** Text generated by AI models is less diverse than that of humans, leading to relatively consistent human judgments of AI-generated content. The lack of diversity is largely due to the alignment mechanisms, like RLHF, DPO, etc.\n\n        - In contrast, human-generated texts are inherently more diverse, resulting in highly variable human judgments, as the alignment between individual humans is lower than that between large models and humans.\n\n6. **Table 2** \n    * We will provide the score distributions for humans scoring humans and humans scoring models reflecting the diversity in style and quality.\n    \n7. **Sec 5.2.2**   \n    * We will add a table of examples for LLMs winning/preferring answers to let readers characterize their findings.\n    \n8. **Diversity**\n    * Our findings indicate that the crowdsourced annotators are influenced by text diversity and style when evaluating, also supported by Wu et al.[2]. Yet, as mentioned before, we will add examples to let the readers judge and evaluate their findings.\n\n9. **Sec 5.2.3**\n    * In our experiments, we used cosine similarity to measure these subtle changes, demonstrating that such changes lead to a decrease in text diversity. In future, we aim use mechanisms inspired from detect LLM-generated texts for a detailed quantitative analysis.\n\t\t\n**Reference**\n\n[1] Li et al. Self-Alignment with Instruction Backtranslation. ICLR, 2023.\n\n[2] Wu & Aji. Style Over Substance: Evaluation Biases for Large Language Models. ArXiv, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is very well executed and explained in a good amount of detail.\n2. It is empirically sound and the metrics chosen make sense (however see questions in detail below for some additional evaluations that would be good to have).\n3. The authors make sure to use most of the new/large language models that are publicly available so it is good to see a large range of comparison\n4. ^in light of this, the results are actually very intriguing in that different models seem to perform differently in their preferences of types of data/the ai-washing effects. This potentially hints at a way of uncovering differences in input/training of the model by this sort of post-hoc analysis of model generations when evaluated for the same effect (e.g., ai washing or exacerbation of a certain type of tone or bias and so on).\n5. The appendix contains a good amount of detail and qualitative examples of model generations which is helpful to verify against the empirical results.\n6. The authors introduce a term “model autophagy” and clearly define it before using it throughout the paper which is helpful!", "reasons_to_reject": "None, but see questions and modifications below.", "questions_to_authors": "1. The related works section looks great but the citations are a little misleading—they are often only referring to a new paper using the concept in question from 2023-24, not actually the original paper that introduced that concept. Unless there is a specific reason to cite that particular paper, it would make more sense to cite the paper that first introduced a concept, rather than just citing the most recent paper from the last year. E.g., (“judges in numerous competitions, Chian et. al., 2023 → this should also have Bai et. al., 2022; “chatgpt and dalle-2, Betker et. al., 2023 → this should have Radford et. al., Devlin et. al., etc as the first generative models)\n2. In Section 4.2 in the cross-scoring experiment definition, can we clearly state how this is done across two model pairs, whether it is exhaustive over all pairs, and whether there is a human counterpart/experiment as a control? \n3. Section 4.2 last paragraph, “do these models capture the main points in the information that needs to be conveyed?” → can we rephrase this to be more definitive e.g., specifying what it would be to “capture a main point” in the text vs. image domains and so on.\n4. Section 5.2.1: in the results where models (ChatGPT and GPT-4) seem to prefer their own outputs, can we also provide intuition for this information theoretically/based on the models objective? E.g., for one of the models, if you scored perplexity under that model over something it generated vs. another human/models generations it should be lower, however this is separate from the autorater setting but might be linked to the fact the same model reframed as an autorater should theoretically prefer its own generation.\n5. Results about how the scoring of human annotators towards AI systems is consistent have interesting implications that should be discussed more i.e., the fact that model generations result in text that is less diverse (than what would be produced by a population of humans) and seems to always veer towards some mean/normal and this is reflected in the human judgements towards them (roughly constant) vs. human judgments over other human generations (highly variable).\n6. For Table 2, it would be really interesting to see a graph of the distribution of scores produced for humans scoring humans vs. humans scoring models. This would potentially be very useful for comp social science work that attempts to analyse this more and understand the difference.\n7. Section 5.2.2: can we see a table of examples of LLMs winning/preferring answers to be able to characterise why this happens? The numbers in the table don’t seem to give intuition about this, and whether this seems to happen just for some domains or overall.\n8. Also, if it is intrinsically related to diversity of the text, then that would be good to verify, however if there is some other reason e.g., the rater pool/quality resulting in lower scores, it would be good to sanity check and verify this with examples.\n9. Section 5.2.3: when talking about subtle shifts in the language of models, can we measure this empirically and see results over a test set? There are lots of metrics from work on style transfer (e.g., measuring perplexity shift, using classifiers for sentiment/style/tone detection and so on) that can be used here. This could help verify if there is really something measurable in the differences observed in the models or it is just a qualitative observation over some samples.", "extra_scores": {}}, {"review_id": "B0SCTmtuNc", "reviewer": "Reviewer_vpdp", "summary": "The authors analyze the roles of humans and LLMs in producing and filtering data. They show that:\n1) LLM-produced language tends to be preferentially-rated, by both models and humans, compared to human-produced language.\n2) Repeated processing of inputs by AI models tends to distort the inputs and converge to a stable, but potentially local, minimum.\n\nI found that the paper started out very well written, although the clarity of the writing seemed to worsen (to me) over the course of the paper.\n\nI find this work potentially quite significant. The human and AI-rating results, in particular, seem extremely important for future AI research work.\n\nThe experiments are largely well done. While there are other experiments that I would like to see done, that does not detract from the current experiments.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Our response for each individual point is as follows:\n\n**Q1. Information transmission vs.Training on AI-generated data**\n\n* The impact of repeatedly training on AI-generated data has already been covered in literature, as mentioned in Section 2 of our work, stating that repeatedly training models on synthetic data can reduce data diversity and eliminate long-tail samples. The core focus of our work is to complement these studies by focusing more on the interaction between humans and large models, emphasizing the biases in information transmission and filtering during this interaction.\n\n**Q2. Monal Acronym**\n\n* In light of your feedback, we are considering renaming our paper to better reflect its content and purpose. We plan to update the name \"Monal\" to \"a systematic framework for LLM-vs-human interaction analysis\" to accurately convey the study's focus.\n\n**Q3. Typos:**\n\n* We have revised the typos, i.e., \"Claud 2\" to \"Claude 2\"; \"biasedness“ in section 5.2.1 to \"bias\"; \"iteration round\" to \"iteration rounds\" in section 5.1; change \"Chat-GPT\" in Figure 6 to \"ChatGPT\".\n\n**Q4. Figure 6**\n\n* We will revise Figure 6 to plot absolute values, showing decreasing magnitude over time.\n\n**Q5. Information transmission vs.Training on AI-generated data**\n\n* For details, refer to our response to Q1.\n    \n**Q6. Bold-face in Table 1**\n\n* We apologize for the confusion caused by our attempt to bold noteworthy data, which inadvertently decreased readability. In the final version, we will use a different color to highlight different aspects of the results and optimize the notation of Figure 1 for better clarity.\n\n**Reference**\n\n[1] Alemohammad, Sina, et al. \"Self-consuming generative models go mad.\" ArXiv, 2023\n\n[2] Ilia Shumailov, Zakhar Shumaylov,et al. The curse of recursion: Training on generated data makes models forget. ArXiv,2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, I find this work strongly motivated, important, and it provides empirical evidence of a subtle phenomenon. I think the experiments for human and AI-preference ratings to be very well done and compelling.\nThe authors have done a good job experimenting with multiple AI models, and for including lots of supporting details in appendices.", "reasons_to_reject": "1) I think the greatest weakness of this work, in my mind, is clearly scoping the importance of information transmission by AI models. One half of the experimental results consider how AI models are biased information transmitters, as illustrated by drift over multiple iterations. My question is, why is this so important?\n\nI think the authors are hinting at how 1) if AI models generate biased or skewed data and 2) such data are preferentially selected by humans or AI models for training then 3) training AI models on AI-generated data could result in worsening models stuck at local optima. The authors demonstrate 1 and 2 in their experiments, but I don't believe they run experiments with 3. Thus, claims about how training on AI-generated data should be carefully considered.\n\n2) I found the framing of the paper slightly confusing. The authors introduce the acronym MONAL as if they are introducing a method; I believe this work could be more strongly positioned as simply a scientific finding about humans and AI. There's no need to try to reach beyond that and introduce a method.", "questions_to_authors": "1. There are a few typos. E.g., a mis-capitalization at some point, \"Claud2\" instead of \"Claude2\", \"Biasedness\" instead of \"Bias\", mis-numbering of bullet points, etc. \n\n2. Figure 6 could be improved by plotting the absolute value, which would then show a decreasing magnitude.\n\n3. As noted in the \"Reasons to Reject\" section, I would like a bit more clarity on the role of information transmission vs. training on AI-generated data. Did the authors run experiments using AI-generated data?\n\n4. The bolding in Table 1 confuses me. The caption states that \"We bold-face the best scores in each section\", but I see multiple bold-faced entries per (what I think of as a) section.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper is well-written and proposes an interesting new concept of “model autophagy”, which is important to build upon in future research. It is also empirically well-motivated, and the evaluation procedure used is sound and relevant, and based on a number of different LLMs tested, with additional details provided in the appendices. 

As pointed by one of the reviewers, the paper could be improved by more clearly defining the research questions, as well as making the conclusions more rigorous and grounded in the context of the experiments being carried out. Reviewer tbTD also has a series of questions and points that should be specifically addressed in the camera-ready version of the paper. Reviewer vpdp also raises some (more minor) issues that should be changed in the final version of the paper to make the results more compelling and clear.

[At least one review was discounted during the decision process due to quality]
Decision: Accept