Does Writing with Language Models Reduce Content Diversity?
Authors: Vishakh Padmakumar
He He
Venue: ICLR 2024 poster (2024)
Keywords: collaborative writing
text generation
language models
evaluation
human-AI collaboration
diversity
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We show via a controlled experiment that users collaborating with InstructGPT write with less content diversity than those collaborating with GPT3 and solo writers without model help.

Abstract: Large language models (LLMs) have led to a surge in collaborative writing with model assistance. As different users incorporate suggestions from the same model, there is a risk of decreased diversity in the produced content, potentially limiting diverse perspectives in public discourse. In this work, we measure the impact of co-writing on diversity via a controlled experiment, where users write argumentative essays in three setups---using a base LLM (GPT3), a feedback-tuned LLM (InstructGPT), and writing without model help. We develop a set of diversity metrics and find that writing with InstructGPT (but not the GPT3) results in a statistically significant reduction in diversity. Specifically, it increases the similarity between the writings of different authors and reduces the overall lexical and content diversity. We additionally find that this effect is mainly attributable to InstructGPT contributing less diverse text to co-written essays. In contrast, the user-contributed text remains unaffected by model collaboration. This suggests that the recent improvement in generation quality from adapting models to human feedback might come at the cost of more homogeneous and less diverse content.

Published conference paper at ICLR 2024
DOES WRITING WITH LANGUAGE MODELS REDUCE CONTENT DIVERSITY?
Vishakh Padmakumar New York University vi vishakh@nyu.ed edu
He He
New York University hehe@cs edu
ABSTRACT
Large language models (LLMs) have led to surge in collaborative writing with model assistance. As different users incorporate suggestions from the same model, there is a risk of decreased diversity in the produced content, potentially limiting diverse perspectives in public discourse. In this work, we measure the impact of co-writing on diversity via a controlled experiment, where users write argu- mentative essays in three setups- using a base LLM (GPT3), a feedback -etdbe LLM (InstructGPT), and writing without model help. We develop a set of diversity metrics and find that writing with InstructGPT (but not the GPT3) results in a statistically significant reduction in diversity. Specifically, increases the similarity between the writings of different authors and reduces the overall lexical and content diversity. We additionally find that this effect is mainly attributable to InstructGPT contributing less diverse text co-written essays. In contrast, the user-contributed text remains unaffected by model collaboration. This suggests that the recent improvement in generation quality from adapting models to human feedback might come at the cost of more homogeneous and less diverse content.
INTRODUCTION
Large language models (LLMs) rapidly changing how people create content (Lee et al.) 2022a Mirowski et al. 2023). While LLM-based writing assistants have the potential to improve writing quality and increase author productivity, they also introduce algorithmic monoculture (Kleinberg and Raghavan 2021). As millions of users rely on the same underlying model to produce text, there a potential risk of shifting the content towards the modc-good but not colorful writing. In this work, we aim to assess whether writing with LLMs unintentionally reduces content diversity.
Existing evidence has already hinted that LLMs may influence users' opinions. Jakesch al. 2025 and Bhat et al. (2023) find that users' opinions expressed in their writing can be biased by controlling ling LLMs to promote specific argument (e.g., social media is good/bad for society). Additionally, has been shown that current LLMS do not equally represent views from various demographic groups (Santurkar et al. 2023] Durmus et al., 2023). These results suggest that writing with L.LMs may limit the perspectives expressed in the writing. We hypothesize that incorporating model suggestions dilutes the writer's unique voice, leading to different writers producing similar content, and this homogenization in turn reduces the overall diversity of content produced by many writers.
To test our hypotheses, we design a controlled experiment (Figure M where users are asked to write an argumentative essay given topic from the New York Times student opinion series following Lee et al. (2022a), e.g.. "What are the most important things students learn at school?" We assign participants three groups: control group where participants write essays without model help, an I.IM treatment group where participants write essays with a base language model (GPT3), and a feedback-tunee LLM treatment group where participants write essays with language model finetuned with human feedback (Ouyang et al. (002) (InstructGPTT. We hired 38 writers from Upwork. For each group, we collected 100 essays on 10 topics. We then develop a set of metrics and measure the effect of LLMs on content diversity at both the individual level and the collective level:
Homogenization: Do users write more similarly to each other when writing with L.Ms?

---

Published conference paper at ICLR 2024
Solo
Homogenization
Users with InstructGPT write more simiar essays than other groups
Topic most important things studintss leamn at school?
GPT3
Diversity
Essays co tten with InstrucrGPT has the least content diversity
InstructGPT
Figure 1: We measure the content diversity of essays written by three groups of users: a control group writing without model help (Solo), treatment group writing with a base language model (GPT3), and second treatment group writing with feedback-tuned language model (InstructGPT). Essays co-written with InstructGPT are more similar to each other and exhibit the least lexical and content diversity, whereas those co-written with GPT3 are not significantly different the Solo essays.
We find that essays written by the group using InstructGPT exhibit higher degree of homogenization compared to the control group and the GPT3 group (Section In particular, by matching montr-con text to the summarized key points of each essay, we find that the key points contributed by InstructGPT result in increased homogenization (Section O
Diversity: Does writing with LLMs reduce the diversity of content produced by a group of users?
We find that the set of essays written with InstructGPT show both lower lexical diversity and reduced diversity in the key points expressed as compared the other two groups (Section 55). This is seen in the increased repetition of phrases (higher-order n-grams) introduced by the model (Section 5.2 and reduction in the number of unique key points in the essays.
Interestingly, while writing with the human feedback-tung L.LM (InstructGPT) incurs increased homogenization and reduced diversity, did not observe statistically significant effect from writing with the base LLM (GPT3) despite a similar model contribution (Section 3. Further analysis shows that InstructGPT tends to provide less diverse suggestions and introduce less diverse text compared to GPT3. This finding echoes prior results showing reduced diversity after reinforcement learning from human feedback (Bai et al. 2022: Song al., 2023).
As LLMs become more tightly integrated into text editing applications, our findings highlight a new axis of evaluation in interactive settings. Reduced content diversity is not only detrimental to personal expression and creativity, but also risks creating a feedback loop as future models trained on the homogenized content further propagate this pattern (Taori and Hashimoto (023). To facilitate future research on understanding the impact of LLMs in co-writing, we release essays from all three groups along with all model suggestions recorded by the CoAuthor platform (Lee et al. 2022a
DATA COLLECTION
Our approach to studying the impact of LLMs on content diversity is to conduct controlled exper- iments. Specifically, we control the set writers and the topics of the essays. We then compare essays written with and without model help.
2.1
TASK SETUP
We consider the task of argumentative essay writing using topies from the New York Times Student Opinion series following Lee et al. (2022a). Specifically, users are given a topic such as "What are the most important things students learn school?" and are asked to write an essay expressing their opinions in around 300 words. We choose this task because the topics are sufficiently open-ended to allow for diverse responses while remaining accessible to users with all backgrounds. The instructions and the complete list of 10 prompts used in the experiment are provided in Appendix A.3
Code and data are available at http:://gggthooooo/////aaa////ar
2

---

Published conference paper at ICLR 2024
We adopt the CoAuthor platform developed by Lee et al. (022a). When the user hits the TAB key, we obtain 5 text continuations from backend model and present them in dropdown menu. Figure Appendix shows a screenshot of the interface the user requests model suggestions. The user then has the option to either accept and edit one of the suggestions or reject all suggestions and continue with their writing process. We ask the users to request suggestions minimum of 5 times per essay but do not require them to accept any of these. The entire writing process, including all keystrokes, suggestions, and user actions, is recorded by the platform. This allows to differentiate the parts of the essay introduced by the users and models at the character level for subsequent analysis.
2.2 EXPERIMENT SETUP
We recruit participants from Upwork who are native English speakers and have prior experience in writing or copyediting on the platform. We provide further details on participant recruitment and remuneration in Appendix [ We then collect essays from three settings: control group, where participants write essays without any model help; an L.LM treatment group, where participants write with the help of base language model; and feedback-tuned LLM treatment group, where participants write with a language model finetuned on human feedback. We distinguish the two types of language models because prior work has shown that finetuning language models on human feedback tends to decrease the entropy of the output distribution [ai et al. (2022) and thus may decrease text diversity in general. We henceforth refer to the three settings as Solo, GPT3, and InstructGPT respectively.
We obtain model continuations from the OpenAl API using davinci the base language model and text-davinci-00 as the feedback-tuned language model. To avoid reduced diversity due decoding, we use a decoding strategy biased towards higher entropy. Specifically, we sample continuations from both models with a temperature of 0.9 and frequency penalty of 0.5 (detailed parameters listed in Appendix (A.)), following the "high randomness" setting from Lee al. (2022a).
Each participant is asked to complete a session consisting of three essay writing tasks-one one in each of the three settings and each on a different topic. In each session, the order of the three settings and the assignment of topics are randomized. Each essay takes 15 minutes average. In total, we obtained 10 essays on each of the 10 topics for each setting, resulting 300 essays total.
3 How MUCH DO USERS ENGAGE WITH THE MODEL.?
Before diving into the analysis of content diversity, we first examine whether the user and the language models form an effective collaboration by examining the usage statistics and modellcontributed text.
Model
# Queries
Acceptance Rate (%)
Written Percentage
Word Count
InstructGPT 9.15
70.49
32.45
368.39
GPT3
9.62
71.32
35.57
380.87
Table 1:
Averaged per-essay usage statisties of GPT3 and InstructGPT. Both models are helpful to users, evidenced by the high acceptance rates of suggestions and the sizeable model written fraction of essays. Furthermore, we find no statistically significant difference in usage of the two models.
Usage statistics. We find that users actively query the model and use model suggestions during writing. As shown in Table users query the accept about 70% of them (note that they are asked to query at least times). Since users may further edit the suggestions after accepting them, we want to know whether the accepted suggestions are retained in the final essay The interface used for data collection records keystroke-level information about whether each character was introduced by the model or the user. Thus, we report the average percentage of characters introduced by the model in each essay. We observe that the model contributes roughly 35% of the characters in each essay, suggesting that they are reasonably helpful to the writing task. Finally, we are curious whether the users find InstructGPT to be more helpful than GPT3 given its additional feedback tuning. However, the two language models appear to be equally helpful to
around 9 times on average for each essay and
2 Participants are not obligated to accept any suggestions from the model. We ask that they query the model for suggestions at least times when writing an essay, accepting these only when appropriate (Appendix A.3
3

---

Published conference paper at ICLR 2024
While believe the concerns regarding children's screen time are valid, I believe it is somewhat biased to not
take this problem, which is genuine issue right now, an everyone problem. [skipped] know that I, along with many other teenagers, would like nothing more than to go back to school, play sports outside, meet new people, and such. [skipped] They are also places for new idcas, watching college lectures, and political discourse [skipped]
The problem of screen time should be considered an everyone problem, not just a student one
Social media can he used for educational purposes
Limiting screen time may not be effective in the long run
Parents should trust their teenager more and not worry too much about their screen time
Table 2:
Excerpt from an essay written by user with InstructGPT and the generated key points. Bold text was introduced by the model. We align each key point to a sentence in the essay with which it has the highest Rouge-L overlap, as illustrated by the highlight. Since the majority of the text in the aligned sentence is written by the model, we attribute the first key point to the model.
users. We do not observe significant difference between their usage statistics: an independent samples t-test the number of queries and acceptance rates shows significance the 5% level.
Model contribution to key ideas. The usage statisties tell us that LLMs contribute to fair amount of the text written. However, are they contributing key arguments or merely supporting the elaboration of point? To quantify model contribution the main ideas, we need way to represent the important content in an essay. Motivated by recent encouraging results of using LLMS for zero-shot summarization (Goyal et al. 2022). we summarize each essay into list of key points by prompting 3et-3.5-turbo An example is shown in Table
Given a list of key points for each essay, we then estimate the fraction of key points written by the model and the user. Specifically, we align each key point to a sentence in the essay with which it has the highest overlap, as measured by Rouge-L (Lin) 2004). If more than half of the sentence's characters are
based on the recorded keystroke data, the key point is attributed to attributed to the user. We show the fraction of key points written by the model for the two groups of co-written essays in Figure in Appendix[[] We observe that the median fraction for both GPT3 and InstructGPT is around 0.4, indicating that models make a significant contribution to the key content of the essay. However, there is also a large variation among users, with some essays having none or all of the key points attributed to the model, suggesting diverse levels of reliance the model.
the model; otherwise,
Given the significant model contribution in co co-written essays, we hypothesize that this intervention would result in increased content similarity among users. We test this hypothesis in Section
DOES WRITING WITH LLMS RESULT IN MORE SIMILAR ESSAYS?
In the previous section, we have seen that models contribute a substantial portion of the essays' content. Does this lead to more similar essays, or do the essays retain the individual styles of the users? In this section, we measure the homogenization effect of co-writing with LLMs.
4.1 METRIC
We first define the homogenization of single essay its average pairwise similarity all other essays written on the same topic. Let D, denote the set of essays on topic t. The homogenization score of essay d (written in response to topic f) is hom(d t) |.I-I Ex'e,, En sim((d, d') where denotes the size of the set. Correspondingly, we define corpus homogenization as the average homogenization score of all essays. We two metrics compute the similarity between two documents: Rouge-L, an overlap-based metric; and BertScore (Zhang et al. 2000),, an embedding- based metric Both homogenization metrics range from 0 to 1 with higher score indicating more
'We use gpt 3. 5-5-5m0 5-turbo due to its high performance on summarization on the HELM benchmark (Liang al. 2022).
SSe Table 6 in Appendix A.4 for the full example. On average, each essay is summarized into 6.86 key points with a standard deviation of 1.53. There is no significant difference between the average number of key points generated for essays from any pair of groups based independent samples t-tests the 5% level. model, the highest ranked model in the BertScore repository
We use the mi microooft/deeer ta ta-xxaagee-maam according to Pearson correlation with human ratings of similarity.

---

Published conference paper at ICLR 2024
0.220
Solo
GPTS
InstrunGGPT
0.200
0.180
Hanggenizllaa
0.180
0.140
0.120
0.080
(*)
Ciiizens
Audicbokks
Stersolypes
Welfore
Athletes
Figure 2: Boxplots of homogenization scores for all three groups (Solo, InstructGPT, GPT3) when comparing essays at the key point level using Rouge-L as the similarity metric. The left-most column (Total) shows essay homogenization scores for all topics and the other columns show the scores for essays on each topic. Essays written with InstructGPT exhibit higher corpus homogenization by statistically significant margin (Section 4.2) and the highest median topic homogenization on out of 10 topics. We observe the same trend when using BertScore in Figure 9[b) in Appendix [
similar content. We measure homogenization both the essay level and the key point level. For key point homogenization, we represent each essay as a concatenation of its key points (Section before calculating its homogenization score.
4.2 RESULTS
Writing with InstructGPT produces more similar content. The corpus homogenization scores when comparing essays from the three groups at the key point level using Rouge-L. are 0. 0.1536 for Solo, 0.1578 for GPT3 and 0.1660 for InstructGPT. InstructGPT exhibit higher corpus homogenization than both other groups to statistically significant degree as verified with p < 0.05 on an independent samples (-test. This result holds across both similarity metrics as well as when the essays are compared at the essay level (Table TO in Appendix C). FigureZshows the homogenization scores across topics at the key point level using Rouge-L We observe that the InstructGPT group has a higher median homogenization than other groups 7 out of 10 topics. While the effect of model intervention varies across topics, the trend of increased homogenization persists (Appendix B.3)
Writing with GPT3 does not increase homogenization. Although GPT3 and InstructGPT con- tribute a similar amount of text and key points to the co-written essays (Section 3). unlike InstructGPT, GPT3 does not appear to increase the similarity between essays written by different users. In particu- lar, there is no statistically significant difference between the corpus homogenization score of GPT3 and that of Solo at the 5% significance level using an independent samples t-test. This discrepancy between InstructGPT and GPT3 suggests that the impact of LLMs content homogenization is not uniform. While InstructGPT often outperforms GPT3 on benchmarks across a wide range of metrics (Liang et al. 2022), our results highlight that this performance boost may come the cost of creating more homogeneous content in interactive settings.
5 DOES WRITING WITH LLMS REDUCE THE OVERALL DIVERSITY?
In the previous section, we have seen that writing with the feedback sendback-tuned model leads different users writing similar essays on the same topic. A direct consequence of this homogenization effect that may reduce the overall diversity of the collection of writings from many authors. this section, we test whether writing with LLMs reduce the diversity of the produced content.
5.1 METRIC
To measure the diversity of collection of essays, represent the corpus bag of information units (e.g., n-grams), and compute the number of unique information units divided by the total number of information units, such the type-token ratio. Intuitively, the essays all repeat each other (least diverse), the entire collection can be represented as a single essay (small ratio).
Fiiuure and Figure in Appendix contain similar plots of the homogenization scores at the essay level and key point level using both Rouge-L. and BertScore.
5

---

Published conference paper at ICLR 2024
We consider two types of information units: n-grams and key points. The fraction of unique n-grams straightforward measure of lexical diversity (Li et al.) 2023) Meister al. 2023). For key point diversity, we first represent each essay set of key points (as described in Section and aggregate the key points from all essays. To compute the number of unique key points, we perform agglomerative clustering and consider all key points in cluster to be equivalent. We use the implementation agglomerative clustering from Scikit-learn (Pedregosa et al.) (2011) using the complete linkage criterion | We convert Rouge-L to a distance metric for the clustering algorithm by subtracting the similarity score from (which ranges from 0 to 1).
5.2
RESULTS
n-gram size
Solo
GPT3
InstructGPT
Thresholds
Solo
GPT3
InstructGPT
0.119
0.116
0.115
0.5
0.982
0.971
0.950
2
0.602
0.585
0,579
0,941
0.927
0.877
0.898
0.886
0.869
0.973
0.967
0.953
0.7
0.792
0.779
0.738
0.8
0.543
0.514
0.494
0.991
0.988
0.977
(h) Fraction of unique key points.
(a) Fraction of unique r-grams
Table 3: Content diversity calculated in two ways: (a) N-gram diversity in each setting. Bold values indicate the lowest diversity score as measured by the fraction of unique n-grams. Essays written with InstructGPT are the least diverse across n-gram sizes. (b) Diversity measured by agglomerative clustering of the key points of essays with RougeL using various distance thresholds for clustering. Bold values are different from other columns by statistically significant margin (p < 0.05). InstructGPT consistently exhibits lower content diversity across all selected thresholds.
0.44
Solo
Sols
Prabgaalyy
GPT3
InstructPPT
Probeehiit
0.0003
GPT3
InstuctPP
0.0002
1-grams decreesing order of frqquancy
5-grams in decreasing oreer of frqquinyy
(a) Unigram Distribution
(b) 5-gram Distribution
Figure 3: Distributions of the top-50 (a) unigrams and (b) 5-grams from each group. Despite similar unigram distributions, essays written with InstructGPT exhibit higher repetition of common 5-ngrams.
Writing with InstructGPT reduces lexical diversity. In Table Bal we show the diversity measured by n-grams (where n = 1, 2, 3, 4, 5) for essays written in all three groups. We see that the InstructGPT group consistently exhibits lower diversity across all values of n, whereas GPT3 does not significantly decrease the diversity score. The reduced lexical diversity showcases a disadvantage of co- writing with L.Mss, particularly creative writing genres like stories and poems, may wash away unique expressions the long tail of human writings.
Writing with InstructGPT reduces key point diversity. In Table 3b report the diversity scores measured by the fraction of unique key points. The clustering process involves a threshold that determines the maximum distance above which two clusters will not be merged We vary this threshold and find that the InstructGPT group consistently exhibits lower diversity across all selected values 2 In addition, the diversity of the InstructGPT group is lower than both the Solo and GPT3 groups by statistically significant margin (p < 0.05) by a pairwise permutation test detailed in
Complete linkage measures the distance between two clusters by maximum distance between any pair of items from them.
'A higher distance threshold indicates that key points having lower similarity can be clustered together. At higher thresholds, the diversity hecomes lower for all groups because more key points considered equivalent.
9hhe same trend holds when using BertScore as a distance measure (Table 12b in Appendix C

---

Published conference paper at ICLR 2024
Appendix 1.1
More concerning than reduced lexical diversity, this decline in key point diversity raises concerns with long-term social impact, emphasizing the need to safeguard perspectives from underrepresented groups collaborative writing gains prominence.
Essays written with InstructGPT repeat higher-order n-grams more frequently. To further analyze how diversity is changed, we plot the distribution of unigrams and 5-grams in each group of essays in Figure 3 (truncated to the most frequent 50 n-grams from each group) [ In the InstructGPT group, a notable concentration of probability mass in the head of the distribution for 5-grams suggests higher degree of repetition and lower text diversity in the co-writing process. We verify the significance of the difference in 3-gram frequencies between InstructGPT and Solo via chi-square test (p 0.05) To see what kind of phrases the model repeats, we list the ten most frequent 5-grams in Solo and InstructGPT in Table We find that common 5-grams from InstructGPT often contain topical words overlapping with the essay prompt, whereas those from Solo more often generic phrases. As users may seek suggestions for key ideas in the essays (Section B the similarity of the suggested content manifests in these common topical 5-grams which occur repetitively.
Solo
InstructGPT
5-Gram
Count
5-Gram
Count
keeping up with the news in my opinion the most
7
keep up with the news on animal welfare when humans
14
7
12
keep up with the news opinion the most important things on the other hand the most important thing that wrong to facus on animal focus on animal welfare when unfair when it considered in my opinion listening
6
to focus on animal welfare selfish pursue risky sports students should learn school
11
6
11
5
5
wrong to focus on animal sports like extreme mountain climbing
10
5
10
5 keeping up with the news
5 the end of the day
4 things students should Iearn in
Table 4: Most frequent 5-grams in Solo and InstructGPT corpora along with the corresponding repeti- tion counts. Highlighted 5-grams are those that contain topic-specific information from the prompts. We observe higher repetition overall and more occurrence of topical 5-grams with InstructGPT.
We additionally show in Appendix C that the reduction of diversity measured by linguistic units such as n-grams and key points also correlates with less diversity in an - sense.
WHY DOES WRITING WITH INSTRUCTGPT REDUCE DIVERSITY?
We consistently observe that writing with InstructGPT results in a reduction of content diversity, whereas this is not the case with GPT3 despite users seemingly engaging equally with both models (Section 3 This raises the question of how writing with InstructGPT makes a difference. There are two potential contributors. First, InstructGPT may produce less diverse text, influencing the essay by directly contributing text. Second, users may write differently in the presence of assistance, e.g., adopting the model's 's opinion (Jakesch al. 2023). In this section, we aim to answer this question by disentangling the effect from the model and the user in collaborative settings.
InstructGPT generates less diverse text than GPT3. We first confirm that generations from InstructGPT are less diverse than GPT3, which has also been observed prior work. The technical report for GPT4 finds that the feedback-tuned model is less calibrated (OpenAl, 2023) and Bai et al. (2022) find that finetuning leads decreased entropy of the output distribution. In our setting, we measure the diversity of model generation by the pairwise similarity of the five generated continuations upon each user query, similar to self-BLEU (Zhu et al., 2018). Thus, higher similarity scores indicate lower diversity. We observe that InstructGPT indeed generates less diverse text than GPT3 average (0.20 vs 0.11 on Rouge-L), with significance at the 5% level using an independent samples t-test Since users incorporate text from both models similar rates (Table I). it is plausible that the less
Fach line Figure corresponds to the 50 most frequently repeated n-grams in a specific group, which vary across groups. Figure 10 in Appendix C shows similar plots for 1 to 3-grams.
We observe significant difference in r--rrm usage for 4- and 5-grams detailed in Appendix B.2 We also obtain the same conclusion using BertScore as the similarity metric (see Figure 12 in Appendix
7

---

Published conference paper at ICLR 2024
0.200
0.285
GP13 IntbekiPP
0.0005
GP13 Conotaution Humans
Inmestt Contitttionn
GPTS
0.175
Prokapoits
0.0005
Human Cont bution
0.150
0.0004
3,185
0,100
0.0002
0,076
0,000
Humen Conniicttin
Mode Contritziion
S-grams decreasing onder of frequency
10
20
30
40
60
(a)
(b)
Figure 4: (a) Homogenization scores calculated separately on key points attributed users and the model using Rouge-L. InstructGPT contributed key points show higher degree of homogeniza- tion whereas user-contributed key points have similar homogenization scores when writing with InstrucIGPT and GPT3. (b) Distributions of 5-grams introduced by the user (dashed lines) and the model (solid lines). The distribution of 5-grams from Solo essays is also provided as a reference. Distributions of user-written text in all settings are similar regardless of model assistance, whereas InstructGPT contributed text has notably larger probability mass on common 5-grams. This indicates that the reduced lexical diversity in the InstructGPT group is mainly due model introduced text.
diverse suggestions from InstructGPT lead to a decrease in the diversity of the final co-written essay. Next, we examine the diversity of model-writte and user- ber-wwrtten text in the essays directly.
InstructGPT increases repetition of higher-order n-grams while user-written text is unaffected. The essays consist of both modet-written and usir-written text. Which contributes more to the decrease in diversity? To study this, we attribute each token in the essay to either the user or the model
We then examine the distribution of 5-grams from user-written and model-written text Figure 4(b) 14 We see that the 5-gram distributions of user- written text remain the same regardless of whether the user writes with a model (Solo vs. GPT33InstructGPPT) and which model they write with (GPT3 vs. InstructGPT). Hence the phrase-usage pattern of users is not affected by the presence of model assistance. Instead, the increased repetition of 5-grams observed in Section 5.2 is more model- related as evidenced by the high probability mass on common 5-grams introduced by InstructGPT 15
InstructGPT increases similarity between key points while user-written text is unaffected. Finally, we disentangle the effect from the model and the user for the increase in homogenization observed in Figure We attribute each key point to either the user or the model (same in Section 3) and calculate the essay homogenization scores on user-contributed and model-contributed key points respectively. In Figure (aa we compare the user-contri and model-contributee homogenization across the InstructGPT and GPT3 groups. We observe that, while the average homogenization score of user-contributed key points does not change between the two groups, the InstructGPT contributed key points have higher average homogenization than GPT3 (p < 0.05). This suggests that user behavior with respect to content creation is largely unchanged when writing with different models! 16 and the increased homogenization is mainly due to InstructGPT contributions.
Takeaway. Our results suggest that the users' lexical choice and argument generation is not affected when writing with model assistance. The reduction in diversity collaborative writing is attributed more to text contributed directly by the InstructGPT model. This paints a relatively optimistic picture as the risk of forming a feedback loop through the mutual influence between the user and the model is potentially limited. However, it is important to note that our study focuses on single interactions between users and models, and the dynamics might change through repeated interactions over time.
We use the recorded character-level keystroke information (Section 2.1 determine the authorship of each token. Most tokens attributed entirely to either the user the model. In cases of mixed contribution, authorship assigned based on majority of characters.
"We ensure that each entire gram here was written by the user or model in a single continuous span of text ensure do not count incoherent n-- re-grams.
15 Figuree [1 in Appendix shows similar distributions varying n from 1 to 5, We see the increased repetition of n-grams introduced by InstructGPT in 3 and 4- 4-grams well.
We can also compare the homogenization of user- contriibtd key points with and without model assistance. However, we note that cont-cbntd key points in co- co-written essays are a subset of the original key points and hence exhibit more variation across essays. Therefore, they are not directly comparable to those in Solo.

---

Published a conference paper at ICLR 2024
7 RELATED WORK
Human-Al AI collaborative writing. Collaborative writing predates LLMs where users assisted
Chen et al. 2014, Roemmele and Gordon 2013) or generated by task-specific RNNs Ghazvinnneejad et al. |017
by suggestions retrieved from a knowledge base (Swanson and Gordon 2008 2012
Roemmele and Gordon 2018 Clark et al. 2018). The early systems can produce relevant suggestions, but they often need to be revised before being incorporated into the draft. In contrast, LLMs offer suggestions that are often directly incorporated into user writing, either through continuations or in response to instructions, resulting in a surge of studies demonstrating their effectiveness (Gero and Chilton 2019: Akoury et al., 2020; Yuan et al. 2022 Swanson et al.) 2021 Chakrabarty et 2022 !ppolito et al., 2022 Mirowski 2023) However, despite the improved productivity, we show that I.LM-contrbbuted text can also affect the writing in subtle and unintended ways.
Effect of human-Al co-writing. Recent research focuses on analyzing user-model interactions and their influence on users and the writing process. [Buschck et al. (2021) investigate how the number of suggestions affects users in email writing. highlighting the tradeoff between suggestion utility and writing efficiency. [ee et al. (2022a) collect data on user interactions with GPT3 as collaborator, noting increased vocabulary diversity and productivity. Our work extends this research by formalizing metrics to quantify the homogenization effects resulting from user interactions with LLMs by comparing collaborative essays with essays written without model help.
Evaluation of interactive text generation. Traditional evaluation metrics primarily assess text similarity to reference text (Papineni al. 2002: Lin 2004, [ellla al. 2020p. In interactive contexts where references are absent, alternative metrics gauge model assistance effectiveness indirectly. These include user-rated model helpfulness (Roemmele and Gordon 2018 Clark et al. 2018, Padmakumar and He 2022), retention of model suggestions in the final text (Akoury et al. 2020), and completion time [Buschek et al. 202T). Lee et al. (2022b) additionally considers user ownership and enjoyment which emphasizes maintaining user writing style. Our work extends existing metrics that evaluate lexical diversity and text style based on token and POS n-grams statistics (Roemmele et al.] 2017]] (See al.] [219:) (Teeet and Berant 2021 Meister al.) 2023). We adapt these metrics for co-writing scenarios and introduce content-specife diversity measures.
Social impacts of LLMs. Bommasani et al. [2021); Bender et al. (2021); Solaiman et al. (2023) extensively discuss the societal risks of the adoption of LLMS in user-facing applications. Most relevant to work is the concern over the potential emergence of an algorithmic monoculture (Kleinberg and Raghavan (202T), which assesses outcome homogenization in decision-making Creel
and Hellman (2022); [Bommasani al. (2022). Additionally, [Hancock al. (2020) discuss the potential language homogenization during Al-mediated communication, while Arnold et al. (2020) demonstrate that predictive keyboards lead to shorter image captions with fewer rare words. Pakesch iet al. )2023): Bhat et al. (2023) show that writing with biased language model can change user opinions. Liao and Xiao (003)) advocate for novel evaluation methods to uncover socio-technical issues arising from increased general-purpon LLM use. Our work investigates one axis of such evaluation, namely content diversity, in co-writing settings.
CONCLUSION
In this work, we study how writing with LLMs impacts the diversity of content produced. Through a controlled study, we find that users writing with InstructGPT produce more similar content than those writing with GPT3 or without model help. This homogenization also results in a reduction of the overall diversity of content produced by many users. Furthermore. our analysis indicates that this effect is attributable to the less diverse text contributed by InstructGPT, while the user-contributed text remains largely consistent with and without model help. These findings highlight a new axis on which to evaluate the impact of LLMs prior to their deployment. While adapting a model with human feedback leads to an improvement in instruction following, this may be accompanied by a reduction in content diversity. Combined with other recent findings that LLMs influence user opinions during the writing process (Jakesch et al., 2023), this calls for a careful user-centered evaluation to ensure that these models do not suppress the voice of users in scenarios where personal expression is desired. Finally, our work also adds to the growing body of open problems on reinforcement learning with human feedback (Casper al. 2023). Learning from the diverse feedback of many users and personalizing model generations to individuals are challenging future directions.

---

Published conference paper at ICLR 2024
ACKNOWLEDGEMENT
We would like to thank Mina Lee. Nick Lourie, Naomi Saphra, Richard Pang, and Nitish Joshi for their imput at various stages of the project. We would also like to thank Abby Rabinowitz and Alexander Landfair from the NYU Expository Writing Program for valuable discussions during the planning stages. We would finally like acknowledge all the writers recruited for this project without whom this work would not have been possible. This work is supported by Open Philanthropy, AWS AI, and the National Science Foundation under Grant No. 1922658.
REPRODUCIBILITY STATEMENT
All data collection experiments were done between March and July 2023, For the sake of repro- ducibility, the essays with character-level logs as recorded by the interface along with all model suggestions presented users will be released after the review period. This allows readers replay entire writing sessions using the code released by [ee et al. (2022a). We also make public all the scripts to process the raw logs to perform the analyses run this paper.
ETHICS STATEMENT
We detail all the procedures for user recruitment and remuncration in Appendix A..1| The plan for our user study was approved by the Institutional Review Board of our university. We obtained consent from all participants to share the essays publicly post the completion of the study. We detail further limitations of our study in Appendix D
REFERENCES
Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Tyyer. 2020. Storium: A dataset and evaluation platform for machine-in-the-loee story generation. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages (470-6484.
Kenneth C Arnold, Krysta Chauncey, and Krzysztof Gajos. 2020. Predictive text encourages predictable writing. In Proceedings of the 25th International Conference on Intelligent User Interfaces, pages 128-138.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2rXiv:2204008866
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 202 ACM conference on fairness, accountability, and transparency, pages 610-623.
Advait Bhat, Saaket Agashe, Parth Oberoi, Niharika Mohile, Ravi Jangir, and Anirudha Joshi. 2023. Interacting with next-phrase suggestions: How suggestion systems aid and influence the cognitive processes of writing. Proceedings of the 28th International Conference on Intelligent User Interfaces, pages 436-452.
Rishi Bommasani, Kathleen Creel, Ananya Kumar, Dan Jurafsky, and Percy S Liang. 2022. Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? Advances in Neural Information Processing Systems, 35:3663-3678.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annic Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Ste- fano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter
10

---

Published conference paper at ICLR 2024
Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zancle Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Sh- iori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga. Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv: 208.:iv:::..0.. 108.07258.
Daniel Buschek, Martin Zür, and Malin Eiband. 2021. The impact of multiple parallel phrase suggestions on email input and composition behaviour of native and non-native english writers. In Proceedings of the 202/ CHI Conference Human Factors in Computing Systems, pages 1-13.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforeement learning from human feedback. arXiv preprint 0rXiv:2307.15277
Tuhin Chakrabarty, Vishakh Padmakumar, and He He. 2022. Help me write a poem: Instruction tuning as a vehicle for collaborative poetry writing In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6848-6863, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Quanze Chen, Chenyang Lei, Wei Xu, Ellie Pavlick, and Chris Callison-Burch. 2014. Poetry of the crowd: A human computation algorithm convert prose into rhyming verse. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 2, pages 10-11.
Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. 2018. Creative writing with a machine in the loop: Case studies on slogans and stories. In 23rd International Conference on Intelligent User Interfaces, pages 329-340.
Kathleen Creel and Deborah Hellman. 2022. The algorithmic leviathan: Arbitrariness, fairness, and opportunity in algorithmic decision-making systems. Canadian Journal of Philosophy, 52(1):26- 43.
E. Durmus, K. Nyugen, T. I. Liao, N. Schiefer, A. Askell, Bakhtin, Chen, Hatficl-Doodis D. Hernandez, N. Joseph, L. Lovitt, S. McCandlish, O. Sikder, A. Tamkin, J. Thamkul, J. Kaplan, J. Clark, and D. Ganguli. 2023. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306. 16388.
Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2023. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. Journal of Artificial Intelligence Research, 77:103-166.
Katy Ilonka Gero and Lydia B Chilton. 2019. Metaphoria: An algorithmic companion for metaphor creation. In Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1-12.
Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. Hafez: an interactive poctry generation system. In Proceedings of ACL 2017, System Demonstrations, pages 43-48.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the of gpt-3. arXiv e-prints, pages arXiv-2209.

---

Published conference paper at ICLR 2024
Jeffrey T' Hancock, Mor Naaman, and Karen Levy. 2020. Ai-mediated communication: Definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication, 25(1):89-100.
Daphne Ippolito, Ann Yuan, Andy Coenen, and Sehmon Burnam. 2022. Creative writing with ai-powered writing assistant: Perspectives from professional writers. arXiv preprint arXiv:2211.05030.
Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co-writing with opinionated language models affects users' views. In Proceedings of the 2023 CHI Conference Human Factors in Compuling Systems, pages 1-15.
Kleinberg and M. Raghavan. 2021. Algorithmic monoculture and social welfare. National Academy of Sciences.
Mina Lee, Percy Liang, and Qian Yang. 2022a. Coauthor: dataset for exploring language model capabilities. In Proceedings of the 2022 CHII Conference Human Factors in Computing Systems, pages 1-19.
Designing a human-ai collaborative writing
Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, et al. 2022b. Evaluating human-language model interaction. arXiv preprint arXiv:2212.09944
X. L. Li, A. Holtzman, D. Fried, P. Liang, Eisner, Hashimoto, L. Zettlemoyer, and M. Lewis. 2023. Contrastive decoding: Open-ended text generation optimization. In Association for Computational Linguistics (ACL).
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110.
Q Vera Liao and Ziang Xiao. 2023. Rethinking model evaluation as narrowing the socio-technical gap. arXiv preprint arXiv:2306.03100
Chin Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries In Text Summa- rization Branches Out. pages 74 74-81, Barcelona, Spain. Association for Computational Linguistics.
C. Meister, Pimentel, G. Wiher, and R. Cotterell. 2023. Locally typical sampling. Transactions of the Association for Computational Linguistics (TACL), 11.
Piotr Mirowski, Kory Mathewson, Jaylen Pittman, and Richard Evans. 2023, Co-writing screen- plays and theatre scripts with language models: Evaluation by industry professionals. In Proceed- ings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-34.
OpenAL. 2023. Gpt-4 technical report
L. Ouyang, Wu, X. Jiang, D. Almeida, L. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203002155
Vishakh Padmakumar and He He. 2022. Machine-ing ceetiine-i--thh-eee thephheloopp rewriting for creative image captioning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Teechnologies pages 573-586.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Association for Computational Linguistics (ACL).
F. Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel. M. Blondel, P. Pretten- hofer. R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapcau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-leam: Machine learning in Python. Journal of Machine Learning Research, 12200
12

---

Published conference paper at ICLR 2024
Melissa Roemmele and Andrew S Gordon. 2015. Creative help: A story writing assistant. In Interactive Storytelling: 8th International Conference on Interactive Digital Storytelling. ICIDS 2015, Copenhagen, Denmark, November 30-December 4, 2015, Proceedings 8, pages 81-92. Springer.
Melissa Roemmele and Andrew S Gordon. 2018. Automated assistance for creative writing with an mn language model. In Proceedings of the 23rd international conference on intelligent user interfaces companion, pages 1-2.
Melissa Roemmele, Andrew S Gordon, and Reid Swanson. 2017. Evaluating story generation systems using automated linguistic analyses. SIGKDD 2017 Workshop on Machine Learning for Creativity, pages 13-17.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Perey Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL.), pages 843-861, Hong Kong, China. Association for Computational Linguistics.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004. 04696.
Irene Solaiman, Zeerak Talat, William Agnew, Lama Ahmad, Dylan Baker, Su Lin Blodgett, Hal Daumé III, Jesse Dodge, Ellie Evans, Sara Hooker, et al. 2023. Evaluating the social impact of generative ai systems in systems and society. arXiv preprint arXiv:2306.05949
Z. Song, T. Cai, J. D. Lee, and W. J. Su. 2023. Reward collapse in aligning large language models. arXiv preprint arXiv:2305.1 17608.
Simeng Sun, Ori Shapira, Ido Dagan, and Ani Nenkova. 2019. How to compare summarizers without target length? pitfalls, solutions and re-examination of the neural summarization literature In Proceedings of the Workshop on Methods for Oplimizing and Evaluating Neural Language Generation, pages 21-29, Minneapolis, Minnesota. Association for Computational Linguistics.
Ben Swanson, Kory Mathewson, Ben Pietrzak, Sherol Chen, and Monica Dinalescu. 2021. Story centaur: Large language model few shot learning as a creative writing tool. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational I Linguistics: System Demonstrations, pages 244-256.
Reid Swanson and Andrew S Gordon. 2008. Say anything: A massively collaborative open domain story writing companion. In Interactive Storytelling: First Joint International Conference on Interactive Digital Storytelling, ICIDS 2008 Erfurt, Germany, November 26-29, 2008 Proceedings 1, pages 32-40. Springer.
Reid Swanson and Andrew S Gordon. 2012. Say anything: Using textual case-based reasoning to enable open-domain interactive storytelling. ACM Transactions Interactive Intelligent Systems (Tii), 1-3)
R. Taori and T. B. Hashimoto. 2023. Data feedback loops: Model-driven amplification of dataset biases. In International Conference on Machine Learning (ICML).
Guy Tevet and Jonathan Berant. 2021. Evaluating the evaluation diversity in natural language generation In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 326-346, Online. Association for Computational Linguistics.
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Worderaft: Story writing with
large language models) In 27th International Conference on Intelligent User Interfaces, IUI '22, page 841-852, New York, NY, USA. Association for Computing Machinery.
13

---

Published conference paper at ICLR 2024
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. |ertscoree Evaluating text generation with bert.
Y. Zhu, S. Lu, L. Zheng, Guo, W. Zhang. J. Wang, and Y. Yu. 2018. Texygen: benchmarking platform for text generation models. arXiv preprint arXiv:1802.0/889
Jacob Ziv and Abraham Lempel. 1977. A universal algorithm for sequential data compression. 1EEE Transactions on information theory, 23(3):337-343.
Jacob Ziv and Abraham Lempel. 1978. Compression of individual sequences via variable-rate coding. IEEE transactions on Information Theory, 24(5):530-536.
A TASK DETAILS
A. USER RECRUITMENT ON UPWORK
We recruit a total of 38 participants from Upwork, all of whom are native English speakers and have prior experience in writing or copyediting on the platform. Their prior experience varied from to over 200 previously completed tasks on the platform. Our participants are exclusively from the United States and encompassing diverse racial backgrounds with an approximately equal distribution between genders. Each participant is asked to complete session consisting of three essay writing tasks-one in each of the three settings and each on a different topic. each session, the order of the three settings and the assignment of topics are randomized. We don't enforce that users include certain amount of machine- generrtee text as we want to study how collaborative writing in its most natural form differs from solo writing without model help. The average machine-writter fraction 0.32 for InstructGPT and 0.35 for GPT (Table 9 indicating that users obtain useful help from the models without over-relying fractions ranging from 0.2 to 0.5 with a few outliers being as low as 0.1 and as high as 0.8 (Figure 13 in Appendix C). Users are allowed to edit their essays post-completion. Their objective is convey their opinions effectively. We acknowledge that this might affect word usage patterns which again motivates the contribution of our proposed key point-based evaluation these key points are less likely to be changed during post-editing, Each essay takes 15 minutes on average and hence a session usually takes the participants under one hour. Compensation was provided through hourly contracts, prorated to $20 per hour. After each participant completed one session, their essays were reviewed manually by the authors of this work for the relevance to the essay prompt as well as coherence of writing. This process does not involve policing the actual content of the essays and we encourage our participants to express their opinions in detail. This was used to filter out a few participants whose sessions were reassigned. The remaining participants were then invited to complete two additional sessions, each with different topics. As result, each the 38 participants completed between 1 and 3 sessions. In total, obtain 10 essays each of the 10 topics for each setting, resulting in 100 essays per setting.
the same. The vast majority of essays have sacchnne-writter
A.2 DECODING PARAMETERS ON OPENAI API
To avoid reduced diversity due to decoding, we use decoding strategy biased towards higher diversity. Specifically, sample continuations from both models following the "high randomness" setting from Lee et al. 2022a). The decoding parameters we use are:
Engine: davinci for the base language model and text-davinci-O00 for the feedback- tuned language model
Response length (word piece): 30
Temperature: 0.9
Top P:
Frequency penalty: 0.5
Presence penalty: 0.5
14

---

Published conference paper at ICLR 2024
Prompt Code
Prompt (Source URL)
school
What Are the Most Important Things Students Should Learn in School? In your opinion, what are the most important things students should learn in school? What the most important thing you have learned in school? How has this knowledge affected your life? How do you think it will help your success in the future? (Link)
stereotype
What Stereotypical Characters Make You Cringe? What stereotypical characters books, movies or television shows make you cringe and why? Would you ever not watch or read something because of its offensive portrayal of somcone? (Link)
audiobook
Is Listening to a Book Just as Good as Reading It? Do you listen to audiobooks? What are the benefits, in your opinion, of listening instead of reading? Are there advantages to reading that cannot be gained by listening? Which method do you prefer? Why? (Link)
athletes
Should College Athletes Be Paid? Do you think college athletes should paid? a college scholarship and other non-monne non-monetary perks like the opportunity to play in front of cheering fans enough? What possible difficulties or downsides might there be in providing monetary compensation players? (Link)
extremesports
Selfish to Pursuc Risky Sports Like Extreme Mountain Climbing? Some sports, like extreme mountain climbing, are dangerous. Since there are varying degrees of risk in most, if not all, sports (such as the possibility of concussions, broken bones and even death), how does decide where the line might be drawn between what reasonable and what not? Are some sports simply too dangerous to be called sport? (Link)
animal
Is It Wrong to Focus Animal Welfare When Humans Are Suffering? Would you be surprised hear that study found that research subjects basehall bat than of adult similarly beaten? Or that other researchers found that forced to choose, percent of people would save their pet dog over a foreign tourist. Why do you think many people are more empathetic toward the suffering of animals than that of people? In your opinion. is wrong to focus animal welfare when humans are suffering? Why do you think so? (Link)
more upset by stories of a dog beaten by
news
Are We Being Bad Citizens If We Don't 't Keep Up With the News? In your opinion, are we being bad citizens if we don't 't keep up with the news? Do you think all people have some responsibility to know what is going in the world? Does engaging with current events actually do anything at all? Why do you think the way you do? (Link)
mindfulness
Should Schools Teach Mindfulness? Have you ever tried mindfulness or meditation, practices that focus on the present moment and being aware of your thoughts, feelings and bodily sensations? If so, what it like for you? If not, does sound like something you'd like try? you think that such practices have place schools? Why or why not? (Link)
screen
How Worried Should We Be About Screen Time During the Pandemic? The coronavirus pandemic ended the screen time debate: Screens won. We all now find ourselves for school, for work and for connecting with family and friends during this time of social distancing and increased isolation. should right now? Or should we finally get over it and embrace the benefits of our digital devices? Link)
sereens
worried about this excessive screen use
dating
How Think Technology Affects Dating? Have you had any experience with dating? Have you ever used dating apps? so, what has it heen like you? not, why not? How you think technology - like apps, Netflix, social media and texting - affects dating and relationships? In your opinion, does it improve or worsen romantic interactions? (Link)
Table 5: List of prompts used in the experiments.
15

---

Published conference paper at ICLR 2024
Libeetigtt book fraeiig
macing
enverlage froth beok whe book people heve pry skay
got twer with rout to visut wit rotiln
wis andertadie drt
aise affono you expeciane norval hoaning
Figure 5: Text editor interface showing suggestions when requested by the user.
A.3 USER INSTRUCTIONS
Essay Requirements To complete an essay assignment successfully, you would need to write a short 3-passage essay given the prompt presented to you. The essay should reflect your opinion of the assigned topic and you't re writing an argumentative piece for why you feel that way. The expected length of the essays is around 300 words (3-4 short paragraphs of 3-4 sentences each). Each piece should take you between 10 and 15 minutes (as calculated from prior experiments). When writing with AI help, you required to obtain model suggestions at least 5 times (we encourage you to do this more if you find helpful, more the better). Some example responses are provided below (though these were written by non-experts I' I'm sure you can identify flaws and improve the style)
Instructions in Detail
When you are completing task, make sure you have a Session ID. If you don't have this already, please send us a message on Upwork.
View the spreadsheet of assignments and navigate your assigned session. Each session will correspond to three rows in the spreadsheet, each with an accompanying URL.
Each of these links corresponds to the three assignment essays you would have to complete in order to finish the task.
If you click of these, you will see the text editor pop up which looks like this which shows the prompt for the assigned essay.
Out of the three assignments, please note that one is the essay that you will have to write without the assistance of the AL. Please complete the three assignments in the order provided.
When you write with the AI, you will have the option to hit the TAB key your keyboard and obtain suggestions from the model in the form of a dropdown like this. You are free to continue the writing process as you normally would, however, use model help when you are looking for ideas when you are stuck or just want to take a look at some possible continuations :)
encourage you to make
When you write with the AI, you need to request least 5 suggestions from the model. You do NOT need to accept all of the suggestions, you are free to accept one then edit it, ask for suggestions again the same time, even just reject all of the model suggestions. We encourage you to use the model as much as possible :) The objective of the study is to understand the effect of the model, so more interaction is better.
Once you complete the essay, make sure you hit 'Save' at the bottom of the screen obtain a verification code on completion. We ask that you complete the writing in one sitting. We aim to record the entire writing process so please refrain from copy-pasting any text into the editor.
16

---

Published conference paper at ICLR 2024
Essay
While believe the concerns regarding children's 's sereen time are valid, believe is somewhat biased to not take this problem, which is genuine issue right now, as everyone problem, and not a student one. With that mind, believe that for the majority our generation it would not as hard to disconnect with devices after the pandemic.
I know that I, along with many other teenagers, would like nothing more than to go back to school, play sports outside, meet new people and such. doubt could say the same for our older generations, because while parents ban teenagers from using their phones, they often spend many more hours their devices. Cranky grandparents grumble about how 'back in their day they lived without mobile phone' usually forget the point of their stories. Back in their day indeed.
We live in different era, and think that the same rules should apply teenagers, especially during Youtube, Twitter aren't 't just places where teens while away their
pandemic, isn't 't time. They are also places for new ideas, watching college lectures, and political discourse. Limiting screen time, while in the short run is helpful, is doubtful to do anything in the long run. Banning technology not stopping kids from using it. It's just stopping them from telling their parents about it.
I would like to end by saying, parents, please trust your teenager just little bit more, and don't 't be too worried about their screen time!
Keypoints
The concerns regarding children's 's screen time are valid
The problem of screen time should be considered an everyone problem, not just a student one
For most young people, it would be hard to disconnect from devices after the pandemic
Older generations often spend more time on their devices than teens, despite their complaints
Social media can be used for educational and constructive purposes
Limiting screen time may not be effective in the long run
Parents should trust their teenager more and not worry too much about their screen time
Table 6: Example of summarization into keypoints
A.4 SUMMARIZATION INTO KEY POINTS
Motivated by recent encouraging results of using LLMs for zero-shot summarization (Goyal et al. 2022), summarize each essay into list of key points by prompting apt-3.5-turbe Table 6 6)con- tains a full example of an essay with the generated key points. We use a simple prompt, "Summarize this essay into a of simple and distinct bullet points. Make sure that the bullet points cover all the information from the essay."
B SIGNIFICANCE TESTING
B. PERMUTATION TEST FOR SIGNIFICANCE OF DIVERSITY RESULTS
To test for the significance the results on diversity (Section 5.2). since we do not have multiple instances of essay corpora from each setting (Solo, GPT3 and InstructGPTT,
employ three permutation tests between each pair of settings. walkthrough example of the permutation test setup as follows. We wish evaluate the significance of the difference between the diversity scores of Solo and InstructGPT, as measured by the clustering of key points (Table 3 or by traditional lossless
17

---

Published conference paper at ICLR 2024
compression algorithms (Table 13 scores) on the collected corpora of essays. We then take the union of Solo and InstructGPT essays, randomly partition this into two equal sets, and recalculate the statistic on these two permuted sets of essays. This process then repeated for 1000 different permutations. We then obtaín the p-value of this two-tailed permutation test as the proportion of times the absolute value of the statistic calculated on the permuted data is greater than the statistic calculated on the observed data from the user study. This is then repeated for all pairs out of Solo, GPT3, and InstructGPT. Bold values in the results tables (Table 3 Table 13) indicate those instances where the p-value on the permutation test for a setup (InstructGPT) significant at the 5% level over both other setups (GPT3 and Solo).
We first calculate the statistic (difference between the diversity
B.2 CHI-SQUARE TEST FOR SIGNIFICANCE FOR n-GRAM DISTRIBUTIONS
To confirm the significance of the difference between the categorical distributions of n-grams in the different setups, we employ a chi-square test on the count of occurrences. We evaluate if writing with InstructGPT results in change of 5-gram usage in Figure and Figure To perform this test, we first identify and take the union of the 50 most frequently occurring 5 b-grams in each setup. We then obtain the counts of occurrences of each of these 5-grams in both corpora and perform a chi-s test for significance on these frequencies. A p-value < 0.05 results in a rejection of the null hypothesis and the observation of a significant difference the categorical distributions. We perform these for all pairs out of Solo, GPT3, and InstructGPT and find that the repetition of n-grams when writing with InstructGPT is more similar by a significant margin compared to both other setups (Figure
B.3 SIGNIFICANCE TESTING FOR RESULTS ON HOMOGENIZATION
As noted in Section 2.2 and Appendix A.I we recruit fixed set of users who wrote essays in each of the three setups in a randomized order. We ensure that they write essays on different topics in each setup prevent a repetition of their opinions. To test for the significance of the change in homogenization across all topics between the three setups, we conducted independent samples I-tests in Figure To further test the significance of this result, we observe that within each topic, each writer only submits an essay to one of the three setups. This matches the between-group experimental setup, albeit with less power as we only collected 10 essays per setup per topic. We compute the significance of the difference in homogenization scores per topic via an independent samples t-test. We report the p- p-lalues comparing each pair of setups within each topic at both the key point (Table and essay level (Table 8) Comparisons with significance at the 5% level are marked with an asterisk. We observe that the InstructGPT setup exhibits higher homogenization over Solo writers with significance at the 5% level on 8 out of 10 topics at the raw essay level and 6 out of 10 at the key point level using both Rouge-L and BertScore. The same trend holds on GPT3 on 7 topics at the raw essay level and 5 at the key point level. We believe that this confirms the overall trend that writing with InstructGPT results in higher homogenization in the essays created.
C ADDITIONAL RESULTS
Reporting more basic statistics about the collected essays We aim to improve the comprehen- siveness of our analysis by reporting additional metrics on the collected essays in Table 9
We compute basic statistics such as perplexity (via GPT2), average sentence length, and essay length on the essays collected. Predictably, writing with model-help reduces the perplexity of the essays which now incorporate suggestions sampled from an LM distribution. The total length of essays, in terms of word count, is roughly similar, with writers writing slightly shorter sentences when writing with InstructGPT (albeit with a slightly high standard error).
Additionally, by calculating the average height of the dependency parse trees of the sentences in the essays, we observe that writing with InstructGPT results in sentences with fewer nested structures, indicative of slightly lower complexity.
Writing with model help also results in fewer unique POS-Ngrams again providing some evidence that collaborative writing could result in more homogenized usage of language. The effect here is even between GPT3 and InstructGPT, a slight departure from the results
18

---

Published conference paper at ICLR 2024
Significance Testing Key point
BortScore
Topic
Solu strucIGPT
InstrucGPP
Topic
Sulu
InstraoGPP
Solo
Solo
Stereotype
0.02*
0.34
0.01
Sterrotspe
0.18
0.81
School
0.03*
0.05*
0,86
School
0.78
0.15
0.17
Mindfulness
0.00"
0.00
0.03
Mindfulness
0.00-
0.11
0.64
Andiobouk
Audiobook
0.00-
0.06
0.01
Animal Welfare
0.67
0.13
0
Animal Welfare
0.31
0
0
Extreme Sports
0,03*
0,02
0,93
Extreme Sports
0.43
0.11
0.58
Time
Dating
0,17
0
0
Dating
0
0
0.41
Citizens
0.51
0.08
0.16
Citizens
0.16
0.11
0.87
Table 7: Evaluating the significance of the difference of homogenization between the three setups across each topic (Corresponding the box plots in Figure 9.. We conduct an independent samples t-test to compare the document homogenization values between the setups pairwise. Pairs with a significant difference at the 5% level are highlighted with an asterisk. The overall trend that InstructGPT has significantly higher homogenization across a majority of topics.
Significance Testing Raw essay level
Rouge-L.
BertSicore
Topic
InstructGPP
GPT3
Topic
Solo
InstraciPPT
GPT3
Storentype
0.02*
0.01
0.00
Stereotype
0.02-
School
0.17
0.12
0.71
School
0.14
01.22
0.62
Mindfulness
0
0,1*
0.42
Mindfulness
0.02-
0.22
0.8]
Atbletes
0.00-
Andiobook
Audiobook
A Animal Welfare
0.49
0.11
0.15
Animal Welfare
0.13
0
0
Extreme Sports
0.35
000*
0.00
Extreme Sports
0.04-
0.00
0.09
Sereen
0.00*
0.00-
Dating
0
0.61
0.00
Dating
0
0.02*
Ciiizens
0.00
0.14
0.11
Chiizens
0.000
0.04
0.11
Table 8: Evaluating the significance of the difference of homogenization between the three setups across each topic (Corresponding the box plots in Figure 8 We conduct an independent samples I-test to compare the document homogenization values between the setups pairwise. Pairs with significant difference at the 5% level are highlighted with an asterisk. The overall trend is that InstructGPT has significantly higher homogenization across a majority of topics.
19

---

Published conference paper at ICLR 2024
Solo
GPT3
InstructGPT
Perplexity Essays (via GPT2)
25.067
22.10
20.26
Sentence Length (in words)
23.51 (0.25)
23.66 (0.24)
22.51 (0.24)
Height of Syntax Tree
5.93 (0.06)
5.98 (0.06)
5,71 (0.05)
Essay Length (in words) 376.44 (8.03)
380.87 (9.30)
368.39 (8.43)
Relevance to Prompt
4.30
4.15
4.10
Human Ratings
Grammaticality
4.00
4.10
4.00
Depth of Discussion
3,95
3.85
3.80
1
73
56
58
2
487
437
426
Unique POS Ngrams
3
1297
1261
1235
4
2044
1975
1988
5
2423
2338
2414
Table 9: Reporting basic statistics about the collected essays. Writing with the help of a model reduces the perplexity (measured via GPT2) of the essays in InstructGPT and GPT3. We also observe that users write essays of similar length across all setups with slightly shorter sentences in InstructGPT. We also examine the height of the dependency parse trees of the sentences in each setup to find that writing with InstructGPT also results in sentences with less complcxity average numbers, we report the standard errors brackets. We also find that writing with model help results in fewer unique POS-n-grams. Finally, rate the quality of one essay per topic per setup (selected randomly) along three different dimensions (grammaticality. relevance to the prompt, and depth of discussion) find that there is significance in essay quality across setups.
average. Along with these
on diversity which is more focused on content. We finally plot the distributions of the 50 most frequent POS-Ngrams (Figure 6 and find that on 4 and 5-grams, writing with model help leads to higher repetition of frequent POS-Ngrame
To ensure that the quality of the essays is not confounder in the result, we present human ratings for random sample of 10 essays (one topic) from each setup that we conducted as part of our quality control checks. Here we judged the essays to the topic, grammaticality, and the depth of discussion on their opinions. This was per the advice from faculty at the Expository Writing program at our university. This analysis showed no clear differences between the setups, explained in part due to the observation that the machine- written fraction in the collaborative essays was on average around 32%-35% (Table
different relevnceev
Writing with InstructGPT results in more similar content at the key point and raw essay level In addition the results presented in Section 4.2 we also report the corpus homogenization scores at the key point and essay level in Table 1O Across both Rouge-L. and BertScore, the corpus of essays written with InstructGPT exhibits higher corpus homogenization than both other groups by statistically significant margin (p-value < 0.05) on both levels. We also plot the homogenization scores of all the essays calculated at the raw essay and key point levels in Figure S and Figure We choose report the homogenization at the key point level mainly because string similarity metrics such as Rouge-L and BertScore are less reliable on longer text documents (Sun et al. 2019 Gehrmann
al. 2023
Essays written with InstructGPT repeat higher- order n-grams more frequently Building on from the results in Section 5.2 we plot the r--ram distributions of the raw essays varying from I to in Figure 10 Here we truncate the distribution to the most common 50 n-grams from each setup. While the distributions for 1, 2 and 3-grams are almost identical across the setups, 4 and 5-grams we see a concentration of probability mass at the head of the distribution for the essays written with InstructGPT. The reduction in lexical diversity (Table Ba is manifested in this increased repetition of higher-order n-grams.
20

---

Published conference paper at ICLR 2024
Sale
0.04
Solo InstndGGPT
IntructGPT
Probanilty
0.025
0.000
30
1-grama decreasing order frequency
2-grams decreesing order of frequency
(a) Unigram Distribution
(b) Bigram Distribution
Sale
Solo
0.016
InttudGPPT
GPT3
GPT3
Probabity
Piobebiity
0.004
0.010
0,003
0.005
0.002
3-grams decreasing order frequency
4-grams decreasing order trequancy
(c) Trigram Distribution
(d) 4-gram Distribution
Solo
0,0020
GPT3
Probsbiliyy
0.0016
0,0010
5-grame decreasing frequency
(e) 5-gram Distrihution
Figure 6: Distribution the top-50 POS n- grams in the essays from the various setups varying from to 5. While the lower order n- n-gram usage patterns are similar across groups, essays written with InstructGPT exhibit higher repetition of common 4- and 5-POS -ggaass.
21

---

Published conference paper at ICLR 2024
Fraction of Kay Points by Models
1.0
0.8
0.6
0.4
0.2
GPT3
InstructGPT
Figure 7: Boxplot of the fraction of key points contributed by the model in each essay. Both models contribute a considerable amount of key points. However, the high variance suggests varied reliance on the model by different users.
Solo
GPT3
Instruct- GPT
Key point Level
Rouge-T. 0.1536
0.1578
0.1660
BertScore
0.6305
0.6284
0.6437
Essay
Rouge-L 0.1498
0,1523
0.1621
Level
BertScore
0.6044
0.6130
0.6318
Table 10: Corpus homogenization scores comparing essays from the three setups at the key point and essay level using both Rouge-L and BertScore to calculate homogenization. Essays written with InstructGPT exhibit higher homogenization than Solo and GPT3 across both levels and similarity metrics by statistically significant margin (p-value 0.05).
compressible. We also show that the reduction of diversity measured by linguistic units such as n-grams and key points also correlates with less diversity an information-thorrtte sense. Specifically, measure the compression ratio of essays written in the three settings using various lossless compression algorithms: LZMA, ZLIB, and GZIP, each being variation the Lempel-Ziv algorithm (Ziv and Lempel, 1977; 1978). These algorithms compress data by identifying repeated patterns and using dictionary mappings for encoding. To compute the compression ratio, we concatenate all essays from setting into single text file. We then run the compression algorithm on it and calculate the ratio of the compressed file size to the original file size. From Table 13 we see that the InstructGPT essays are consistently more compressible across all three compression algorithms, which is statistically significant with p < 0.05 according to permutation tests (Appendix B.I)
Essays written with InstructGPT
InstructGPT presents less diverse suggestions users than GPT3 To test the hypothesis that adapting model with human feedback reduces the diversity of the presented suggestions. calculate the average similarity of all pairs of the five suggestions presented by InstructGPT and GPT3 upon each user query. In addition to the similarity plotted with Rouge-L in Figure 12 we also plot the average similarity scores of suggestions computed using BertScore in Figure 12(b)
Higher correlation between AI written fraction of the document and homogenization, on InstructGPT than GPT3 To study the relationship between increased model intervention and homogenization, we plot the fraction of the essay written by the model as a function of the document homogenization with BertScore and Rouge-L in Figure [3 We observe weak correlation when users write with InstructGPT, particularly in the case of BertScore.
22

---

Published conference paper at ICLR 2024
0.220
GPT3
InstructGPT
Homogenizalon Scores
0.200
0.180
0.160
0.140
0.120
0.100
0.080
Total (*)
Citizens
Dating
Screen Time
Extreme Sports
Animal Welfare
Audiobooks
Collage Mindfulness School Athletes
Stereotypes
Homogenization calculated with Rouge-L
0.725
InstrucGPPT
Homegentzallon Soores
0.700
0.675
0.850
0.625
0.600
0.575
0.550
Total (")
Citizens
Daling
Screen lime
Extreme
Animal Welfare
Audiobaoks
Callege Mindfulness
School Stereolypes
Sports
Athletes
(b) Homogenization calculated with BertScore
Figure 8: Boxplots of homogenization scores for all three groups (Solo. InstructGPT, GPT3) when comparing the raw essays and calculated using (a) Rouge-L and (b) BertScore measures of similarity. The left-most column (Total) shows essay homogenization scores for all topics and the other columns show essay homogenization scores by topic. Essays written with InstructGPT exhibit higher corpus homogenization by a statistically significant margin (Section 4.2
23

---

Published conference paper at ICLR 2024
0.220
GPT3
InstructGPT
Homogenizalon Scores
0.200
0.180
0.160
0.140
0.120
0.100
0.080
Total (*)
Citizens
Dating
Screen Time
Extreme Sports
Animal Welfare
Audiobooks
Collage Mindfulness School Stereotypes Athletes
Homogenization calculated with Rouge-L
0.725
InstrucGGPT
Homegentzallon Spores
0.700
0.675
0.850
0.625
0.800
0.575
0.550
Total (")
Citizens
Daling
Screen
Extreme
Animal Weffare
Audiobaoks
College Mindfulness
School Stereolypes
Time
Sports
Athletes
(b) Homogenization calculated with BertScore
Figure 9: Boxplots of homogenization scores for all three groups (Solo. InstructGPT, GPT3) when comparing essays the key point level and calculated using (a) Rouge-L and (b) BertScore as measures of similarity. The left-most column (Total) shows essay homogenization scores for all topics and the other columns show essay homogenization scores by topic. Essays written with InstructGPT exhibit higher corpus homogenization by a statistically significant margin (Section 4.2
24

---

Published conference paper at ICLR 2024
0.04
Sale
0.004
Solo
IntructGPT
InstndGGPT
Probabilty
6.003
0,002
0.11
1-grams in deecressing order of frequency
2-grams in decreasing order frequency
(a) Unigram Distribution
(b) Bigram Distribution
0,0025
Sale
0.0006
Solo
InttudGPPT
Pobabity
0.0020
GPT3
Probetillty
GPT3
0.0004
0,0010
0. 0.0055
0,0002
3-grams decreasing order fraquency
4-grams decreasing order of frequency
(c) Trigram Distribution
(d) 4-gram Distribution
Sole
0.0003
GPT3
Probabiity
0.0002
0.0001
5-grame decreasing frequency
(e) 5-gram Distrihution
Figure 10: Distribution of the top-50 n- n-grams in the essays from the various setups varying from 1 to 5. While the lower order n-gram usage patterns are similar across groups, essays written with InstructGPT exhibit higher repetition common 4-ngrams and 5-grams.
25

---

Published conference paper at ICLR 2024
Contribtion
GPT3 Contribution
strrupt Contibutio Humans
0.004
InstrucGPT Contribution Humans
Human Contribuion GPTS
Human Contribution wi GPT3
Probabilty
008
Contiibtionn
Probabilty
Contribttin
Solo
Salo
0.001
10
20
30
20 2-grama decr eesling order of frequency
10
1-grams decteasing order of frequency
(a) Unigram Distribution
(b) Bigram Distribution
0,0012
GPT3 Contribtion we Humans
GPT3 T3 Contribution w/ Humans
0.0010
Human Contribution w/ GPT3
0.0008
Human Contribution wi GPT3
Probabiity
Human Contrbution InstructGPT
Probability
Human Contribttio wi InstruciSPT
0,0000
0.0005
0.0008
0,0004
0.0004
0,0002
10
20
30
40
50
10
20
50
B-grams in decreasing order frequency
4-grams in decreasing order of frequency
(c) Trigram Distribution
(d) 4-gram Distribution
0.0008
InstrutSPT Contribuion Humans 0.0006
Probabiity
Humann Comtibution w/ InstuaGPT
0.0004
0,0002
5-grame decreasing frequency
(e) 5-gram Distribution
Figure 11: n-gram distribution of tokens introduced by the and the model during the writing setup with InstructGPT and GPT3 respectively. The distribution human-written n-grams from Solo essays is also provided as a reference. The distribution of user-written in all settings is similar to each other regardless of model assistance, whereas InstructGPT contributed text notably has larger probability mass on common 4, and 5-grams. This indicates that the reduced lexical diversity in the InstructGPT group is mainly due to model introduced text.
0.50
Avg Pairwise Similarit
GPT3
InstrucIGPT
Avg Pairwise Similarity
0.80
GPT3
InstrucIGPT
0.40
0.30
0.70
0.20
0.60
0.50
0.00
GPT3
InstruciGPT
InstrucIGPT
(a) Rouge-L
(b) BertScore
Figure 12: Boxplot of average pairwise similarity of suggestions provided by InstructGPT and GPT3 when calculated using (a) RougeL and (b) BertScore as a measure of similarity. InstructGPT presents more similar suggestions average the user by statistically significant margin.
26

---

Published conference paper at ICLR 2024
606
a
Dasa au an
Figure 13
GPT3
InstrucIGPT
Homogenization Scores
0.250
-
0.150
0.100
0.050
0.000
Total
Screen
School
Audiobaoks
Callege Athletes
Dating
Stereotypes
Extreme Sports
Animal Welfare
Citizens Mindfulness
(a) Homogenization calculated with Rouge-L
0.800
GPTS
InstructGPT
Homogenizateo Scores
0.700
0.500
0.400
0.300
Total
Screen Time
School
Audiobooks
College
Dating
Stereolypes
Extreme
Animal Walfare
Ciüzens Mindiulness
Athietes
Sports
(b) Homogenization calculated with BertScore
Figure 14: Boxplots of homogenization scores comparing essays generated solely via InstructGPT and GPT3 models the raw essay level and calculated using (a) Rouge-L and (b) BertScore as measures of similarity. The left-most column (Total) shows essay homogenization scores for all topics and the other columns show essay homogenization scores by topic. Essays written with InstructGPT exhibit higher corpus homogenization by statistically significant margin.
27

---

Published conference paper at ICLR 2024
n-gram size
Solo
GPT3
InstructGPT
1
0.119
0.116
0.115
2
0.602
0.585
0.579
3
0.898
0.886
0.869
4
0.973
0.967
0.953
5
0.991
0.988
0.977
Table 11: N-gram diversity in each setting. Bold values indicate the lowest diversity score as measured by the fraction of unique n-grams, Essays written with InstructGPT are the least diverse across n-gram sizes.
Thresholds
Solo
GPT3
InstructGPT
Thresholds
Solo
GPT3
InstructGPT
0.5
0.982
0.971
0.950
0.1
0.998
0.997
0.992
0,6
0,941
0.927
0.877
0.2
0.981
0.976
0.941
0.7
0.792
0.779
0.738
0.3
0.805
0.787
0.730
0.8
0.543
0.514
0.494
0.4
0.321
0.338
0.292
(a) RougeL
(b) BertScore
Table 12: Diversity measured by agglomerative clustering of the key points of essays with (a) Rougel. and (b) BertScore as the metric. We report the scores using different distance thresholds for clustering. Bold values are different to other columns by a statistically significant margin (p < 0.05) using permutation test. InstructGPT consistently exhibits lower content diversity across both similarity metrics and all selected thresholds.
D LIMITATIONS
Interaction interface. Our interface provides suggestions to users in the form continuations of the current text in the draft. Further investigation is needed evaluate if the reduction in content diversity with feedback-tune models can be mitigated with prompt engincering or richer forms of interaction (e.g., through dialogue).
User selection. The outcome of the experiments can be affected by the specific group of participants. We try ensure a diverse user group and detail our recruitment procedures Appendix(A.] However, it is unclear whether these results will generalize to other groups such as students learning to write or second language speakers, who have different goals and incentives for using writing assistants.
LLM access. Our experiments conducted using two limited-access models from OpenAL. While we believe they are representative of properties of current LLMs, is possible that the other models may exhibit different behavior, especially given that the RLHF pipeline is highly customized.
Algorithm
Solo
GPT3
InstructGPT
LZMA
0.305
0.302
0.293
ZLIB
0.353
0.351
0.342
GZIP
0.352
0.341
0.350
Table 13: Compression ratios of the essays using various lossless compression algorithms. Bold values are different from each the other columns with statistical significance (p < 0.05) using permutation tests. InstructGPT exhibits lower diversity, i.e. lower compression ratios, than GPT3 and InstructGPT across all three compression algorithms.
28


Reviews:
[{"review_id": "LbBRgako64", "reviewer": "Reviewer_ttyL", "summary": "This paper investigates whether writing assisted by large language models (LLMs) leads to reduced diversity in the composed text. It uses a within-subject design (but see my comment about the confusion below) that includes three groups of participants that wrote essays without using an LLM, a group that used a base LLM (GPT3) to assist writing, and a group that used an LLM tuned with human feedback (InstructGPT) to assist writing. The main finding of the paper is that writing with InstructGPT leads to significantly more homogeneous context and less diverse n-gram (n>1) distribution and semantic key points (as determined by prompting gpt-3.5-turbo) than writing without an LLM and writing with the GPT3 base LLM. The authors further attributed the increased homogeneity and reduced diversity to the portions of text contributed by the LLM (instead of those portions contributed by the human).", "questions": "Q1. See my comments in W1 above. Please clarify whether the study design was between-subject or within-subject. Please ensure that the proper statistical analysis is used accordingly.\nQ2. Were the base LLM (davinci) and feedback-tuned LLM (text-da-vinci-003) based on the same model architecture and the same parameter count to make it a fair comparison between the texts composed with them?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "S1. Taking an empirical approach to answer the interesting and important question about how humans interact with LLMs when composing text. This is a first investigation in this topic area to my knowledge.\nS2. Defining a number of metrics to quantify the diversity and homogeneity of text written under different human-LLM collaboration settings, encompassing both the lexical aspect (n-grams) and semantic ones (key points extracted by gpt-3.5 and BertScore based similarity).", "weaknesses": "W1. There are some confusion / lack of clarity about the experiment design. While Appendix Sections A.1 and A.3 clearly indicate that the authors used a within-subject design wherein each participant wrote text under all three settings, the text on p. 1 (\"control group\", \"LLM treatment group\", and \"feedback-tuned LLM treatment group\") seems to indicate that the study was based on a between-subject design, where nonoverlapping sets of participants were tested under the three different settings. This confusion is further acerbated by the fact that the authors used independent-samples t-test for statistical analysis throughout this manuscript, which indicates this is a between-subject design. If the study was really based on a within-subject design, as Sections A.1 and A.3 indicate, then the statistics should be redone with repeated-measures ANOVA and paired t-tests, which may alter the significance test results of this paper. \nW2. While reading this manuscript, I can't help but feel some duplication between Sections 4 and 5, which focuses on \"homogeneity\" and \"diversity\", respectively. Although these two terms usually refer to slightly different aspects of a collection of text, the authors never tried to define them and how they are related to and different from each other in this paper. So in the end, the reader has to read and understand that \"homogeneity\" is operationally defined as the average of a semantic distance metric (e.g., BertScore) among pairs of the composed text in the corpus, while diversity refers to both lexical and semantic aspects, and are calculated in a different way (# of unique lexical or semantic units). \nW3. The user interaction paradigm is limited to generating and adopting continuations with the LLM (whether base or feedback-tuned), as described throughout the paper and in Section A.3. However, anecdotally we know that users of LLM and associated chatbots (e.g., ChatGPT and Bard) often prompt the models to write or rewrite text by using their own prompt templates or instructions. The flexibility is higher in such prompting-based interactions than in the more stereotypical continuation paradigm. The authors did not mention such instruction- and prompting-based interaction paradigms, which are arguably more prevalent and useful compared to simple continuation.\nW4. To evaluate the text composed by the study participants, the authors used only homogeneity and diversity metrics. They didn't report more basic metrics of the composed texts, which IMO should also be reported. These include the length, the perplexity (e.g., as determined by GPT3), and writing quality (e.g., grammaticality and other human rater-based writing quality) metrics. These basic metrics are not only important for ensuring comprehensiveness of the result reporting, but also useful for ruling out the possibility that differences in homogeneity and diversity are somehow due to the confounding factors due to differences in those basic aspects of writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QQ9EYJlTru", "reviewer": "Reviewer_qs9Z", "summary": "The authors measure the diversity of outputs in LLM-assisted essay writing. They compare three essay setups: written solely by a user, written by a user + gpt3, and written by a user + instruct-gpt. They find that essays written using instruct-gpt are significantly less diverse than the others. Disaggregating the text by LLM-authored or user-authored shows that this lack of diversity comes from the model itself (ie, the user is *not* more repetitive when using instruct-gpt). They also present a novel diversity metric (by first summarizing the essay into \"key points\") to measure the diversity of the essay on a structural level.", "questions": "- If the key points are generated with GPT3.5, aren't they inherently more likely to be similar to other text generated with GPT3 or ChatGPT than with human text? If so, won't that skew the results?\n- (More questions in \"weaknesses\")", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It sometimes feels like there's a mismatch between how the academic world studies and evaluates LLMs, and how LLMs are actually used in the real world. This paper bridges that gap, and directly asks important questions that are relevant to today's users of ChatGPT.\n- It's interesting that you use the LLMs as part of the process in generating the key points .\n- The diversity metric of \"key points\" that the authors present is useful and interesting-- as LLMs become ubiquitous, it's clear that measures of text diversity/similarity/etc are often insufficient, so this type of work can have a significant impact.", "weaknesses": "- If we know that instruct-gpt generates lower-diversity text, then I think I am missing a bit of why this is a surprising result overall.\n- I'm not convinced that these results can be extrapolated to other LLM-based writing tools. When designing a good tool, questions like \"is it limiting what the writer wants to express\" are some of the things you'd first think about addressing. In fact, other work has found that having the LLM can be helpful for getting users out of their normal writing patterns [(e.g.)](https://wordcraft-writers-workshop.appspot.com/learn). But then again, ChatGPT is so ubiquitous that maybe it's enough to just evaluate it alone.\n- Having a precise character number on the text written by the model vs the user can be misleading. For example, what if the model generates a sentence, and the user fully rewrites it but keeps the underlying meaning? (having worked on similar tools, this seems to happen frequently)\n- Nit: Figure 4a-- it's confusing that the \"InstructGPT\" and \"GPT3\" color key is horizontal and lines up with the vertical center division line (I thought maybe GPT3 was the label for the left half or something). Maybe just put them on top of each other like 4b?\n- It would be good to compare to essays written entirely with instruct-gpt and GPT3\n- Figure 4b is generally kind of confusing, especially the x axis. Can you just simplify and say \"average probability for top 10 ngrams\" or something? (This is just a suggestion-- I'm not sure it's actually better. But the current version could use more explanation).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F6yIlSPotf", "reviewer": "Reviewer_vExG", "summary": "The paper describes extensive work on the evaluation of language models for content diversity and homogenization. The work defines homogenization as the degree of similarity between the produced content of users co-writing with LLMs measured using overlap metrics such as Rouge-L and BERTScore; while diversity as the degree of linguistic variation of the text measured by n-grams and/or type token ratio (TTR). The authors cite the main motivation of the study being to assess the quantifiable degree of which LLMs reduce content diversity and homogenization in a co-writing scenario. The author/s then conduct a controlled experiment to gather data in 3 setups across several thematic prompts: writing essays w/ LLM assistance (solo), using GPT as is (GPT-3), and using a GPT model finetuned for instruction-following (InstructGPT). From the collected data, the author/s perform several experiments geared towards answering the research questions on (a) how users often engage with the model, (b) homogenization between essays from the three setups, and (c) differences in diversity between the two generative models used in the co-writing scenario. In terms of homogenization, the results of the paper show that using instruction-tuned GPT models like InstructGPT produces much more similar content compared to using these models (obviously) and using regular GPT-3 models without the instruction-tuning. In terms of content diversity, using InstructGPT does have some tendency to reuse word combinations, as evidenced through the n-gram analysis by the authors, and shows a considerable reduction in key-point and lexical (word-based) diversity. Overall, the paper paints a picture of an empirical evidence of how using these models in a co-writing setup affects the quality of texts we co-produce with them.", "questions": "1. Once the essay has been completed, were the users required to reread and revise the essay for another round? This may negatively bias the result with the n-gram pattern analysis as sometimes users may freely switch words for more fitting synonyms in phrases while the GPT-generated remains constant.\n\n2. Are the writers recruited classified as native speakers of English? There are multiple claims in literature (https://arxiv.org/abs/2304.02819, https://hai.stanford.edu/news/ai-detectors-biased-against-non-native-english-writers, https://arxiv.org/abs/2306.04723) stating that there are similarities with writing styles of non-native speakers of English with machine-generated texts. Recruiting non-native English speakers in the mix may introduce some bias.\n\n3. Why is the model-suggested text not required for groups using GPT models? Does this mean that there are possibilities that some passages in those two groups (users using GPT and InstructGPT) have fully human-written text since they are not required to use the model generations anyway?\n\n4. For writers that are in the group using LLMs, is there a checker to check whether their outputs are a certain percentage of human and LLM-generated content? If a writer fully made use of the LLM generations and then only contributed one to two sentences, would this be considered a qualified entry for analysis? I understand that the experiment is on co-writing but it would be pointless then if more than 80% of the submitted text passage has been written by GPT.\n\n5. Is there a possible method that you can integrate to quantify the usable words forming the overall vocabulary for each language model? I believe this would greatly support the claim that using LLMs may reduce diversity and creative in writing. In addition, what does the literature say about the quantifiable reduction of the cost of instruction-tuning LLMs to the degradation of its diversity? The papers made a mention of this but only at a surface-level detail without going deeper into its discussion.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written and structured, and the narrative flows smoothly. I had no problem understanding the picture the author/s are constructing, starting with the problem definition, motivation, methodology, and the discussion of findings. I believe anyone, even a non-technical audience from interdisciplinary fields, will fully understand and appreciate the results of the paper in one sitting.\n\nAs highlighted, one of the most important contributions of the paper is the dataset built from the controlled experiment from the three different groups (solo, using GPT3, using InstructGPT). I find value in the availability of this dataset for public use. \n\nI find the quantitative and controlled experiment with how LLMs affect the diversity of texts, especially in a co-writing setting explored in the study, to be extremely useful and timely. It is a common intuition and suspicion in the education community that LLMs do reduce creativity when used for content-generation tasks such as writing essays, but this study fully supports and grounds that intuition with succinct and convincing experiments.", "weaknesses": "One thing that feels quite lacking and limited in the paper is that the metric for diversity only focuses on word-based evaluation through n-grams and keypoint analysis. As a reader, I was expecting a more expansive set of linguistic features to be explored to really shed light on the diversity of generated texts from (Instruct)GPT models. This includes syntax or parse tree-based differences and some perspectives even on information theory by measuring the suprisingness (entropy) of human vs machine-generated texts. Well-known previous works in text generation such as (https://arxiv.org/abs/1904.09751) have shown that human-written texts are naturally surprising–which can be a prerequisite for its texts being diverse–therefore exploring this form of feature with the co-writing setup done by the author/s is necessary. \n\nThere is a dedicated section for the author/s’ inferences as to why the use of InstructGPT produces less diverse content, but it doesn’t supply the expectations of the reader. I’m quite confused with this takeaway “The reduction in diversity in collaborative writing is attributed more to text contributed directly by the InstructGPT model.” This is an obvious case. Upon reading Section 6, I was expecting more of a deeper discussion as to why InstructGPT reduced diversity more than models not trained for following instructions, such as a regular GPT-3, which can be traced back to some analysis about the quality and characteristics of data used for training these models. In order to level the playing field and to compare these two models equally, you may need to extensively compare their generated content to even more fine-grained linguistic analysis beyond word-based patterns such as n-grams or type-token ratio.\n\nThe paper makes heavy use of the Co-Author system by Lee et al (2022), and it seems in the paper that the authors of the system have already done a similar experiment on co-writing with LLMs, especially on the lens of creativity, vocabulary, and grammaticality. The author/s of the submitted paper have only extended this through the addition of homogenization and diversity analysis. I’d like to see more clear and explicit differentiation of the work compared to what analysis has been in the Co-Author paper so that it does not seem to be incremental. I invite the authors to discuss this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)