Who Writes Better? Exploring Creativity in Human and AI-Authored College Essays
Creators: name: Moon, Kibum
name: Kushlev, Kostadin
Publisher: name: Open Science Framework
Year: 2023
Subjects: subject: Science and Technology Studies
subject: Communication Technology and New Media
subject: Psychology
subject: FOS: Psychology
subject: Social and Behavioral Sciences
subject: Communication
subject: Social Psychology
subject: LLM
subject: Creativity
subject: NLP

description: Study Name: Who Writes Better? Exploring Creativity in Human and AI-Authored College Essays 1) Have any data been collected for this study yet? ☒ (b) Some data have been collected but not analyzed If (b) or (c), please explain briefly: We will use archival data from college students' admission essays that have already been written. We will also generate our own data prompting ChatGPT to generate college admissions essays in response to the same prompts. None of these data has been previously analyzed. 2) Hypothesis: What’s the main question being asked, or hypothesis being tested? The main goal of this study is to examine how the creativity of AI-powered Large Language Models (LLMs) like ChatGPT compares with human creativity. For this study, we will compare the creativity of college admissions essays generated by actual college applicants with AI-generated essays responding to the same prompt. Building on previous research that has compared AI and human creativity on an individual level, our investigation aims to compare the aggregated creativity across multiple human- and AI-generated content. In a pilot study, we analyzed 100 college admission essays written by human applicants, comparing them with 100 essays written by ChatGPT. Based on the results of this pilot study, we have posited the following hypotheses: H1: Human-written essays will show a higher level of creativity compared to AI-generated essays. H2: Aggregating multiple human-written essays will have a lower declining marginal utility than aggregating multiple AI-generated essays. In other words, at the point at which adding more AI-generated essays will stop increasing aggregated creativity, adding more human generated essays will continue to increase aggregated creativity. 3) Give a brief (1-2 sentence) overview of the methods. Measuring Individual Creativity To quantify the level of creativity for each essay, we will employ the semantic distance approach called Divergent semantic integration (DSI). The DSI involves measuring the creativity of content by calculating the average distance between all word pairs in the content. The distance between two words can be determined using the cosine distance between their respective word embeddings. A larger average distance in the content indicates the presence of diverse concepts and ideas, thus indicating higher creativity. Conversely, a smaller average distance suggests a narrower range of concepts and ideas, indicating lower creativity. Measuring Aggregated Creativity In addition to measuring individual creativity, we will also assess aggregated creativity. This process involves combining multiple essays and treating the pooled essays as a single input to calculate the DSI score. However, the classic DSI approach is not practical for estimating aggregated creativity due to the immense number of computations necessary to execute. Therefore, we will randomly select 34 sentences—the average essay length from the original essays—and then compute the DSI scores with words in those selected sentences. This analysis aims to determine whether increasing the number of human-written essays contributes to the greater increase in aggregated creativity than adding more AI-generated essays. 4) Independent variables: Describe the conditions (for an experimental study) or predictor variables (for a correlational study). The independent variable in this study is whether the text was authored by AI or by a human. We will follow the following procedure to prepare AI- and Human-generated admission essays to compare. 1. Human-generated admission essays We will randomly select 100 college admission essays from a pool of 49,804 essays by applicants to a private university on the East Coast. For simplicity of the comparison, we will only choose the essays based on one specific prompt. In order to ensure a fair comparison, we will specifically select essays from the years 2018 to 2022, which is prior to the official launch of the ChatGPT service. Essays previously analyzed in our pilot study will not be included. 2. AI-generated admission essays We will employ the ChatGPT-4 models to generate AI-generated admission essays based on the following prompt: "As a college applicant, please generate an admission essay of approximately 650 words in response to the following prompt: {essay prompt}." We will prompt ChatGPT models to produce essays that are 650 words in length, which is the average length of such essays. As there is no title in human-authored essays, we will omit essay titles in essays authored by ChatGPT-4. As the ChatGPT-4 model is currently only available to us through the chatting interface, we manually generated 100 essays using the ChatGPT-4 between Jun 24th and 31st, 2023. 5) Dependent variables: Describe the key dependent variable(s) specifying how they will be measured. Individual Level Comparison We aim to compare the creativity levels of AI-generated and human-generated college admission essays at an individual level. To compute the DSI of each essay, we will follow four steps. First, we will tokenize all the words in the essays using the BERT tokenizer. Next, we will obtain the embeddings (especially, layer 6 and layer 7 of the BERT model) of each token. Then, we will calculate the distances between all possible pairs of embedding. Finally, we will average the distances to measure the overall semantic distance. Aggregated Level Comparison As mentioned earlier, in addition to comparing the creativity levels of individual essays, we will also examine creativity aggregated levels. To compute the DSI at aggregated level, we will follow five steps. First, we will assemble a certain number of essays together, treating them as a single essay. Next, we will randomly select 34 sentences from the pooled essays to ensure that the length of the pooled essay is similar to that of the individual essays. This procedure will enable us to calculate the DSI even if the length of a pooled essay is too long to compute with the available computational resources. We will repeat this process 5000 times to obtain a distribution of aggregated DSIs. 6) Analyses: Describe what analyses (e.g., t-test, repeated-measures ANOVA) you will use to test your main hypotheses. For H1, we will conduct a one-way ANOVA. For H2, we will compare the maximum number of essays that can contribute to increasing aggregated creativity for both human-generated essays and AI-generated essays. To examine this we will follow the steps below: A. We will derive aggregate creativity scores from bootstrapped samples for each number of essays from 2 to 100. We will employ the bootstrapping method, using a sample size of 5000 with replacement (e.g., 5000 samples of 2 essays each, for both AI-generated and human-authored essays). Then, we will compute the DSI with the randomly selected 34 sentences for each pooled essay sample. B. We will apply the Multivariate Adaptive Regression Splines (MARS) to estimate the breakpoints for human-generated and AI-generated essays in our study. This statistical technique excels in modeling nonlinear relationships and interactions, which makes it a suitable choice for identifying the turning point or the maximum number of essays that contribute to increasing aggregated creativity. 7) Mediator variables: Describe any variables you expect to mediate the relationship between your IV’s and DV. Specify how they will be measured. NA 8) Moderator variables: Describe any variables you expect to moderate the relationship between your IV’s and DV. Specify how they will be measured. NA 9) More analyses. Are there any secondary analyses you plan to conduct? (e.g., order or gender effects) 10) Sample. Where and from whom will data be collected? How will you decide when to stop collecting data (e.g., target sample size based on power analysis, set amount of time)? If you plan to look at the data using sequential analysis, describe that here. We will compare 200 essays (100 essays per group). Though power analyses indicated that we need much fewer participants to detect a significant difference between groups for H1, we wanted to select a large enough number of essays so that we can test H2. H1: We have obtained an effect size of 1.79 (Cohen’s d) when comparing the creativity of individual essays written by humans and those generated by GPT-4. Preliminary power analysis for a two-sample t-test, which assumes a two-sided alternative hypothesis, suggests that approximately 9 participants per group would be needed to detect an effect size of 1.79 with 95% power at a significance level of 0.05. Therefore, our proposed sample size significantly exceeds the minimum requirement indicated by the power analysis. H2: Unlike traditional statistical methods, the MARS method is a non-parametric method that does not require specifying a sample size in advance. That said, we chose to compare 100 essays. 11) Exclusion criteria: Who will be excluded (e.g., outliers, participants who fail manipulation check, demographic exclusions)? Will they be replaced by other participants? If there are any errors, such as essays being too short, too long, or illegible, in the selected essays, we will replace them with other essays from the essay pool. Pre-registration written by (initials): KM Pre-registration reviewed by (initials): KK