Unraveling Downstream Gender Bias from Large Language Models: A Study on AI Educational Writing Assistance
Authors: Thiemo Wambsganss
Xiaotian Su
Vinitra Swamy
Seyed Parsa Neshaei
Roman Rietsche
Tanja Käser
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
Gender Bias
Writing Support
Human-AI Collaboration

Abstract: Large Language Models (LLMs) are increasingly utilized in educational tasks such as providing writing suggestions to students. Despite their potential, LLMs are known to harbor inherent biases which may negatively impact learners. Previous studies have investigated bias in models and data representations separately, neglecting the potential impact of LLM bias on human writing. In this paper, we investigate how bias transfers through an AI writing support pipeline. We conduct a large-scale user study with 231 students writing business case peer reviews in German. Students are divided into five groups with different levels of writing support: one in-classroom group with recommender system feature-based suggestions and four groups recruited from Prolific -- a control group with no assistance, two groups with suggestions from fine-tuned GPT-2 and GPT-3 models, and one group with suggestions from pre-trained GPT-3.5. Using GenBit gender bias analysis and Word Embedding Association Tests (WEAT), we evaluate the gender bias at various stages of the pipeline: in reviews written by students, in suggestions generated by the models, and in model embeddings directly. Our results demonstrate that there is no significant difference in gender bias between the resulting peer reviews of groups with and without LLM suggestions. Our research is therefore optimistic about the use of AI writing support in the classroom, showcasing a context where bias in LLMs does not transfer to students’ responses.

Unraveling Downstream Gender Bias from Large Language Models: A
Study on AI Educational Writing Assistance
Thiemo Wambsganss" Xiaotian Su' 2
Vinitra Swamy Seyed Parsa Neshaei Roman Rietsche 1 Tanja Käser
Bern University of Applied Sciences (thiemo. wanbsganss, roman. coman.rietscio rietsche
2 EPFL, Lausanne, Switzerland
(xiaotian.su) vinitra. swamy, neyednneshaei, tanja.kaesel jeefflr
Abstract
practitioners have been developing a plethora of ed- ucational writing tools based on Natural Language Processing (NLP), for example for writing sugges- tions, (Wambsganss et al., 2022a; Lauscher et al., 2018) conversational interaction (Ruan et al., 2019; Schmitt et al., 2021), or to support writing skills and learning at scale. Nevertheless, the increasing use of LLMs for personalized support (especially in education) bears issues of critical concern, Re- search has found that these models can carry and propagate significant bias (Bolukbasi et al., 2016; Sun et al., 2019). LLMs, at any stage in their devel- opment pipeline, can harbor and propagate biases (e.g., gender, racial, or conceptual) and thus reflect the data they are trained on (Baker and Hawn, 2021; Hutchinson and Mitchell, 2019; Sun et al., 2019). Such biases, specifically gender bias, in LLMs, can reinforce harmful stereotypes in automated writing support applications, for example in automated es- say scoring (Östling et al., 2013; Yannakoudakis et al., 2011) 1) or individual writing support (Wamb- sganss et al.. 2022b), and thus can inadvertently influence students' writing styles and perspectives (Su et al., 2023; Lee et al., 2022a).
Large Language Models (LLMs) are increas- ingly utilized in educational tasks such pro- viding writing suggestions to students. De- spite their potential, LLMS are known to harbor inherent biases which may negatively impact learners. Previous studies have investigated bias in models and data representations sepa- rately, neglecting the potential impact of LLM bias on human writing. In this paper, we inves- ligate how hias transfers through AT writing support pipeline. We conduct large-scale user study with 231 students writing business case peer reviews in German. Students divided into five groups with different levels of writ- ing support: one classroom group with feature- based suggestions and four groups recruited from Prolific - a control group with no assis- tance, two groups with suggestions from fine- tuned GPT-2 and GPT-3 models, and one group with suggestions from pre-trained GPT-3.5. Us- ing GenBit gender bias analysis, Word Embed- ding Association Tests (WEAT), and Sentence Embedding Association Test (SEAT) we eval- uate the gender bias at various stages of the pipeline: in model embeddings, in suggestions generated by the models, and reviews written by students. Our results demonstrate that there is no significant difference in gender bias be- tween the resulting peer reviews of groups with and without LLM suggestions. Our research is therefore optimistic about use of writing support in the classroom, showcasing con- text where bias in LLMs does not transfer to students' responses
Previous research has explored the existence of bias in different language models and their rep- resentations, focusing on the English language (Baker and Hawn, 2021). However, a growing body of literature advocates for the necessity of conduct- ing comprehensive bias analyses, i.c. analyzing the impact of language models when embedded human educational tasks (Lee et al., 2022b; Baker and Hawn, 2021; Blodgett et al., 2020). Especially for gender stereotypes, recent studies such as An- dersson et al. (2021) and Cheng et al. (2022) have demonstrated the effects of such stereotypes on CV screening and child welfare programs, respec- tively. Nevertheless, detailed examinations of the effects of M.LM-based writing suggestions on stu- dents and their use of gender stereotypes have been rather sparsely investigated, especially in other lan-
1 Introduction
Large Language Models (LLMs) have proven use- ful for improving the adaptivity and individualiza- tion of educational technology (Jones and Stein- hardt, 2022; Xu et al., 2021). Researchers and
"These authors contributed equally to this work 'Our code and data available at: https: httb...cotuuuco///o epfl-m1 4ed44//r///aeell lonngllmmae 11m-bias

---

guages than English (Baker and Hawn, 2021; ; Lee
(G0), Furthermore. none of the GenBit gender bias, WEAT tests and SEAT tests reveal biases in the provided suggestions from any of the LLMs, despite that the analysis GPT-2 embeddings de- tects significant gender bias for the GPT-2 model. Our results therefore demonstrate that LLM-ban based writing support can be positively used for specific educational scenarios without bias.
and Kizilcee, 2020). Given the expanding use of these models in educational settings and for writ- ing assistance in general, (e.g., (Lee et al., 2022a; Chang et al., 2023), addressing this literature gap is of utmost importance. Understanding the nuanced ways in which biases in LLMs can seep into edu- cational tasks can help rescarchers create safer and more effective learning environments that promote equitable outcomes.
Related Work
In this paper, we analyze how bias transfers through an AI writing support pipeline and we in- vestigate whether bias in LLMs translates into bias in human writing. We use an educational context, namely German peer reviews collected from 231 students divided into five groups: one group of 52 students in University classroom setting re- ceiving feedback from a traditional feature-based recommender system (GO) and four groups (179 students in total) recruited through Prolific. Groups G1-G4 include a control group receiving no writing support (GI), two groups with suggestions from GPT-2 (G2) and GPT-3 models (G3) fine-tuned on an extended version of the non- non-biased corpus of Wambsganss et al. (2022b) containing 11,925 925 peer reviews in German, and one group with suggestions from pre-trained GPT-3.5 (G4).
2.1 Bias in Educational Writings
Research has analyzed bias in educational tech- nology since around the 1960s and many parts of today's research on algorithmic bias and fairness have been anticipated (Baker and Hawn, 2021). To effectively probe bias, it is imperative to establish our viewpoint, as the term "bias" is multifaceted and defined differently across various research works (see literature reviews such as Hutchinson and Mitchell (2019); Baker and Hawn (2021)). In our study, we adopt the view of algorithmic bias as "situations where model performance is sub- stantially better or worse across mutually exclusive groups" (Baker and Hawn, 2021, p. LLMs, such as GPT-2, GPT-3, or BERT, have been increas- ingly utilized for educational writing assistance (Mirowski et al., 2023; Gero et al., 2022). These models, trained on extensive and diverse data, have proven instrumental in predicting subsequent text, thereby producing coherent responses (Lee et al., 2022a,b). Research has studied the risk that they might reflect by investigating the biases inherent in the training data and the models (Kurpicz-Briki. 2020). We aim to scrutinize the impact of bias from LLMs on student writings, thereby focusing on the lower end of the NLP pipeline, investigating the impact of writing suggestions on educational downstream tasks as Lee et al. (2022b) suggested.
We apply the GenBit gender bias test Bordia and Bowman (2019), the German adaptation Kurpicz- Briki (2020) of the Word Embedding Association Tests (WEAT) (Caliskan et al., 2017), and Sentence Embedding Association Test (SEAT) (May et al., 2019) translated to German to the collected peer reviews well as the suggestions provided by the different LLMs and the embeddings of fine- tuned GPT-2 model. With our analyses, we aim to answer the following two research questions:
1. In real-world peer review writing exercise with Al writing support, does LLM bias trans- fer student writing (RQ1)?
2.2 NLP Bias Analysis
Research within the field of computational linguis- tics and NLP has progressively delved into bias present in language systems. This includes investi- gations into bias in areas such as embedding spaces (Caliskan et al. (2017); Bolukbasi et al. (2016)), language modeling (Lu et al., 2018), co-reference resolution (Rudinger et al., 2017), machine transla- tion (Stanovsky et al., 2019), and sentiment anal- ysis (Kiritchenko and Mohammad, 2018). Sun et al. (2019) have investigated strategies to mitigate gender bias across various NLP tasks. variety
2. How does bias transfer across the different stages (i.e., model embeddings, model sugges- tions, student output) of the AI Al writing support pipeline (RQ2)?
Our results reveal promising trend: groups re- ceiving suggestions from LLMs (G2-G4) exhibit no significant measurable difference in gender bias in their writing compared to the control group without writing support (GI) and the in-classroom students receiving recommender- based feedback

---

of methods have been proposed for the detection of gender bias in text. These range from multi- dimensional classifications of gender bias (Dinan et al., 2020) to the exploration of the frequency of gender bias metrics using word embeddings (Valentini et al., 2022). Some studies have even analyzed human gender stereotypes via word as- sociation tests (Du et al., 2019). Instruments such the Word Embedding Association Test (WEAT) (Caliskan et al., 2017) are commonly utilized for bias identification, enabling the quantification of biases within word embeddings by assessing the strength of correlations between target words and attribute words (Du et al., 2019). WEAT is com- posed of different tests that aim to reveal racial, conceptual, and gender bias. WEAT tests 6, 7, and 8 have been also used to investigate gender bias in German texts (Kurpicz-Briki, 2020). Furthermore, Bordia and Bowman (2019) proposed test de- tect gender bias in word-level language models and suggested bias score that has been widely used in the NL. community (e.g., Sengupta et al. (2021)) )). For sentence-leve bias analysis, there is the Sen- tence Embedding Association Test (SEAT) (May et al., 2019) which extends WEAT to measure bias in sentence encoders.
to contribute towards shaping future where NLP rescarchers and practitioners are aware of biases of downstream educational models and hence strive to minimize the potential harm by those models when applied in sensitive contexts like education, potentially involving sensitive user groups (such as under-aged students).
3 Methodology
To investigate the impact of LLMs on gender biases in human writing within an educational context, we performed bias analysis in three steps (see Figure 1). In a first step, trained Prature-based recom- mender system and fine-tuned GPT-2 and GPT-3 models on corpus of 11, 925 student-writtes busi- ness peer reviews in German to be able to provide automated suggestions in a peer review exercise. In a second step, we conducted a large-scale user study with 231 students in a classroom and an on- line context, providing students with different lev- els of writing support (feature-based recommender, no support, suggestions by fine-tuned GPT-2 GPT-3, suggestions by pre-trained GPT-3.5) in a peer review writing exercise in German. In the third and final step, we used the GenBit Gender Bias Score (Bordia and Bowman, 2019), WEAT, and SEAT tests to analyze the gender bias of the resulting reviews, the model suggestions, as well as the GPT-2 embeddings.
2.3 Studies on Bias in Writing Assistance
Nevertheless, studies that investigate biases in downstream applications and particularly the in- fluence of LLMS on human writing are rather rare. While Lee al. (2022b) have proposed a frame- work for investigating the impact of LLMs on hu- man writing, they have only motivated and not in- vestigated the impact of toxicity and bias on human texts after receiving writing assistance. Further- more, studies in the educational domain involving real students especially outside of North America remain limited (Baker and Hawn, 2021: Sun et al., 2019). Our work centers on the impact of writ- ing suggestions provided by LLMS on students' gender stereotypes. Our goal extends past work on revealing bias in the educational NLP pipeline (Wambsganss et al., 2022b) and on human eval- uation (Lee et al., 2022b) by shedding light on the impact of potentially biased LLMs on students. We do so by investigating the case of student peer reviews in the German language since this domain-indeppnddnt and increasingly embedded educational context fostering writing competencies in large-scale learning scenarios. With this, we aim
3.1 Model Development
In the FUSM group (GO), students received auto- mated advice based on the Feature Utility Satura- tion Model (FUSM). This model is trained on a corpus of 9, 000 student-written peer reviews in German. The recommender system predicts the utility of specific types of feedback in improving the review quality (Bauman et al., 2020). A person- alized subset of 3 out of 15 features, such as adding suggestions, decreasing sentence length, and writ- ing more directive words, is provided upon clicking the get advice button. The features have been de- rived based on existing literature on peer feedback and a dictionary with keywords was developed to operationalize the features. We followed Bauman et al. (2020) for the implementation of our model. To provide students with adaptive writing sug- gestions, we fine-tuned GPT-2 and GPT-3 models (used by groups G2 and G3) and embedded GPT- 3.5 in the peer review tool (used by group G4). The two models for G2 and G3 were fine-tuned on

---

Model Development
Study Design
Gender Bias Analysis
Participants
Procedure
Window 10 60
Classroom Sludy
9,000 Corpus
Recommender syetem suggestions (G0)
Introduction to a peer's
GenBit Gender Bias Scores
context
o' Fealure- model
Online Study (Prolific
Infinite context
No CG (G1)
Peer review writing in German
f1,925
Fine-tuned GPT-2 generated suggestions (02)
WEAT Tests (Gander)
Test 6: male names
GPT-2
Corpus
Test
Sludent-vriten business peer
Fine-tuned GPT-3 generated suggestions (G3)
reviens German
GPT-3
SEAT Tests (Gender)
Test 8: science
GPT-3.5 generated suggestions (G4)
Post-eurvey
Data Proceesting
Modeling
Data Collecioon Data Proceesing
Analyeis
Figure I: Overview of pipeline. We first prepared machine learning models as well as LLMs provide writing suggestions for learners, see Section 3.1 for more details. We then conducted user study with a peer review writing exercise. Section 3.2 presents the whole study design including the procedure and all five groups finished the exercise with different writing assistance tools. We provided details on data collection as well as data processing. Finally, we analyzed gender bias using the GenBit Bias Score, WEAT and SEAT tests (see Section 3.3).
extended version of the non-biased corpus of peer reviews in German reported by Wambsganss al. (2022b). The data was collected over four years and includes 11,925 reviews from 610 unique reviewers and 607 reviewees. Students wrote ap- proximately nine peer reviews per course with an average length of 220 words. This extensive corpus served as solid foundation for fine-tuning models and preparing them to provide writing suggestions to students. We started data processing by expand- ing abbreviations, removing HTML tags, irrelevant information like PDF file names and specifie infor- mation like URLs, keywords (revealing the identity of students), and questions asked to write reviews which some students copied to their review text. Then, we shuffled and divided the cleaned data into train and test datasets with proportions of 0.8 and 0.2 for fine-tuning and evaluating the language models.
additional statistical analysis confirms that there are significant differences between groups and that the randomization has worked correctly. For the age attribute, a Shapiro-Wii Wilk test indicates that the data is not normally distributed and we have there- fore conducted a non-parametric Kruskal-Wail test, which confirms that there are no significant differences between groups GI-G4 in terms of age (H 2.24, p .52). Furthermore, Chi-Square test confirms that there also no significant differ- ences between the four Prolific groups in terms of gender (X2 0.0149, p= .99) or education level (X2 = 6.65, p == .35). Each group was provided with writing suggestions from a unique model: GI received suggestions from a Seature-based recom- mender system, for example, on text length or sen- timents, G2 received suggestions from fine-tuned GPT-2 model, G3 from fine-tuned GPT-3 model. and G4 from pre-trained GPT-3.5. Table provides an overview of each group and the demographics of the participants.
3.2 Study Design
Participants. Participants of the user study were 231 students who were split into five distinct groups (GO- G4), controlling for the sensitive variables of education level, language, age, and gender. While there might be (small) observable differences the sample means or medians for sensitive variables,
Procedure. In order control for the content pect (as the content of the provided business model could inadvertently have an influence on the ex- pected bias), we present each participant with ex- actly the same predefined business model.

---

Context
Group
#
Gender
Age
Highest Education
Male
Female
Mean
Std.
Below BSc
BSc and above
Classroom
FUSM (GO)
52
62.0%
38.0%
25.7
1.9
0.0%
100%
None (G1)
40
50.0%
50.0%
30.0
8.3
30.0%
70.0%
GPT-2 (G2)
50
50.0%
50.0%
28.3
8.5
22.0%
78.0%
Online
GPT-3 (G3)
44
50.0%
50.0%
29.9
12.6
34.1%
65.9%
GPT-3.5 (G4)
45
54.3%
45.7%
30.9
12.3
39.1%
60.9%
Table Overview of the data sample and participant demographics of our user study across five groups.
Students in the FUSM group (GO) received writ- ing suggestions through a dashboard next to the text input in the form of syntactical and semantic advice. We collected data in lecture at Western European university where 52 students were writ- ing up three reviews on a peer's business model and they went through three peer feedback rounds.
case. Because of the linguistic characteristic of the German language and its grammatical genders, special words like Ihr, Ihrem, Ihren, Ihrer, Ihres, and Sie have both gendered meanings ("she") and non-gendered meaning (e.g. "you"). We filtered them by checking their position the sentence. If they related the polite form (Höffichheitsform)) we removed them to not inflate the bias score in the later calculation. Otherwise, we kept them. After- ward, we did human evaluation. Three German researchers checked the outputs manually to con- firm the quality of the processed data.
The data of the three LLMs groups (G2-G4), well as the control group (GI), was collected through the online platform Prolific? in order to not cause potential harm to real-world students. The task in the Prolific study involved the partic- ipants writing review on a pre-defined business model. Specifically, participants were asked to elaborate on strengths, weaknesses, and sugges- tions for improvement of the given business model. We instructed people not to use search engines and spend minimum of 15 minutes on the task. A countdown indicated the remaining time. Students received different forms of writing support on that task. Specifically, we followed the interface of the human-centric educational tool of Su al. (2023) (see Figure 4 in the appendix) to ensure that stu- dents received beneficial writing aid with a satis- factory user experience. We presented users with next-sentence predictions, providing three sugges- tions at each point in time. Students could use the Tab key accept suggestion, the Ese key to re- ject suggestion, and the Up or Down arrow keys toggle through different suggestions. During the writing process, we collected the final writings, suggestions, as well as keystrokes of participants.
3.3 Gender Bias Analysis
To investigate the bias in the resulting peer reviews, the model suggestions, and the model embedding. utilized three different gender bias tests.
3.3.1 GenBit Gender Bias Score
The GenBit Gender Bias Score was introduced by Bordia and Bowman (2019) and is included in the Microsoft Responsible AI Toolkit (Sengupta et al., 2021). The idea of the method is to identify gen- der bias by measuring the association between pre- defined gendered words and other words in the corpus via co-occurrence statistics.
Bias score definition. To compute the bias score, we first estimate the probability of a word occurring in the context of gendered words within text cor- pus. The probability P(wlg) indicates how likely for specific word, denoted w, appear near (within a context window k) gendered words, denoted as g:
We cleaned the collected data by removing HTML tags, emojis, punctuation, abbreviations, digits, and stop words (excluding words in the gen- der lists 3). The text was transformed to lower-
county (w, (w.g)
P(wlg) Σ. count, (w..g)
www.prolific.co
where to is any word in the corpus, excluding stop words and gendered words, counta(w, g) repre- sents the count occurrences where the gendered
unraveling www.aveelng-l----- 1aag--1lmbiass http:////m//iaattee/e////aaaaaaa 20list

---

word 9 appears in the context window k of the tar- get word w and Σ; county(u,, (u,,g) caleulates the total count of occurrences where any word w2 ap- pears in the context window k of gendered words g. Finally, 9 is the set of gendered words that belong to either of the two categories: male or female. In other words, we count how many times the word and at least one gendered word from the set g appear within certain distance of each other. In terms of defining gender pairs, we adopted the same set of gender term pairs as Sengupta et al. (2021). Additionally, given the linguistic character- istic of the German language, we added gendered pronouns to better capture the grammatical gender. For pronouns with ambiguous meanings, we did filtering process in the data processing stage under the instruction of three German researchers.
nine WEAT tests and their corresponding targets and attributes, which we translated into German. Our analysis focuses on the gender bias dimension of WEAT using tests 6, 7, and 8.
To quantitatively compare across WEAT analy- ses, we use the effect size as proposed by (Caliskan et al., 2017). Effect size is a normalized measure of the distance between the two distributions of associations and targets, calculated as follows:
1 [x
Exex 5(2, B) M8
1 Eyer s(y, A, B)
SWEXUY (s(w: A, B))
where X and Y are two sets of target words of equal size, A, B are two sets of attribute words, s(w, A, B) measures the association of embed- dings of the target word w with the attribute words, and S denotes the standard deviation. We conduct
The bias score of specific word is then de- fined as:
WEAT analyses at granularities as proposed by (Wambsganss al., 2022b); in the raw text corpora (through co-occurrence and GloVE models) and in the fine-tuned language model.
P(w[m)) bias(w)= log( f(wlf))
This bias score measured per word and review. A positive bias score implies that a word co-occurs more often with male words than female words.
3.3.3 Sentence Embedding Association Test In addition to word-level metrics like GenBit and WEAT, we have sentence-level metrics, the SEAT test, for more comprehensive analysis. This test was defined in May et al. (2019) and implemented in Meade al. (2022). We used the implementa- tion from Fairpy (Viswanath and Zhang, 2023), an open source Toolkit for measuring and mitigating biases in large pre-trained language models. By applying WEAT to the vector representation of a sentence, SEAT compares sets of sentences instead of sets of words. In the ideal case, the embedding representation of each word in the vocabulary is expected to be equidistant from the two attribute classes. Any deviation suggests bias in one direc- tion. The greater the deviation, the greater the bias (Viswanath and Zhang, 2023). Same as WEAT tests, our analysis focuses on the gender bias di- mension of SEAT using tests 6, 6b, 7, 7b, 8, 8b. However, to the best of our knowledge, there is no available German version of SEAT tests. We first translated the templates from English to German with the translation software, then the translations were revised by two native German speakers. To facilitate future work, we publicly provided the translated files
Windows size and context window. For con- text size there are k/2 words before and I/2 words after the target word w for which the bias score is being measured. Qualitatively, a smaller context window size has more focused information about the target word. On the other hand, larger window size captures topicality (Levy and Goldberg, 2014). According to Bordia and Bowman (2019), the opti- mal window size for fixed context is ls 20, which assigns an equal weight of 5% to the ten words be- fore and the ten words after the target word. For an infinite context, weights diminish exponentially based on the distance between the target word w and the gendered word g. This method emphasizes the fact that the nearest word has more information about the target word.
3.3.2 Word Embedding Association Test
To assess bias along the NLP pipeline suggested by Hovy and Prabhumoye (2021), we rely on the Word Embedding Association Test (WEAT) proposed by Caliskan al. (2017). WEAT calculates the se- mantic similarity between two sets of target words (e.g., male vs. female names) and two sets of at- tribute words using word embeddings (c.g., carcer vs. family). Table 6 in the appendix indicates the
'https://gitthhi wttptehhhh.coo//ee--eeeeeee
inraveling-111 htt.m//iss//eee/aaaa///////////e http://aan/n/////aaattttteeeeeee 20de

---

4 Results
GenBit Gender Bias Scores of luman-WWitten Reviews
To evaluate whether bias transfers from AI assis- tants to students for a real-world writing support scenario (RQ1) and to investigate where in the pipeline the bias persists (RQ2), we applied the GenBit Bias score as well as the WEAT tests to the five different subsets from user study: four from the Prolific user study (control group GI, GPT-2, GPT-3, and GPT 3.5 assisted reviews G2-G4) and one from the classroom study with recommender system suggestions (GO).
D00
42
Franknod
Prevknal
0
32
4.1 RQ1- Does Bias Transfer?
Group
Figure 2: GenBit gender bias score of human-written reviews for fixed context with window size of 20.
Group
#
GenBit Bias Score
FUSM (GO)
310
-0.024 + 0.275
Control (G1)
40
0.065 + 0.487
(Shapiro and Wilk, 1965) then showed that our data was not normally distributed. Additionally, Lev- ene test (Levene, 1960) confirmed heteroscedas- ticity within the data. Therefore, we chose the non-parametric Mann-Whitney U test (MWU) analyze the difference in mean bias scores between any pairing of two groups and the Kruskal- Wallis test (Kruskal and Wallis, 1952) for the difference among all four groups.
GPT-2 (G2)
50
-0.099 + 0.486
GPT-3 (G3)
44
0.115 + 0.570
GPT-3.5 (G4)
45
-0.058 = 0.592
Table 2: Total number of reviews from each group (if) and statistical summary of the GenBit bias score for all groups. The large standard deviations indicate the variability of the range of bias scores. Traditional MI. feedback (GO) has the lowest variability, while the LLM feedback (G2-G4) and the control group with no feed- back (G1) have similar ranges of variability.
Table 3 presents the results of statistical tests analyzing the mean bias scores of reviews written with and without suggestions from LLMs. We did not find any statistically significant difference between the bias scores of the four groups.
To answer first research question, we com- puted the GenBit Bias Score for the peer reviews written by the students. The goal was iden- tify which biases the models transfer to humans through collaboration. We experimented with both a fixed context for windows sizes ranging from 10 to 60 as well as an infinite context, Since there was no significant difference between the results from the fixed context and infinite context, and the optimal window size was determined to be 20 by previous work Bordia and Bowman (2019), in this section, we only present results for fixed context with a window size of 20. The resulting GenBit bias scores are illustrated in Figure? 2. The total number of reviews from each group and statistical summary of bias scores for all groups are presented in Table 2. We observe that all five groups exhibit bias scores close to O. Additional results pro- vided in the appendix in Section B.
p-value
p-value MWU Test
p-value MWU Test
MWU Test
Group
GPT-2 (G2)
GPT-3
GPT-3.5
(G3)
(G4)
Control (GI)
0.170
0.619
0.551
GPT-2 (G2)
0.075
0.635
GPT-3 (G3)
0.269
Table 3: Mann- Mhan-Whitney U (MWU) test of bias scores for reviews from multiple groups Asterisks indicate sta- tistical significance (#**; p<.001; *s; p<.01; #; p<.05). There is no statistically significant difference between bias scores the four groups.
4.2 RQ2 - Where is Bias Present?
To answer our second research question and inves- tigate where bias is present in the pipeline, we used the GenBit Gender Score, WEAT tests, and SEAT tests to analyze the LLMs' suggestions and (where available) the model's 's raw embeddings. This in-
To analyze differences between the bias scores of the four Prolific groups (G1 (GI-G4), we first ag- gregated the mean bias scores each review from these groups. The results a Shapiro-Wii Wilk test

---

WEAT Test
Test
WEAT Test
WEAT Difference Stores over Attributes For each Target
[Caneey)
[M
0.1
[Neutral) 0.0
(Neutral) 0.
Neusaaj
(Famli)
(Fomato)
[Female]
Science
Art
Figure 3: Comparing WEAT bias test scores for the fine-tuned GPT-2 model used the human evaluation study.
depth analysis allowed us to dig deeper into the potential roots of bias within the suggestions made by the models and further understand how these may have influenced the students' writings.
score of 1.57 (with effect scores ranging from -2 to 2), indicating that the career attribute is much closer Male than Female names.
WEAT Test 7 focuses on the target disciplines of Math and Art, while examining Male and Fe- male attribute lists. The difference scores in Figure (middle) showed minimal disparity, show- ing slight bias in favor of the male attribute for both Math and Art targets. The five most similar attribute words to the Math target list, translated from German to English, (young, son, brother, mas- culine, and sister) depicted more extreme picture with 4 out of 5 mnie-triented words, aligned with the 5 most similar attributes to Art (young, brother, son, man, and daughter). The effect size was 0.27, showing that Math is slightly more aligned the Male attribute than the Art target.
Post-Hoc Analysis of Suggestions. We mea- sured the bias in the raw text corpora of model suggestions from fine-tuned GPT-2, GPT-3, and pre-trained GPT 3.5 with three methods: Gen Bit bias scores, WEAT co-occurrence tests, and WEAT GloVE embedding tests. In each of these tests, we did not identify any significant bias in the suggestions. We pre-filtered suggestions that had more than 10 words to measure bias. On aver- age, there were 13 15 words in each suggestion across groups. For a fixed context with a window size of 10, the bias scores for fine-tuned GPT-2 and GPT-3 were 0.000092 and respectively, and 0.000062 for GPT-3.5. In the GloVE embedding model trained on each text subset, we identified one instance of bias in GPT-2 suggestions in the class- room study for WEAT test 7 (examining female vs. male targets for math vs. art attributes), but this was considered insignificant. Overall, through the WEAT tests of co-occurence and embedding models, we determined that the raw text corpora across the study were unbiased.
In WEAT Test 8, examined the relationship between Science and Art targets and Male and Fe- male attribute lists (see Figure 3 (right)). Again, we identified similarities in the top five attribute words with a strong male bias in the most related words. For the Science target, (uncle, son, father, daughter, and brother) were the five most similar attributes (four out of five male-oriented). For the Art target, the most similar attributes were (uncle, brother, son, father, and daughter) (again four out of five male-orientedd The effect size for this test was 0.556, indicating that Art is more related male attributes than Science.
Post-Hoc Analysis of Model Embeddings. To dive deeper into the bias behavior of a fine-tuned model used for writing assistance, we examined the gender bias of the fine-tuned GPT-2 model em- beddings in both word-level and sentence-level. As shown in Figure (left), we identified notable bias in the fine-tuned model that was not present in the raw text corpora. the WEAT Test 6, we ex- amined attributes Carecr vs. Family in relation to Male vs. Female names. We obtained an effect
SEAT test results are summarized in Table 4, across tests 6, 6b, 7, 7b, 8, and 8b, which corre- spond the analogous gender tests of WEAT, there no significant difference found between the em- beddings of the target sentences and attribute sen- tences (i.e. in SEAT 6, male and female names are both equally similar to career and family words),

---

#
Tragets
Attributes
Effect size
6
Male vs. Female Names
Career vs. Family
0.021
6b
Male Female Terms
Career vs. Family
-0.074
7
Math vs. Arts
Male vs. Female Terms
-0.705
7b
Math vs. Arts
Male vs. Female Names
-0.209
8
Science vs. Arts
Male vs. Female Terms
-0.069
8b
Science vs. Arts
Male vs. Female Names 0.078
Table 4: SEAT gender bias analysis for the fine-tuned GPT-2 model used in the human evaluation study.
computed using p-values and a hypothesis test as per the standard implementation. Each test also has an effect size, ranging from -2 to +2, with representing a completely neutral effect between both target words and attribute words. Most effect sizes are within 0.1, so there are only minimal bias effects. The comparatively largest effect (SEAT test 7) is -0.705 from comparing Math and Arts in association with Male and Female terms, but this was still found to have insignificant bias. These results are in line with the WEAT analysis on the word level, which also found no significant gender bias in the model embedddings.
(model embeddings, model suggestions, student output), LLMS do not inadvertently foster and per- petuate gender bias. One possible explanation for these results is that the biases present in the original training data and embeddings of the LLMs were not transferred to the model suggestions, as indicated by the lack of bias in model suggestions. Unbiased suggestions led to positive learning outcomes for students without inadvertent bias transfer.
Our post-hoc deep dive into the bias dimensions of the suggestions and the GPT-2 embeddings re- vealed the inherited bias in the fine-tuned models. This analysis provided valuable insight into how biases become ingrained in LLMs and also identified where in the pipeline the bias stops. It suggests that even when models are fine-tuned with the intention of reducing bias, their original train- ing on vast amounts of data from the internet can leave an indelible imprint of bias. An important takeaway from our study is that the applied domain of education, although it is often sensitive context, is moving towards integration with LLM assistants. Before we can be fully confident of model impacts on sensitive young minds for downstream tasks, we need to strive for not only more sophisticated bias detection and mitigation techniques but also more transparency in how these models are trained and fine-tuned. Also, future research is needed to better investigate the impact of potentially biased LLMs in different educational settings in addition to writ- ing, for example, language learning (Xiao et al., 2023), STEM education (Lee and Perret, 2022), and legal education (Weber et al., 2023).
Interestingly, demonstrated by our analyses, the gender biases revealed in GPT-2 embeddings did not translate into gender biases in suggestions.
5 Discussion and Conclusion
LLMs are increasingly used in educational settings, despite that they harbor inherent biases which may have negative impact learners. Our work ana- lyzed how bias transfers through Al writing sup- port pipeline an educational context and whether the bias in LLMs translates to bias in students" writings. To do so, we conducted a large-scale user study, providing students with different levels of writing support in peer review writing exercise.
Our analysis of data collected from in-classroom and Prolific participants yielded positive results that provide optimism for the field of NLP and its application within educational contexts. The most notable finding was that students who received writ- ing suggestions from LLMs exhibited the same degree of gender bias in their written text stu- dents who received no suggestions. The group receiving suggestions from a recommender system with human interpretable features also showeased similar amount of bias in student responses. Our results seem initially promising, showing that by measuring gender bias through multiple tests (Gen- Bit, WEAT, and SEAT) at each stage of the pipeline
In conclusion, our study contributes towards a more nuanced understanding of how L.LMs inter- act with bias for educational tasks, and produces the indication that although models contain bias, personalized downstream applications might not. We hope that our findings will stimulate further research, contributing to the UN's fourth sustain- ability goal of ensuring quality education for all.

---

Limitations
References
Emma Rachel Andersson, Carolina E. Hagberg, and Sara Hägg. 2021. Gender bias impacts top-merited candidates. Frontiers in Research Metries and Ana- lytics, 6.
While our study provides crucial insights into the impact of LLMs' potential bias on human writing, we acknowledge the need for further research. It would be insightful to extend our study to other gen- erative language models, study populations (differ- ent student levels, different languages), and educa- tional tasks to paint a more comprehensive picture of the complex dynamics at play.
Ryan S Baker and Aaron Hawn. 2021. Algorithmic bias in education. International Journal of Artificiaa Intelligence in Education, pages 1-41.
Konstantin Bauman, Roman Rietsche, and Matthias Söllner. 2020. Supporting Students in the Peer- Review Process by Recommending Features of Writ- ten Feedback that Should be Improved. Statisti- cal Challenges in Electronic Commerce Research (SCECR), Virtual Conference.
The inability to have access to several of the advanced model embeddings, unfortunately, pro- hibited us from conducting a WEAT analysis of GPT-3 and 3.5; we make the informed assumption that bias exists in the embeddings per previous work, but we cannot measure compare it with our GPT 2 findings. Additionally, our educational lask (a business case study regarding ski instruction) neutral and does not easily lend itself to measuring gender bias, compared to other potential settings. It was selected because it is a real-world education example integrated currently in a Western Euro- pean university. If the case study was regarding medicine, for example, the male and female words for the doctor would be different in German, and this could potentially lead to a different bias mea- surement. Other dimensions of bias evaluated in Liang et al. (2023); Lin and Ng (2023) (e.g., racial, cognitive) would also need to be studied before LLMs for writing support can be fully trusted in the classroom.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5454- 5476, Online. Association for Computational Lin- guistics.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to home- maker? debiasing word embeddings. Advances in neural information processing systems. 29.
Shikha Bordia and Samuel R. Bowman. 2019. Identify- ing and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Student Research Workshop. pages 7-15, Minneapolis, Minnesota. Association for Computational Linguistics.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from lan- guage corpora contain human-liie like biases. Science, 356(6334):183 186.
Ethics Statement
We note that this research was conducted by a
Minsuk Chang, John Joon Young Chung, Katy Ilonka Gero, Ting-Hao Kenneth Huang. Dongyeop Kang, Mina Lec, Vipul Raheja, and Thiemo Wambsganss. 2023. The second workshop on intelligent and in- teractive writing assistants. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA '23, New York, NY. USA. Association for Computing Machinery.
mixed team of authors with Western European, Asian, North-American, Middle Eastern, female. and male backgrounds. All student data was anonymized in the use of this study and the study was approved by the human research ethics com- mission of the university. While our results in- dicated that LLM-based AI writing support does not have adverse effects of bias transfer on student writing, our work is not generalizable study over many student populations and many different exer- cise settings. We present case study over several model types and positive finding regarding class- room integration, but we still encourage caution and experimentation before integrating models in the classroom.
Hao-Fei Cheng, Logan Stapleton, Anna Kawakami, Venkatesh Sivaraman, Yanghuidi Cheng, Diana Qing. Adam Perer, Kenneth Holstein, Zhiwei Steven Wu. and Haiyi Zhu. 2022. How child welfare workers reduce racial disparities in algorithmic decisions. In CHI Conference on Human Factors in Computing Systems, pages 1-22.
Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. 2020. Multi- dimensional gender bias classification. In Proceed- ings of the 2020 Conference on Empirical Methods

---

in Natural Language Processing (EMNLP), pages 314-331, Online. Association for Computational Lin- guistics.
Mina Lee, Percy Liang, and Qian Yang. 2022a. Coau- thor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1-19.
Yupei Du, Yuanbin Wu, and Man Lan. 2019. Exploring human gender stereotypes with word association test. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- pages 6133- 6143, Hong Kong, China. Association for Computa- tional Linguisties.
Mina Lec, Megha Srivastava, Amelia Hardy, John Thick- stun, Esin Durmus, Ashwin Paranjape, Ines Gerard- Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong. et al. 2022b. Evaluating human-lange language model in- teraction. arXiv preprint arXiv:2212.09746
guage Processing
Howard Levene. 1960. Robust tests for equality of variances. Contributions to probability and statistics, pages 278-292.
Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for science writing using lan- guage models. In Designing Interactive Systems Con- ference, DIS '22, page 1002-1019, New York, NY, USA. Association for Computing Machinery.
Omer Levy and Yoav Goldberg. 2014. Dependency- based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pages 302- 308.
Dirk Hovy and Shrimai Prabhumoye. 2021. Five sources of bias in natural language processing. Lan- guage and Linguistics Compass, 1243).
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang. Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce 7hang, Christian Alexander Cosgrove, Christo- pher D Manning, Christopher Re, Diana Acosta- Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong. Hongyu Ren, Huaxiu Yao. Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan An- drew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Teard, Tianyi Zhang, Vishrav Chaudhary, William Wang. Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Ko- reeda. 2023. Holistic evaluation of language models. Transactions on Machine Learning Research. Fea- tured Certification, Expert Certification.
Ben Hutchinson and Margaret Mitchell. 2019. 50 years of test (un faimeess::::: Lessons for machine learning. In Proceedings of the Conference on Fairness, Account- ability, and Transparency, FAT '19, page 49-58, New York, NY, USA. Association for Computing Machinery.
Erik Jones and Jacob Steinhardt. 2022. Capturing fail- ures of large language models via human cognitive biases. In Advances in Neural Information Process- ing Systems, volume 35, pages 11785-11799. Curran Associates, Inc.
Svetlana Kiritchenko and Saif Mohammad. 2018. Ex- amining gender and race bias in two hundred senti- ment analysis systems. In Proceedings of the Sev- enth Joint Conference on Lexical and Computational Semantics, pages 43-53, New Orleans, Louisiana. Association for Computational Linguistics.
William H Kruskal and W Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American statistical Association, 47(26)):583- 621.
Ruixi Lin and Hwee Tou Ng. 2023. Mind the bi- ases: Quantifying cognitive biases in language model prompting. In Findings of the Association for Com- putational Linguistics: ACL 2023, pages 5269-5281, Toronto, Canada. Association for Computational Lin- guistics.
Mascha Kurpicz-Bris Briki. 2020. Cultural differences in bias? origin and gender bias in pre-trained german and french word embeddings.
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Aman- charla, and Anupam Datta. 2018 Gender bias in neural natural language processing. In Logic, Lan- guage, and Security.
Anne Lauscher, Goran Glavaš, and Kai Eckert. 2018. ArguminSci: A tool for analyzing argumentation and rhetorical aspects in scientific writing. In Proceed- ings of the 5th Workshop on Argument Mining, pages 22-28, Brussels, Belgium. Association for Computa- tional Linguistics.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622 022-628, Minneapolis, Min- nesota. Association for Computational Linguistics.
Hansol Lee and René F Kizilcec. 2020. Evaluation of fairness trade-offs in predicting student success. arXiv preprint arXhv:2007.00088
Trene Lee and Beatriz Perret. 2022. Preparing high school teachers to integrate ai methods into stem classrooms, Proceedings of the AAAI Conference on Artificial Intelligence, 36(1 12783-3211771
Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. 2022. An empirical survey of the effectiveness of

---

debiasing techniques for pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguistics.
Xinyu Sun, Maoxin Han, and Juan Feng. 2019. Help- fulness of online reviews: Examining review infor- mativeness and classification thresholds by search products and experience products. Decision Support Systems, 124: 14:111: 13099..
Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. 2023. Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals. In Proceedings the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, New York, NY, USA. Association for Computing Machinery.
Francisco Valentini, Germán Rosati, Diego Ferna-- dez Slezak, and Edgar Altszyler. 2022. The undesir- able dependence on frequency of gender bias metries based on word embeddings. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2022, pages 5086-5092, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Robert Östling, André Smolentzov, Bjöm Tyrefors Hin- nerich, and Erik Höglin. 2013. Automated Essay Scoring for Swedish. Proceedings of the Eighth Workshop on Innovative Use of NL.P for Building Educational Applications, pages 42-47.
Hrishikesh Viswanath and Tianyi Zhang. 2023. Fairpy: A toolkit for evaluation of social biases and their mitigation in large language models.
Thiemo Wambsganss, Matthias Söllner, Kenneth R Koedinger, and Jan Marco Leimeister. 2022a. Adap- tive empathy learning support in peer review scenar- ios. In Proceedings of the 2022 CHI conference on human factors in computing systems, pages 1-17.
Sherry Ruan, Liwei Jiang, Justin Xu, Bryce Joe- Kun Tham, Zhengneng Qiu, Yeshuang Zhu, Elizabeth L. Murnane, Emma Brunskill, and James A. Landay. 2019. QuizBot: A Dialogue- based Adaptive Learn- ing System System for Factual Knowledge. pages 1-13.
Thiemo Wambsganss, Vinitra Swamy, Roman Rietsche, and Tanja Käser. 2022b. Bias at second glance: A deep dive into bias for german educational peer- review data modeling. In Proceedings of the 29th International Conference Computational Linguis- tics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 1344-1356. International Committee Computational Linguistics.
Rachel Rudinger. Chandler May, and Benjamin Van Durme. 2017. Social bias in elicited natural language inferences. In Proceedings of the First ACL Workshop on Ethics in Natural Language Process- ing, pages 74-79, Valencia, Spain, Association for Computational Linguistics.
Florian Weber, Thiemo Wambsganss, Seyed Parsa Ne- shaei, and Matthias Soellner. 2023, Structured per- suasive writing support in legal education: A model and tool for german legal case solutions. In Find- ings of the Association for Computational Linguis- tics: ACL 2023, pages 2296-2313.
Anuschka Schmitt, Thiemo Wambsganss, Matthias Söll- ner, and Andreas Janson. 2021. Towards trust re- liance paradox? exploring the gap between perceived trust in and reliance on algorithmic advice. In Inter- national Conference on Information Systems (ICIS),
volume 1, pages 1-17.
Changrong Xiao, Sean Xin Xu, Kumpeng Zhang, Yu- fang Wang, and Lei Xia. 2023. Evaluating read- ing comprehension exercises generated by LLMs: A showcase of ChatGPT in education applications. In Proceedings of the 18th Workshop on Innovative Use of NL. for Building Educational Applications (BEA 2023), pages 610-625, Toronto, Canada. Association for Computational Linguistics.
Kinshuk Sengupta, Rana Maher, Declan Groves, and Chantal Olieman. 2021. Genbit: measure and mit- igate gender bias in language datasets. Microsoft Journal of Applied Research, 16:63-71.
Samuel Sanford Shapiro and Martin B Wilk. 1965. An amalysis of variance test for normality (complete sam- ples). Biometrika, 52(3/4):511-611
Wei Xu, Marvin J. Dainoff, Liezhong Ge, and Zaifeng Gao. 2021. From Human- Computer Interaction to Human-AI Interaction: New Challenges and Oppor- tunities for Fnnalling Human-Centered AI. pages 1-73.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettle- moyer. 2019. Evaluating gender bias machine translation. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 1679-1684, Florence, Italy. Association for Computational Linguistics.
Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. ACL-HLT 2011 Proceedings of the 49th Annual Meeting of the Association for Computational Linguisticx: Human Language Tech- nologies, I: 1:180-189.
Xiaotian Su, Thiemo Wambsganss, Roman Rietsche, Seyed Parsa Neshaci, and Tanja Kser. 2023. Re- viewriter: Al-generated instructions for peer review writing. In Proceedings of the 18th Workshop on In- novative Use of NLP for Building Educational Appli- cations (BEA 2023), pages 57-71, Toronto, Canada. Association for Computational Linguistics.

---

A Interface of peer review writing during the Prolific study
Reviewriter Peer Reviews Writing
?
In this assignment, you need to write peer review of at least 300 words about business model. In doing you should try to include the strengths and weaknesses the business model well your own suggestions and ideas for improvement.
Press the Tab to accept the suggestion, the Esc key to reject it. Press the and down arrow keys to switch between suggestions.
Zunächst kann ich sagen, dass deine erste Aufgabe gut gelöst wurde. Du hast die Umwelt- Unternehmensanalysee gut strukturiert aufgebaut, eine gute Einleitung geschrieben und die einzelnen Fakten auf die verschiedenen Faktoren bezogen, auch die zu hohen Produktionskosten hast du raus gearbeitet. Zukünftige strategischen probleme individuell beurteilen und kreativ lösen, auch die großen fehler hast du immer auf deine strategischen fehler beziehen können. aufgabe hast für die hohen produktionskosten,
Word count: 71
Submit
Figure 4: A screenshot of the interface provide inline suggestions for peer review writing in the Prolific study, By clicking the question mark, people get detailed guidance on the peer review writing task and the usage of the tool. A simple text supports all typical interactions, such typing, selecting, editing, and deleting text, and caret movement via keys and mouse. In the input area, the sentences in black are the actual text, we display the Al-generated instruction in an inline format in gray. The model generates next-sentence predictions to give people a complete view of the idea. We provide three instructions each time, and people may the Tab key to accept, the Esc key to reject, and the Up and Down arrow keys to toggle through different instructions). The total number of words is displayed below the text area to inform people of their writing progress.
B Extra results of GenBit gender bias score with different window sizes
Scanes man-VWithen
Ger 3it Gender Blas Scores of Human-Written Reviows
0o181 Siv15 Bax Bo0o
94
CoIER Conter thes zooe
31
20 s0/0waik
51
Wendow
Windw
(a) Fixed context
(b) Infinite context
Figure 5: Comparing Genbit gender bias scores with two different context types and varying windows sizes ranging from 10 60.

---

C
Statistics on gendered words
Avg. Num words
Avg. Num male words
Avg. Perc male words
Avg. Num female words
Avg. Perc female words
Group
ML-based (G0)
183
1.08
0.58%
0.69
0.39%
No AI CG (GI)
286
1.79
0.64%
2.56
0.94%
FT GPT-2 (G2)
299
2.16
0.71%
2.18
0.71%
FT GPT-3 (G3)
293
2.30
0.77%
3.20
1.17%
GPT-3.5 (G4)
291
2.71
0.93%
2.98
1.03%
Table 5: Statistics of total words and gendered words per review from each group after data processing.
D WEAT analysis categorization form
Bias
#
Targets
Attributes
Flowers vs. Insects
Pleasant vs. Unpleasant
Conceptual
2
Instruments vs. Weapons
Pleasant vs. Unpleasant
9
Mental vs. Physical Disease
Temporary Permanent
Native vs. Foreign Names
Pleasant vs. Unpleasant
Racial
4
Native vs. Foreign Names (v2)
Pleasant vs. Unpleasant
5
Native vs. Foreign Names (v2)
Pleasant vs. Unpleasant (v2)
6
Male vs. Female Names
Career vs. Family
Gender
Math vs. Arts
Male vs. Female Terms
8
Science vs. Arts
Male vs. Female Terms
Table 6: WEAT analysis categorization from Wambsganss et al. (2022b) in three dimensions (conceptual, racial, and gender). this work, we use gender dimension tests. WEAT compares the association hetween two different target word lists (i.e. Math Arts) to attribute word lists (i.e. Male vs. Female terms). indicates the original WEAT test number (Caliskan et al., 2017).


Reviews:
[{"review_id": "YjT7rgWOkg", "reviewer": "Reviewer_bbvs", "summary": "", "questions": "None", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 1, \n\nWe thank you for the positive sentiment regarding the study design and motivation, interesting questions, and the valuable time you spent reviewing this paper. We address your suggestions point-by-point below and contribute several **new** analyses to bolster the paper:  \n1) We have run new sentence-level bias analyses for 6 aligned SEAT tests, further confirming our findings of no significant bias transfer.  \n2) We [publicly provide](https://anonymous.4open.science/r/Gender-bias-in-LLMs-AD7D/German%20SEAT/sent-weat1.jsonl) the German translation of SEAT tests, revised by two native German speakers. \n3) We [publicly provide](https://anonymous.4open.science/r/Gender-bias-in-LLMs-AD7D/README.md) the business model case study, the GPT suggestions for each group, and the cleaned and anonymized student reviews for groups G1 through G4. This dataset is provided in addition to the code for cleaning and analysis. We believe this will be a valuable resource and dataset for the NLP research community.  \n4) We conduct statistical tests (Chi-Squared for the categorical variables gender and education level, Kruskal-Wallis for age) to check for significant differences between demographic attributes (gender, age, education level) of groups G1-G4, finding no statistically significant difference between the populations.   \n\nWe believe that these new resources (dataset, code, study materials, sentence-level bias analysis, German translations for SEAT tests, and statistical analyses regarding study populations) increase our contribution to the field.\n\n**Concerning the limitation that our study was based on one single writing task**: we agree with the reviewer that our experiments are limited to a single scenario. However, this narrow scope was intended as it allowed for an in-depth, controlled analysis with a larger population of students, eliminating confounding variables and enhancing the study's rigor. Our primary purpose was to have high internal validity in this first experiment in order to examine whether the study design, conduct, and analysis answer the research question. The chosen peer review task is highly representative and educationally very relevant. A recent meta-analysis showed that peer feedback has a significant effect on academic performance [1], which holds across primary, secondary as well and tertiary education. The effect of peer feedback on academic performance was shown to be significantly higher compared to a condition without feedback (g = 0.31, p = 0.004) as well as to a condition in which the teacher provided feedback (g = 0.28, p = 0.007). Other meta-analyses have further confirmed the substantive benefits of peer feedback across a wide range of contexts [2, 3], demonstrating that it can yield greater learning benefits than teacher feedback.\n\n\nThe resulting data allowed us to observe the interaction between LLMs and human writing in a real-world context, thereby providing valuable insights into whether bias from LLMs transfers to human writing. We argue that even though the study focuses on one type of task, the implications of our findings are broad and pave the way for future research on the impact of AI in education across modeling, writing, and grading tasks. Our methodology lays the groundwork for future, more expansive studies, and a pipeline to study bias in educational tasks.\n\n\n[1] Double, K.S., McGrane, J.A. & Hopfenbeck, T.N. The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies. Educ Psychol Rev 32, 481–509 (2020). https://doi.org/10.1007/s10648-019-09510-3\n\n[2] Yan, Zi, et al. \"Effects of self-assessment and peer-assessment interventions on academic performance: A pairwise and network meta-analysis.\" Educational Research Review (2022): 100484.\n\n[3] Li, Hongli, et al. \"Does peer assessment promote student learning? A meta-analysis.\" Assessment & Evaluation in Higher Education 45.2 (2020): 193-211.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. To my knowledge, this is the first study investigating the potential bias that might seep into educational tasks and affect students' writing styles and perspectives.\n2. The study was well designed and implemented.", "reasons_to_reject": "(A) As indicated by the authors, a key limitation is that this study was based on one single writing task, which inherently limits the contributions of the paper.", "questions_to_authors": "None", "paper_topic_and_main_contributions": "This paper investigated an interesting problem, i.e., how bias might transfer through large languages models in the setting of AI-human collaborative writing. To investigate this problem, the authors conducted a large-scale user study to evaluate student writing in providing peer reviews for business cases in German. Then, the bias was measured by using GenBit gender bias analysis and Word Embedding Association Tests. The key contribution of this study is that, to my knowledge, this is the first study investigating the potential bias that might seep into educational tasks and affect students' writing styles and perspectives. The study is well designed and the presented findings are somewhat meaningful.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "ECczbX6E76", "reviewer": "Reviewer_TWkC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 2, \n\nWe thank you for the positive view of the work's motivation and writing structure, the constructive feedback, and the valuable time you spent reviewing this paper. We address your suggestions point-by-point below and contribute several **new** analyses to bolster the paper:  \n1) We have run new sentence-level bias analyses for 6 aligned SEAT tests, further confirming our findings of no significant bias transfer.  \n2) We [publicly provide](https://anonymous.4open.science/r/Gender-bias-in-LLMs-AD7D/German%20SEAT/sent-weat1.jsonl) the German translation of SEAT tests, revised by two native German speakers. \n3) We [publicly provide](https://anonymous.4open.science/r/Gender-bias-in-LLMs-AD7D/README.md) the business model case study, the GPT suggestions for each group, and the cleaned and anonymized student reviews for groups G1 through G4. This dataset is provided in addition to the code for cleaning and analysis. We believe this will be a valuable resource and dataset for the NLP research community.  \n4) We conduct statistical tests (Chi-Squared for the categorical variables gender and education level, Kruskal-Wallis for age) to check for significant differences between demographic attributes (gender, age, education level) of groups G1-G4, finding no statistically significant difference between the populations.   \n\nWe believe that these new resources (dataset, code, study materials, sentence-level bias analysis, German translations for SEAT tests, and statistical analyses regarding study populations) increase our contribution to the field.\n\n**Concerning the suggestion to use sentence-level bias metrics such as SEAT rather than word-level metrics like WEAT and GenBit**, we appreciate the input for a more comprehensive evaluation. We have taken this into account, and have now added new results from running a SEAT analysis on our model using the implementation from [Fairpy](https://github.com/hrishikeshvish/fairpy) (and plan to add it directly into the paper in Section 4.2). To understand why we originally did not include sentence-level bias metrics, we emphasize that the focus of our study was to investigate how biases from LLMs transfer into real-world writing tasks, which are fundamentally compositional in nature. Word-level metrics were specifically chosen because they allow for a nuanced understanding of how individual lexical choices—suggested by the LLMs—might or might not contribute to an overall biased narrative in students' writing. Word-level metrics offer granularity, allowing us to pinpoint exact instances of bias within the language model embeddings, a level of detail that is often lost in sentence-level evaluations. Moreover, WEAT and GenBit have been more widely validated than any sentence bias measures and are capable of identifying subtle biases in embeddings. Using sentence-level metrics could potentially dilute these subtle biases, offering a less granular view, especially in the context of peer reviews, which are often short and focus more on objective evaluations than narrative discourse. \n\nDespite these concerns, we believe both types of analyses have merit, and together, they provide a richer picture of bias in embeddings. **We present the results of our SEAT analysis on the fine-tuned GPT model used for G2 below**, the analogous counterpart to our WEAT analysis included in Section 4.2. SEAT is a sentence-based embedding test, computing the sentence embeddings for two sets of target sentences (in test 6, male and female names) and two sets of attribute sentences (career and family). The pairwise embedding distance between each target sentence and attribute sentence (e.g. “Das ist Hans” and “Dies ist eine Exekutive”) is computed. Across tests 6, 6b, 7, 7b, 8, and 8b, which correspond to the analogous gender tests of WEAT, there is no significant difference found between the embeddings of the target sentences and attribute sentences (i.e. in SEAT 6, male and female names are both equally similar to career and family words), computed using p-values and a hypothesis test as per the standard implementation. Each test also has an effect size, ranging from -2 to +2, with 0 representing a completely neutral effect between both target words and attribute words. Most effect sizes are within 0.1, so there are only minimal bias effects. The comparatively largest effect (SEAT test 7) is -0.705 from comparing Math and Arts in association with Male and Female terms, but this was still found to have insignificant bias. These results are in line with the WEAT analysis on the word level granularity, which also found no significant gender bias in the model embeddings.\n\n|Test|Hypothesis Test|Effect size|\n|:----|:----|:----|\n|sent-weat6|no difference between MännlicheNamen and WeiblicheNamen in association to attributes Karriere and Familie;|0.021|\n| |(no difference between Male Names and Female Names in association to attributes Career and Family)| |\n|sent-weat6b|no difference between Wissenschaft and Weibliche Begriffe in association to attributes Karriere and Familie;|-0.074|\n| |(no difference between Male Terms and Female Terms in association to attributes Career and Family)| |\n|sent-weat7|no difference between Mathe and Kunst in association to attributes Männliche and Weibliche Begriffe;|-0.705|\n| |(no difference between Math and Arts in association to attributes Male Terms and Female Terms)| |\n|sent-weat7b|no difference between Mathe and Kunst in association to attributes MännlicheNamen and WeiblicheNamen;|-0.209|\n| |(no difference between Math and Arts in association to attributes Male Names and Female Names)| |\n|sent-weat8|no difference between Wissenschaft and Kunst in association to attributes Männliche and Weibliche Begriffe;|-0.069|\n| |(no difference between Science and Arts in association to attributes Male Terms and Female Terms)| |\n|sent-weat8b|no difference between Wissenschaft and Kunst in association to attributes MännlicheNamen and WeiblicheNamen;|0.078|\n| |(no difference between Science and Arts in association to attributes Male Names and Female Names)| |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is interesting to explore the impact of bias from LLMs on student writing.\n2. The paper is clear and well-structured.", "reasons_to_reject": "To evaluate the bias, maybe it would be better to use sentence-level bias metrics, e.g. SEAT rather than word-level metrics.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates whether bias in LLM affects human writing. They conduct a user study with 231 students, which are divided into five groups: one with suggestions from recommender systems, one with no assistance, two with suggestions from GPT2 and GPT3 fine-tuned on non-biased corpus, and one with suggestions from pre-trained GPT3.5. By analyzing the reviews, suggestions, and model embeddings, they claim that there is no significant difference in gender bias between the peer reviews of groups with and without LLM suggestions.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "p3VIV3jKtH", "reviewer": "Reviewer_YUR3", "summary": "", "questions": "Was each user shown the same pre-defined business model (ski instruction) in the Prolific studies or did this vary across participants?\n \nWhile you mention that the context of the reviews are domain-independent, I wonder if the business models that are being reviewed are not related to topics that would easily exhibit gender bias to begin with? Do you have examples of the topics of these business models?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3, \n\nWe thank you for the constructive feedback and the valuable time you spent reviewing this paper. We appreciate and agree with your perspective on the strengths of the paper: evaluating bias across multiple stages of the LLM pipeline, using multiple models and metrics, and highlighting a relevant and increasingly important use case of LLMs in society. We address your suggestions point-by-point below and contribute several **new** analyses to bolster the paper:  \n1) We have run new sentence-level bias analyses for 6 aligned SEAT tests, further confirming our findings of no significant bias transfer.  \n2) We [publicly provide](https://anonymous.4open.science/r/Gender-bias-in-LLMs-AD7D/German%20SEAT/sent-weat1.jsonl) the German translation of SEAT tests, revised by two native German speakers. \n3) We [publicly provide](https://anonymous.4open.science/r/Gender-bias-in-LLMs-AD7D/README.md) the business model case study, the GPT suggestions for each group, and the cleaned and anonymized student reviews for groups G1 through G4. This dataset is provided in addition to the code for cleaning and analysis. We believe this will be a valuable resource and dataset for the NLP research community.  \n4) We conduct statistical tests (Chi-Squared for the categorical variables gender and education level, Kruskal-Wallis for age) to check for significant differences between demographic attributes (gender, age, education level) of groups G1-G4, finding no statistically significant difference between the populations.   \n\nWe believe that these new resources (dataset, code, study materials, sentence-level bias analysis, German translations for SEAT tests, and statistical analyses regarding study populations) increase our contribution to the field.\n\n\n**Concerning the reviewer's suggestion to first evaluate biases in the students' original reviews and then compare them with the LLM-assisted reviews**, we appreciate the interesting alternative methodology. However, our study aims to explore the influence of LLMs on students' writing in a more realistic setting. Peer-reviewing scenarios conducted in MOOCs often have students receive real-time assistance during the initial writing process rather than after the fact (which is more of an autograding task). We intend to examine LLMs in their role as an interactive tutor and writing collaborator as opposed to a grader. Consequently, the suggested framework is designed to predominantly assess students' reflective capacities rather than their interactions with assistance systems. Our approach captures how biases may or may not transfer when students use LLMs in real time, thereby offering more generalizable and ecologically valid insights.\n\n\nIn order to control for the content aspect (as the content of the provided business model could inadvertently have an influence on the expected bias), we present each participant exactly **the same predefined business model**. We therefore conducted a randomized control study on Prolific: participants were divided into four groups (G1-G4), controlling for the sensitive variables of education level, language, age, and gender. While there might be (small) observable differences in the sample means or medians for these variables, additional statistical analysis confirms that there are no significant differences between groups and the randomization has worked correctly. For the age attribute, a Shapiro-Wilk test indicates that the data is not normally distributed and we have therefore conducted a non-parametric Kruskal-Wallis test, which confirms that there are no significant differences between groups G1-G4 in terms of age (H = 2.24, p = .52). Furthermore, a Chi-Square test confirms that there are also no significant differences between the four Prolific groups in terms of gender (X2 = 0.0149, p = .99) or education level (X2 = 6.65, p = .35). \nWe acknowledge that the setting and population of G0 are different from the settings and populations of G1-G4, and make this explicitly clear in our analysis. We did not find it ethical to adversely impact real-world students (G0) in their core curricula in the case the bias did transfer from LLMs to students. Therefore, we aimed to recruit a comparable student population on Prolific for examining downstream bias transfer from LLMs. **Notably, in our analyses, we do not directly compare G0 results to the rest of the G1-G4 results, and in our figures, we include clear separations of G0 from the Prolific study.**\n\n**Concerning the different contexts in which the business models are being reviewed** (peer reviews in G0 vs. predefined models in G1-G4), we understand your concern. However, it's essential to clarify that despite the difference in the source of the peer reviews, both formats were designed to be rigorously equivalent in terms of complexity, content, and length. The aim of this design was to isolate the effect of the language model's suggestions on writing, thereby keeping the task as consistent as possible across all groups in the Prolific study. Our work aims to use G1-G4 to build confidence with AI-assisted writing tools so that they can be used in the real classroom in a future study without adversely affecting the students in G0. \n\n**Concerning the question of whether each user was shown the same predefined business model for the Prolific studies**, we appreciate that the reviewer points out an aspect that could be considered a variable in the experiment. In the study, all participants in the Prolific groups (G1–G4) were **exposed to the exact same predefined business model** on ski instruction to maintain task uniformity and comparability across groups. We aimed to control for the content variable so that any observed effects or lack thereof could be attributed more directly to the writing assistance provided by the different LLMs (as opposed to the content of the business model itself). This methodological choice is crucial for isolating the impact of writing assistance on potential bias transfer. We will clarify this aspect in the revised manuscript to eliminate any ambiguity.  \n\n**Concerning the question about the domain-independence of the business models under review**, we appreciate the reviewer's scrutiny. We have now included the materials from the ski business case study in our repository to be released alongside the paper. The business model chosen for the study was taken from a real-world classroom **deliberately chosen to be as neutral as possible in relation to gender-biased topics**. The objective was to minimize external variables that might contribute to gender bias in the reviews. Nonetheless, your query highlights a valuable point, and we will include more examples of business model topics (technology, healthcare, food services) in the revised paper to further discuss the nuances of the German word genders in these cases as an extension of the discussion about this raised directly in the Limitations section. This will strengthen the validity of our claim that any detected bias is largely a result of the LLMs or other elements in the writing support pipeline, rather than the subjects under review. Additionally, it is imperative to note that given that all experimental groups were exposed to an identical business model, any potential impact this might have would be uniformly distributed across all groups. Given the complete randomization of our experiment, the sole distinction between groups lies in the treatment they received. Consequently, by definition, any observed differences between groups can be causally attributed to the treatment.\n\n**Concerning the reviewer's request for examples of downstream gender bias that occurs due to the usage of LLMs**, we appreciate the suggestion for enriching the content in a broader perspective. We address the reviewer’s concern with a discussion of multiple examples where downstream gender bias can be measured within the educational context. The focus of our paper is to empirically investigate whether bias is transferred from LLMs to human writing in **educational** settings. We explicitly designed a study to build confidence for whether LLMs can be used without adverse gender bias effects in the classroom instead of a less supported statement on a broader context of downstream tasks. There has been a large rise in AI in education research, specifically focusing on language models, in the last 3 years [3, 4]. To bridge the gap from research to practice, studies focusing on the safety of these models in this exact applied context of education are necessary for the field to move forward, especially as the participants of educational technology tools are often underage. \n\nWithin the context of educational tasks, there are many ways to measure gender bias in classroom writing. The most prevalent way is to measure real-time assistance on writing samples (i.e. peer reviews), but other strategies include measuring changing bias across a time series of reflective journaling, analyzing emotion in essay-writing on gender-sensitive topics, discussion forum interactions with classmates from different demographics, and psychological word association tests. We will include this larger discussion of concrete examples of downstream educational tasks to measure bias.\n\n[1] Philip J. Guo and Katharina Reinecke. 2014. Demographic differences in how students navigate through MOOCs. In Proceedings of the first ACM conference on Learning @ scale conference (L@S '14). Association for Computing Machinery, New York, NY, USA, 21–30. https://doi.org/10.1145/2556325.2566247\n\n[2] Tomlinson, Carol Ann. \"Teaching for excellence in academically diverse classrooms.\" Society 52.3 (2015): 203-209.\n\n[3] Zhai, Xuesong, Xiaoyan Chu, Ching Sing Chai, Morris Siu Yung Jong, Andreja Istenic, Michael Spector, Jia-Bao Liu, Jing Yuan, and Yan Li. \"A Review of Artificial Intelligence (AI) in Education from 2010 to 2020.\" Complexity 2021 (2021): 1-18.\n\n[4] Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors highlight an import aspect of LLM usage in the educational setting for suggestions and rewrites. This is an emergent domain of study given the popularity of LLMs in general society in the past few months.\n\nThe authors use multiple gender bias metrics and models for their study.\n\nThe study evaluates multiple aspects of the NLP pipeline for bias: model embeddings, model suggestions, and output reviews.", "reasons_to_reject": "I have some concerns regarding the evaluation setup. Given that each group contains different sets of users, it seems that this can additionally affect results when comparing LLM usage across groups. This concern also stems from Table 1, which shows that there is a large variation in age and education across the groups.\n\nMy understanding is that users are evaluating each others’ business models in G0 and a pre-defined business model in G1-G4. This additional variation across groups can additionally influence results when compared to G0.\n\nFor me, it would make more sense to evaluate biases in the students’ reviews, then provide students with the interface to edit their reviews with the help of a LLM and evaluate their edited reviews for bias against their original reviews.\n\nIt would be nice to show some concrete examples of downstream gender bias that can occur due to the usage of LLMs.", "questions_to_authors": "Was each user shown the same pre-defined business model (ski instruction) in the Prolific studies or did this vary across participants?\n \nWhile you mention that the context of the reviews are domain-independent, I wonder if the business models that are being reviewed are not related to topics that would easily exhibit gender bias to begin with? Do you have examples of the topics of these business models?", "paper_topic_and_main_contributions": "The authors describe an analysis of the downstream impact of LLM usage on gender bias in the educational setting. This is studied across different groups of users that utilize different, or no, LLMs for writing suggestions in the peer reviewing process. The authors first analyze the final written reviews for biased language and find no significant differences across different groups of users. They then study where bias may stem from, in regards to the model embeddings and model suggestions. These results show gender bias in model embeddings that do not translate to bias in model suggestions.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 410: For “an” infinite context?\n \nLine 513: finetuned"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found the problem interesting and the subject of the paper novel. The methodology was seen as fairly comprehensive and sound overall, and the paper’s writing is clear and easy to follow. There were some concerns about the evaluation setup, but these appear to be addressed by the authors in their rebuttals, so I believe the soundness should not be negatively affected by most of these evaluation issues. One reviewer still felt that the scope of the paper was a bit limited as the paper focused on a single task. None of the reviewers were particularly excited about the paper (all felt ambivalent).
Decision: Accept-Findings