Reviewriter: AI-Generated Instructions For Peer Review Writing
Year: 2023

Abstract:
Large Language Models (LLMs) offer novel opportunities for educational applications that have the potential to transform traditional learning for students. Despite AI-enhanced applications having the potential to provide personalized learning experiences, more studies are needed on the design of generative AI systems and evidence for using them in real educational settings. In this paper, we design, implement and evaluate Reviewriter, a novel tool to provide students with AI-generated instructions for writing peer reviews in German. Our study identifies three key aspects: a) we provide insights into student needs when writing peer reviews with generative models which we then use to develop a novel system to provide adaptive instructions b) we fine-tune three German language models on a selected corpus of 11,925 student-written peer review texts in German and choose German-GPT2 based on quantitative measures and human evaluation, and c) we evaluate our tool with fourteen students, revealing positive technology acceptance based on quantitative measures. Additionally, the qualitative feedback presents the benefits and limitations of generative AI in peer review writing.

heading: 1 Introduction
text: Peer reviewing is a process by which learners provide formative feedback to each other on an individual task based on assessment criteria (Sadler and Good, 2006; Rietsche and Söllner, 2019). Research has found theoretical and empirical evidence for the positive effects of peer reviews on critical thinking skills (Lin et al., 2021; Ibarra-Sáiz et al., 2020), communication skills (Lai, 2016), and learning motivations (Hsia et al., 2016). The prevailing practice of peer review in tertiary education is evident in the eruption of massive open online courses (MOOCs) (Li et al., 2016). In these large-scale learning scenarios, peer review is particularly important since it is challenging for teachers to give effective oneby-one feedback due to immersive workload and
shortage of time (Er et al., 2021). However, according to Oliver (1982), a challenge that plagues many student writers, including those having satisfactory grammar and spelling skills, is writer’s block. It was defined by Rose (1980) as "that frustrating, self-defeating inability to generate the next line, the right phrase, the sentence that will release the flow of words again." A collaborator who provides instructions and points out new directions might help alleviate writer’s block (Clark et al., 2018) and the combination of a writer’s own ideas with suggested ideas is a form of psychological creativity (Boden et al., 2004). Novel LLMs have the potential to address the challenge of writer’s block by generating suggestions for the next lines, right phrases, or sentences, thereby facilitating the flow of ideas (Gero et al., 2022), and helping students compose responses more efficiently (van Dis et al., 2023; Gao and Jiang, 2021). There are LLM-based collaborative writing tools to provide support for various writing tasks, including story writing (Yang et al., 2022), science writing (Gero et al., 2022), and screenwriting (Mirowski et al., 2022). However, few have investigated the utilization of generative AI for peer review writing tasks. Therefore, in this paper, we build and evaluate Reviewriter which can provide AI-generated instructions tailored to students’ needs while writing peer reviews. It suggests possible directions based on students’ input to inspire divergent outcomes while still leaving learners in control of the final text.
To investigate how to provide students with help to overcome writer’s block in peer review writing, we conduct a literature review to gather insights for a peer review support system. We summarize five user requirements from interviews with twelve graduate students. Based on those, we develop seven design principles for providing AIgenerated instructions in peer review tasks. Next, we search peer review corpora satisfying certain criteria and pre-process 11,925 student-written peer
57
review texts in German (Wambsganss et al., 2022b). We use it to fine-tune three language models to provide students with informative instructions. The best results according to training loss and human evaluation of fluency and correctness are achieved by German GPT-2. Then, we implement the design principles into the system to provide AI-generated instructions for peer review writing. Finally, in a mixed-method study with our full-working prototype, we evaluate the performance of the tool in a real-world learning exercise with fourteen students, and four of them also participated in the design interview. We assess the technology acceptance and level of enjoyment of the tool using well-defined constructs from Venkatesh and Bala (2008); Venkatesh et al. (2003) and also collect qualitative feedback from students.
Our research makes three contributions to the innovative use of NLP in education. Firstly, we provide insights and practical design considerations for incorporating AI-generated instructions in peer review writing tasks to overcome the known challenge of writer’s block (Oliver, 1982). Secondly, we present and compare three open-source language models fine-tuned on a selected corpus of 11,925 student-written peer review texts in German. Lastly, we build Reviewriter, which implements seven functionalities with pedagogical design considerations and evaluates it on fourteen students from tertiary education. Our findings suggest that the tool providing AI-generated instructions in students’ peer writing tasks leads to high ease of use and a high intention to use for students in their review writing process. Moreover, in the qualitative feedback, we find that the model has the potential to provide novel ideas for students to continue in depth. However, like other LLMs, it suffers
from hallucination (Maynez et al., 2020) by producing factually incorrect and nonsensical answers, this invites further research to overcome and mitigate artificial hallucination. With Reviewriter, we present an interface with design rationales and an evaluated tool that other researchers can build upon to explore the effects of LLMs and the benefits and limitations of generative AI for writing peer reviews and building educational applications.

heading: 2 Related work

heading: 2.1 Student peer reviewing
text: There has always been significant interest in the study of peer reviews in the NLP community. Jia et al. (2022) introduced an approach called incremental zero-shot learning (IZSL) to address the issue of insufficient historical data for peer reviews. Wambsganss et al. (2022a) used empathy detection algorithms from NLP to analyze the given text and provide adaptive feedback in students’ peer writing process. Moreover, several works have investigated how to embed classification models to support students in peer review writing. For example, researchers have explored the use of these models to develop argumentation skills (Wambsganss et al., 2020), support cognitive and emotional empathy writing (Wambsganss et al., 2021), and assess the specificity of written peer feedback (Rietsche et al., 2022). While NLP models, particularly LLMs, have the potential to deliver adaptive learning content (Adiguzel et al., 2023; Qadir, 2022), little research has focused on how to leverage their ability to provide tailored instructions for students during peer review writing (Darvishi et al., 2022). van Dis et al. (2023) mentioned benefits provided by generative AI for completing peer review tasks quickly. Experimental results from Gao and
Jiang (2021) showed that the effectiveness of generated suggestions, regardless of their performance quality, has consistently helped humans compose responses more efficiently when providing suggestions. In addition, Gero et al. (2022) demonstrated that students find it faster and easier to draw on language from generated texts than to write a sentence from scratch, even when given well-known information. Therefore, we propose a novel peer review writing tool Reviewriter, by leveraging the power of generative models, it can provide students with adaptive instructions to help them overcome writer’s block in peer review writing.

heading: 2.2 NLP for writing support
text: With the massive success of ChatGPT, NLP is rapidly evolving as a key tool in writing support. On one hand, there is widespread adoption of generative AI in practice. Commercial writing assistants like Monica 1, a ChatGPT-powered extension, can support copywriting. And specialized applications like Jenni AI 2, Jasper AI 3 and Notion AI 4 can support creative writing. They are not only able to complete sentences but also generate the whole blog post and many other types of content including essays, emails, stories, and speeches based on users’ input. On the other hand, many studies have focused on the use of language models for writing support in tertiary education. For instance, researchers have explored the use of these models for academic writing (Gero et al., 2022), fiction writing (Yang et al., 2022), and text summarization (Dang et al., 2022). Despite the widespread adoption of NLP in writing instruction, many models, including ChatGPT, remain general-purpose tools that have not been fine-tuned for specific tasks (Chen et al., 2023) or designed for particular educational settings (Kuhail et al., 2023). Embedding the AI techniques in a student-centered design is a complex task with several socio-technical challenges (Xu et al., 2021), including data collection (ZawackiRichter et al., 2019), potential bias (Adiguzel et al., 2023) or discrimination (Pedróf et al., 2019) in the data, inadequate dataset training (Kuhail et al., 2023), incorporating the models, lack of student involvement in the design process (Verleger and Pembridge, 2018), lacking feedback on the generative system (Kuhail et al., 2023), and evaluating
1https://monica.im/ 2https://jenni.ai/ 3https://www.jasper.ai 4https://www.notion.so/product/ai
student perceptions (Xu et al., 2021). The present work provides insights into how to embed generative AI into peer review writing by establishing student-centered design with pedagogical considerations. We carefully select an unbiased corpus with a sufficient amount of peer review text to finetune language models. Furthermore, we evaluate student perceptions quantitatively and collect qualitative feedback on the generative AI system.

heading: 3 Generative modeling to provide students adaptive instructions

heading: 3.1 The peer review dataset
text: To make sure our system is skilled in providing adaptive instructions for writing peer reviews and to improve accuracy and efficiency for human-AI interaction (Lee et al., 2022b), we decide to finetune language models with a peer review dataset. We start by searching the literature for a corpus that fulfilled the following criteria: a) it contains a large amount of student-written text in one particular domain (e.g., business model feedback) (Kuhail et al., 2023), b) it consists of a sufficient size to represent different nuances of characteristics in a balanced fashion (e.g. specificity, helpfulness) (Rietsche et al., 2022), and c) it does not possess a significant bias (e.g. gender, racial or social discrimination) (Adiguzel et al., 2023). The business model peer review corpus published in Wambsganss et al. (2022b) fulfilled all these requirements. The corpus consists of 11,925 peer reviews collected at a university in the German-speaking area of Europe. They were written by first-year master’s students in a business department course. The student population has an average age of 24.6 years old with a standard deviation of 1.7 years. Students wrote approximately 9 peer reviews per course with an average length of 220 words. Furthermore, Wambsganss et al. (2022b) showed that this collected corpus does not reveal many biases in nine WEAT co-occurrence analyses or in the GloVe embeddings. This corpus provides us with a sufficient amount of unbiased peer review texts to fine-tune language models for adaptive instructions in the domain of business peer reviews.

heading: 3.2 Data pre-processing
text: To ensure the model could generate high-quality instructional text, we select reviews written from 2016 to 2021 with a rated helpfulness score greater than five on a 1 - 7 Likert Scale (1: low, 4: neutral,
7: high). We start by removing HTML tags, irrelevant information like PDF file names and specific information like URLs, keywords (revealing the identity of students), and questions asked to write reviews which some students copied to their review text (Appendix A.1). We also expand abbreviations as shown in Appendix A.2. Then, we shuffle and divide cleaned data into train and test datasets with proportions of 0.8 and 0.2 for fine-tuning and evaluating the language model. Lastly, all sentences are tokenized with model-specific tokenizers.

heading: 3.3 The generative models
text: Transformer-based language models, such as BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), using the pre-training and fine-tuning paradigm, have revolutionized NLP and achieved state-of-the-art records on various tasks. These models are first pre-trained in a self-supervised fashion on a large corpus and fine-tuned for specific downstream tasks (Wang et al., 2018). In our case, to provide AI-generated instructions for German peer review writing, we use pre-trained causal language models on the HuggingFace platform (Wolf et al., 2020) for German text generation. We choose them because there is no usage limitation and by utilizing open-source technology, we contribute to LLM transparency (van Dis et al., 2023; Adiguzel et al., 2023), allowing other researchers to easily replicate our find-
ings or build upon them. Therefore, we selected two German GPT-2 models (dbmdz/german-gpt2 5 and benjamin/gerpt2-large 6) and one multilingual model BLOOM (Scao et al., 2022) (bigscience/bloom-560m 7). We did not use GPT3 for fine-tuning since it was not open-source available at the time of our research. For all of them, we fine-tune the pre-trained models following the default hyperparameter settings (Appendix A.3) with block size 128, and 500 warm-up steps.
We compare training loss and used human evaluation to select the best model. Note that GerPT2large already performs well (Appendix A.4 for sample generated text) after ten epochs of training, even with higher training loss compared to the other two models (Table 1). However, it suffers a long inference time (a student needs to wait around 10 seconds to get instructions given 40 words) compared to the other two models (5 seconds with the same input). Therefore, we decide to further evaluate German GPT-2 and BLOOM. We conduct a human evaluation of the quality of the generated response. Specifically, we sample ten instructions generated by each model and present them to two German researchers to evaluate their fluency and correctness. From the evaluation of both parties, German GPT-2 yields more coherent results than
5https://huggingface.co/dbmdz/german-gpt2 6https://huggingface.co/bigscience/bloom-560m 7https://huggingface.co/benjamin/gerpt2-large
the BLOOM model and there are more meaningless sentences from the response generated by BLOOM than by German GPT-2. Therefore, we decide to use the German GPT-2 model as the base for the tool with a default temperature of 1.0 for generating the next token.

heading: 3.4 The generative system
text: To design a system providing AI-generated instructions for peer review writing, we first draw on insights from relevant literature. Following the methodology of Cooper (1988), we analyze humanAI interaction (Shen and Wu, 2023; Chan et al., 2023; Lee et al., 2022b) and NLP-supported peer review systems (Alqassab et al., 2023; Darvishi et al., 2022). Then, to gather insights into the needs of writing peer reviews with AI-generated instructions for tertiary education, we conduct semi-structured interviews with twelve graduate students. We reach out to a group of computer science students who previously registered in a business class and have experience writing peer reviews on business models, and to students in our university for general recruitment. The participants have a diverse background in computer science, business, or psychology, and a mean age of 24.50 years (SD = 2.02), including two females and ten males (representing the distribution of computer science students at our school). Half of them had experience writing peer reviews, while the others did not. Each interview lasts around 30 to 50 minutes. We use the expert qualitative interview method outlined in Brinkmann (2013) and Gläser and Laudel (2009) to gain an initial understanding of students’ needs for receiving adaptive instructions in peer review writing. We ask topics about prior experience with technology-based writing systems, perceptions of existing writing systems (e.g., Grammarly), difficulties in writing peer reviews, and desired functionalities for a system to support peer review writing. We transcribe the interviews and identify five
clusters of requirements following Cohn (2004). We find that 75% of the students would like to interact with a clean and straightforward interface (user requirement - UR 1). Two-thirds of interviewees asked for intuitive guidance on how to interact with the tool (UR 2). And 41.7% of them said that they would like to see more than one instruction to choose from (UR 3). One-third of the students stated that they prefer to view a complete piece of instruction rather than words or phrases to formulate a concrete idea (UR 4). Lastly, two-thirds of them indicated that they would like to see the number of words they have entered to have better control over the structure of the review (UR 5).
With insights derived from the literature review and requirements from student interviews (similar to Rietsche et al. (2018)), we develop seven design principles (Table 2) and further map them to seven functionalities (Figure 3 F1 - F7) in Reviewriter, a responsive web application to provide AI-generated instructions for peer review writ-
ing. The design is student-centered and has two main components: a neat interface with key commands for text editing (Figure 3) and a generative language model in the backend 3.3. To foster the independent thinking of students and discourage overreliance on technology (Adiguzel et al., 2023), we organize a workshop with two senior researchers to deliberate on the optimal timing for presenting the generated instructions. Combined with studies Buschek et al. (2021); Bhat et al. (2021), we decide to present instructions until students have entered a minimum number of words and put a certain amount of delay before showing instructions to minimize potential disruptions caused by irrelevant information from model hallucination (Maynez et al., 2020). Figure 2 presents the system architecture. The student starts with writing the beginning of the review. The system will display instructions until students enter at least 25 words. After this threshold, when the student gets stalled, by pressing the spacebar, they will trigger the model in the backend to generate instructions. After the keypress, there is a delay of eight seconds before they receive instructions. To preserve the context while avoiding too much overhead for querying the mode, we pass the last twenty words from the input to the model. According to UR 4, and supported by Calderwood et al. (2020), overly brief suggestions are often unhelpful. To ensure clarity and concision, we limit each instruction to a maximum of 60 tokens, which is approximately 45 words 8. In their experiment with one, three, and six instructions, Buschek et al. (2021) discovered that multiple instructions can facilitate the identification of useful phrases and boost their acceptance rate. We decide to present three instructions each time considering the cost-benefit tradeoffs for efficiency (e.g. reading time vs diversified content). The student controls the final output by checking multiple instructions and deciding whether to accept or reject them. They are free to add, delete, and replace the generated content.
8https://help.openai.com/en/articles/4936856-what-aretokens-and-how-to-count-them

heading: 4 Evaluation of Reviewriter

heading: 4.1 Experimental setup
text: To assess our prototype, we conduct a mixedmethod study with fourteen students from a public university in Europe. We reached out to students who have participated in our previous design interview and also recruited students on campus. Fourteen students–eleven males and three females– participated in the evaluation. Three of them were undergraduate students and the rest were graduate students. Four graduate students also participated in our previous design interview. They were all native German speakers and expressed interest in getting AI-generated instructions when writing peer reviews. They have diverse backgrounds, including computer science, robotics, and business with a mean age of 25.33 years (SD = 3.60). The evaluation is conducted either face-to-face or remotely with a conference tool. Each student screen records their writing process, the interviews are also recorded and transcribed by a researcher.
1. Pre-test (10 minutes): The experiment starts with a pre-survey that has five questions (Appendix B.1) followed by two videos. The first four questions measure the learners’ level of innovation in the field of information technology, following Agarwal and Karahanna (2000). They need to rate their agreement with a statement on a Likert scale ranging from 1 (totally disagree) to 7 (totally agree), with 4 being neutral (Likert, 1932). Following the pre-survey, we present two videos. The first video introduces a business model for a platform that connects ski instructors with learners, and the second video provides guidance on how to use Reviewriter.
2. Peer review writing (30 minutes): In this phase, students are asked to write a review for a peer’s business model. Specifically, they are asked to elaborate on strengths, weaknesses, and suggestions for improvement of the given business model. We instruct students not to use search engines and spend a minimum of 15 minutes on the task. A countdown indicates the remaining time.
3. Questionnaire and interview (10+10 minutes): In the post-survey, we ask 29 questions (Appendix B.2) to measure perceived ease of use, perceived ease of interaction, perceived level of enjoyment, perceived level of excitement and perceived usefulness, following the technology acceptance model of Venkatesh and Bala (2008) and Venkatesh et al. (2003). All constructs are measured with a 1- to 7-point Likert scale. Moreover, we ask several qualitative questions to further examine students’ attitudes toward AI-generated instructions and capture the demographics.

heading: 4.2 Quantitative analysis and qualitative feedback
text: To measure student perceptions of AI-generated instructions for peer review writing, we calculate the following constructs on a 1- to 7-point Likert scale (Table 3): perceived ease of use (M1 = 6.07, SD1 = 0.83), perceived ease of interaction (M2 = 5.50, SD2 = 1.22), perceived level of excitement (M3 = 5.64, SD3 = 1.15), perceived level of enjoyment (M4 = 5.43, SD4 = 1.16), and perceived usefulness (M5 = 4.64, SD5 = 1.34). The results show that the participants rate positively using Reviewriter to receive adaptive instructions. Moreover, the mean values of the tool are also very promising when comparing the results
to the average of the scale. All results are better than the neutral value of four. This fosters motivation and engagement to use the learning application. Malik et al. (2021) found that perceived ease of use (M1 = 6.07) and usefulness (M5 = 4.64) positively influence student adoption intentions and their attitudes toward AI-based applications. The positive levels of perceived ease of interaction (M2 = 5.50), excitement (M3 = 5.64), and enjoyment (M4 = 5.43) suggest that the technology has been accepted favorably. This is especially important for learning tools to ensure students are perceiving the usage of the tool as enjoyable, useful, and easy to interact with (Marangunić and Granić, 2015). These are promising results for using this tool to receive AI-generated instructions in a peer review setting.
In addition to quantitative scores, we incorporate qualitative open-ended questions to further understand student attitudes toward writing with AI-generated text and how the instructions impact their writing process. We translate the responses from German and cluster the representative ones (Appendix B.3). The general attitude towards Reviewriter was very positive. Five students stated concretely the benefits of Reviewriter on their writing process. Three students mentioned the system is simple and easy to interact with. On the adoption of the generated instructions, one student used them every time, two students stated that they did not find anything useful in the instructions. Another two students reported that they never used the complete instructions but they picked up ideas or keywords from them. Five of them used instructions three to five times, and the rest stated that they use the AI-generated instructions quite frequently and did not provide an exact number. Moreover, it is interesting to note that there are divergent opinions on the delay of the system. Three students complained about the waiting time was too long while two other students were in favor of
the delay and stated that the waiting time left them room to think. Finally, students enjoyed the diverse content in AI-generated instructions while noticing there were ungrammatical sentences and irrelevant phrases from time to time.

heading: 5 Discussion
text: Peer review writing is an increasingly important educational task in large-scale or distance learning scenarios since it enables personalized feedback to be delivered at scale, thereby lessening the workload of instructors (Er et al., 2021) and boosting learners’ motivation (Hsia et al., 2016). However, during writing peer reviews, students may experience obstacles such as writer’s block Rose (1980) where they struggle to generate the next line, the right phrase, or the sentence Oliver (1982). LLMs can help to overcome this obstacle by producing adaptive instructions based on students’ input, which ultimately aid in the seamless progression of thoughts (Gero et al., 2022). To do so, we develop a novel peer review writing tool called Reviewriter. It allows students to use AIgenerated instructions as an inspiration and incorporate those ideas into their own work in a creative and original way, such as by adapting, mixing, or reinterpreting those instructions (Qadir, 2022).
Our study contributes at least three key aspects to the innovative use of NLP in education. First, we explore the personalization of AI-generated instructions in a specific pedagogical scenario - peer review writing (Pardos and Bhandari, 2023) by gathering insights from literature review and student interviews (Verleger and Pembridge, 2018). Second, in contrast to Lee et al. (2022a) which used GPT-3 without adaptation for collaborative writing, we fine-tune three German language models on a corpus selected based on certain criteria to provide specialized content with high quality. Afterward, we choose German-GPT2 based on quantitative measures and human evaluation. Third, as noted
in Kuhail et al. (2023), "lack of feedback" is one of the challenges to using generative models in education. Therefore, we evaluate our tool with fourteen students and the result reveals positive technology acceptance based on quantitative measures. Through our qualitative evaluation, we find that students generally enjoyed seeing generated instructions with varied content to spark ideas. And they were enthusiastic and excited about writing with generative language models. We recognize that there is a need for further research on the effectiveness of LLM-based writing support tools in various contexts, as well as the improvement of faithfulness and factuality in AI-generated instructions (Maynez et al., 2020). Nonetheless, our study contributes to the growing body of knowledge on the potential of generative AI to provide personalized writing instructions and enhance students’ learning experiences (Pardos and Bhandari, 2023).

heading: 6 Conclusion and future work
text: To help students mitigate writer’s block during peer review writing, we design, build, and evaluate Reviewriter, a novel tool that aims to provide students with AI-generated instructions during their peer review writing process. We provide design insights with pedagogical considerations of integrating LLMs into peer-review writing systems. Our evaluation involves fourteen students from tertiary education, who reported enjoying the interaction with the system, finding it easy to use, and expressing interest in using similar tools in the future. They also pointed out that the relevance of the generated instructions could be further improved. We present Reviewriter, including its design rationales and evaluated interface, as a contribution to the exploration of LLMs’ potential in innovative NLP-based approaches in education. As NLP continues to advance, we aspire that our work will encourage other researchers to explore how generative AI can be integrated into educational applications to benefit teachers and students, while promoting responsible and ethical use.
For future work, we will investigate students’ perceptions of peer reviews from different sources: their peers, peers using Reviewriter, and entirely AI-generated reviews. We will collect ratings and feedback from students who receive these reviews and compare the relevance, quality, and usefulness of the texts generated from each source. Additionally, we aim to integrate Reviewriter into
the university’s existing peer review system, enabling widespread adoption among students across various courses. By incorporating AI-generated instructions into routine peer reviews, we can examine the long-term impact on student’s writing skills, critical thinking abilities, and overall academic performance. To enhance the relevance of the AI-generated instructions in Reviewriter, we will refine the algorithms and models based on feedback from our evaluation participants. Our iterative development process will involve incorporating more contextual information, employing advanced NLP techniques, and leveraging user feedback to achieve higher accuracy and helpfulness in the AI-generated instructions.

heading: A Details on data pre-processing and models
text: A.1 Template questions asked students to write reviews which some students copied to their review text
• What do you see as the strengths of the fellow student’s solution?
• What do you see as weaknesses in the fellow student’s solution and how can they be addressed?
• What should be paid attention to in the revision of the solution?
• Provide concrete suggestions for improvement in this regard.
• Give concrete suggestions for improvement (constructive feedback).
• What should you pay attention to in the revision of the solution? Give concrete suggestions for improvement (constructive feedback).
A.2 Abbreviations and expansions
A.3 Hyperparameters for pretrained language models
A.4 Sample text generated by different language models

heading: B Details on evaluations
text: B.1 Pre-test questions asked during evaluation of Reviewriter
1. I like experimenting and trying out new technologies.
2. As a rule, I am hesitant when trying out new technologies.
3. In my circle of friends, I’m usually the first person to try new digital media / new technologies.
4. When I hear about new technologies I look for a way to experiment with them.
5. I have had experience writing reviews/feedback in the past.
B.2 Post-test questions asked during evaluation of Reviewriter
• Transition questions: How many times have you accepted Reviewriter’s recommendations?
• Technology Acceptance Model
1. Assuming the review writing assistance tool is available, the next time I want to write a review/feedback I would use it again.
2. With Reviewriter I can write reviews/feedback more effectively.
3. Learning to use Reviewriter was easy for me.
4. I find using Reviewriter useful for writing reviews/feedbacks.
5. I find Reviewriter easy to interact with. 6. It would be easy for me to become famil-
iar with Reviewriter. 7. Compared to other participants, I
think I wrote a very convincing review/feedback.
8. After using Reviewriter, my ability to write reviews/feedback has improved.
9. I’m sure I wrote a very insightful review/feedback.
10. I’m sure I wrote a very convincing review/feedback.
11. With Reviewriter I can write better reviews/ feedbacks.
12. I think I now know more about how to write well-structured, persuasive, and insightful reviews/feedbacks.
13. Assuming Reviewriter was available, the next time I write a review/feedback I would use it.
14. After using Reviewriter, my ability to pay attention to the different parts of the review/feedback structure has improved.
• Evaluate student perceptions on the AIgenerated instructions
1. I expect Reviewriter will help me improve my ability to write well-structured reviews/feedbacks.
2. I assume Reviewriter would help me improve my ability to write compelling reviews/feedback.
3. I assume Reviewriter would help me improve my ability to write insightful reviews/feedback.
4. Interacting with the tool was fun and enjoyable for me.
5. I expect Reviewriter will help me improve my ability to write helpful reviews/feedback.
6. Interacting with the tool was exciting.
• Open-ended questions for qualitative feedback
1. How has Reviewriter impacted your writing process?
2. What did you particularly like about using Reviewriter?
3. Do you have any other ideas? 4. What could still be improved? 5. Have you used a writing support program
before (e.g. Grammarly)? 6. What is your field of study? 7. Please enter your gender. 8. Please indicate your mother tongue.
B.3 Clustered qualitative student feedback from the evaluation of Reviewriter
Source Text Original input
First of all, I can say that your first task was solved well. You have structured the environmental and company analysis well, written a good introduction, and related the individual facts to the various factors, and you have also worked out the excessively high production costs.
German GPT-2 generated instructions Furthermore, the problems were identified and their elaboration was well coordinated. However, the behaviour of this company is completely unnecessary and one could hardly judge it more. Unfortunately, you went into far too much detail about the individual points and did little about the environmental and company analysis. 2) What do you see as the weaknesses of the environmental and business analysis and how can these be remedied?1.written out above:However, with the exception of case-related, these are other generic terms that have a rather different linguistic form. BLOOM generated instructions
I have already recognized the second task and the reasoning is also understandable for you. Good luck for the future! Strengths) The project idea is cleanly and realistically implemented, as it is a
GerPT2large generated instructions You mastered the task very well. I have a few suggestions for improvement. First of all, you could have explained in the assignment what an environmental and company analysis actually is. I would also replace the word "company" with "firm" or something similar. You answered task two well, but you might have briefly explained the stakeholder approach, which was necessary to understand the task. Perhaps you could have changed the focus of the company analysis, but that is only a small point. In general you could
Table 6: Comparison of the sample text generated by different transformer-based pretrained language models with max_length=150 (we translated all text from German to English for the purpose of this paper).70

heading: Future Work
text: Good luck for the future! Strengths) The project idea is cleanly and realistically implemented, as it is a
GerPT2large generated instructions You mastered the task very well. I have a few suggestions for improvement. First of all, you could have explained in the assignment what an environmental and company analysis actually is. I would also replace the word "company" with "firm" or something similar. You answered task two well, but you might have briefly explained the stakeholder approach, which was necessary to understand the task. Perhaps you could have changed the focus of the company analysis, but that is only a small point. In general you could
Table 6: Comparison of the sample text generated by different transformer-based pretrained language models with max_length=150 (we translated all text from German to English for the purpose of this paper). 70