Augmenting the Author: Exploring the Potential of AI Collaboration in
Academic Writing
JOSEPH TU∗ , Stratford School of Interaction Design and Business, University of Waterloo, Canada
HILDA HADAN∗ , Stratford School of Interaction Design and Business, University of Waterloo, Canada
DERRICK M. WANG∗ , Stratford School of Interaction Design and Business, University of Waterloo, Canada
SABRINA A. SGANDURRA, Stratford School of Interaction Design and Business, University of Waterloo, Canada
REZA HADI MOGAVI, Stratford School of Interaction Design and Business, University of Waterloo, Canada
LENNART E. NACKE∗ , Stratford School of Interaction Design and Business, University of Waterloo, CanadaThis workshop paper presents a critical examination of the integration of Generative AI (Gen AI) into the academic writing process,focusing on the use of AI as a collaborative tool. It contrasts the performance and interaction of two AI models, Gemini and ChatGPT,
through a collaborative inquiry approach where researchers engage in facilitated sessions to design prompts that elicit specific AI
responses for crafting research outlines. This case study highlights the importance of prompt design, output analysis, and recognizing
the AI’s limitations to ensure responsible and effective AI integration in scholarly work. Preliminary findings suggest that prompt
variation significantly affects output quality and reveals distinct capabilities and constraints of each model. The paper contributes tothe field of Human-Computer Interaction by exploring effective prompt strategies and providing a comparative analysis of Gen AI
models, ultimately aiming to enhance AI-assisted academic writing and prompt a deeper dialogue within the HCI community.
CCS Concepts: • Computing methodologies → Artificial intelligence; • Human-centered computing → Human computerinteraction (HCI).
Additional Key Words and Phrases: Generative AI, AI, Natural Language Processing, HCI Research Tools

1

INTRODUCTION

In the realm of scholarly investigation, traversing the preliminary stages of drafting a research manuscript can posechallenges [14]. Some of the most common struggles when writing research papers are the tight schedule, lack ofinspiration (for writing), and the complex process of organizing ideas and creating an outline [2, 9]. This issue isparticularly salient for early career academics and individuals whose first language is not English [7, 16]. We ask
ourselves, could Artificial Intelligence (AI) help?
The purpose of our workshop paper is to explore the potential of using AI to address challenges faced by the academic
research community. Through an exploratory research approach, we aim to gather intellectual feedback and initiate a
dialogue within the HCI community. Instead of presenting a traditional or formal research paper, our goal is to open
a discussion on this topic. We will be sharing our combined reflections on the use of AI, focusing on two popular
Generative AI models - ChatGPT1 and Gemini2 .
However, the application of AI as a writing assistant is often impeded by a dearth of transparency in researchers’
access and utilization [4, 12]. This lack of transparency raises concerns about the reliability and credibility of AIgenerated text [1, 3]. Hence, comparative analysis between ChatGPT and Gemini as writing assistants for research
draft outlines becomes crucial in understanding their capabilities and potential limitations.
Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s).

∗ Authors are also affiliated with the Department of Systems Design Engineering, University of Waterloo

1 ChatGPT. https://chat.openai.com/share/bf65d447-70c8-4117-87ef-f258265e3e43
2 Gemini. https://gemini.google.com/share/259d4d8f69f8

GenAICHI: CHI 2024 Workshop on Generative AI and HCI

1

Augmenting the Author

At its core, our research emphasizes the importance of continual reflection and validation throughout the entire
process of interacting with Gen AI. This includes thoughtful, prompt design, comprehensive output analysis, and anawareness of potential limitations. We contend that prioritizing this process-oriented approach is crucial for responsibly
integrating Gen AI into academic research, leading to more accurate and reliable outcomes for researchers.
For this workshop paper, our discussion will revolve around two key aspects:• Exploring Effective Prompt Strategies: In this section, our attention is directed towards the realm of prompts
that researchers themselves generate (the authors), as well as the pivotal inquiries that guide the outputs of
Gen AI models, including ChatGPT and Gemini. Drawing upon the valuable insights acquired from our weekly
deliberations, we aim to distinguish and articulate efficient strategies for crafting prompts.
• A Comparative Perspective on Results: We compare two Gen AI models by giving them the same prompts
and asking them to generate outlines for a specific research topic. Our collaborative inquiry approach will
reveal the nuanced differences in their responses.
As there is growing research in prompt engineering, we aim to expand upon existing contributions, including recent
works from [6, 13, 15], by undertaking a thorough examination of the merits and drawbacks of different models in theirhandling of various prompt methods. Furthermore, we present a detailed account of the collaborative process withAI, providing the researcher’s perspective and promoting transparency and accountability in assessing the efficacy
and limitations of ChatGPT and Gemini as writing aids. We focus not on formal and statistical comparisons, as we
acknowledge potential subjective biases and confounding factors. Instead, we aim to offer a firsthand experience thatcan serve as a starting point for meaningful discussions within the HCI community, potentially leading to further andmore comprehensive investigations.
This position paper aims to make two significant contributions to the field of HCI. Firstly, it seeks to delineate
effective prompt engineering strategies that enhance the collaborative dynamic between human researchers and AI,
thereby optimizing the integration of AI in the research outline development process. Secondly, the paper aims toprovide a nuanced comparative analysis of the capabilities and limitations of two prevalent generative AI models,
ChatGPT and Gemini, grounded in real-world application scenarios that could serve as a foundational framework for
future research and development in AI-assisted academic writing.
2

RESEARCH APPROACH

We used a collaborative inquiry approach [5], focusing on facilitating structured and inclusive discussions and deliberations among ourselves as researchers; and culminating in us collectively reaching thoughtful judgments aligned with a
specific goal or purpose, such as formulating recommendations or crafting a research outline. This case study aimed toevaluate the strengths and limitations of large language models (LLM), particularly Gemini and ChatGPT. Researchers(4—6) participated in weekly sessions, utilizing Otter.Ai for transcription. Discussions focused on the strengths andlimitations of the models, guided by carefully crafted prompts. The strategic use of AI models in research outline
development was evaluated based on predefined criteria. AI-generated prompts further facilitated critical thinking andreflection (condensed for a workshop see Table 1).
3

PRELIMINARY ANALYSIS OF CHATGPT AND GEMINI: OUR COLLECTIVE REFLECTION.

Through a series of comparisons and observations, we identified key differences in their approach, information handling,
and potential applications within the research process. Exploring the capabilities of ChatGPT and Gemini in various tasks
GenAICHI: CHI 2024 Workshop on Generative AI and HCI

2

Augmenting the Author

Table 1. An overview of our collaborative inquiry approach
Approach
A. Weekly Collaborative
Inquiry Sessions

Description
Format and Frequency: Engaging in regular weekly collaborative inquiry sessions, initiated on January 19th (still on-going
as of this submission), where 4-6 researchers convene for one-hour discussions. These sessions provide a structured platform
for collective exploration and dialogue.
Transcription: We used Otter.Ai as a recording tool during these sessions, fostering collaborative transcription anddocumentation. This software not only ensures accuracy in capturing discussions but also facilitates easy reference andanalysis of key points for us.
B. Collaborative Inquiry Designed Prompts: The research design incorporates carefully crafted prompts. These prompts are strategically formulated
Prompts and Guiding to extract nuanced insights from the collective group, emphasizing the strengths and limitations of each AI model.
Questions
Guidance in Collaborative Deliberations: These prompts play a pivotal role in steering collaborative discussions. By
focusing on specific aspects, the prompts encourage a detailed examination of the AI models, fostering a deeper understandingof their capabilities in the context of outlines for research papers.
C. Collaborative Re- Strategic Utilization of AI Models: At the core of the collaborative inquiry process is the strategic use of both Gemini andsearch Outline Develop- ChatGPT in generating research outlines. We actively explore how these AI models collectively structure and organize theirmentideas coherently, aiming to streamline the initial stages of the research process.
Criteria for Collaborative Evaluation: The effectiveness of each AI model in research outline development is collaboratively
evaluated based on predefined criteria. These criteria include the coherence of the generated outlines, alignment with research
objectives, and overall recommendations and critical questions planned for the research phase.
D. Collaborative Critical Collaborative Process of AI-Generated Prompts: To enhance critical thinking among the research team. We asked
Thinking Facilitation
the tools to provide us with critical thinking questions and suggestions to enhance the writing process. These prompts
are specifically designed to stimulate collective reflection and provoke thoughtful analysis of the research topics underconsideration.
Objectives of Collaborative Analysis: The deliberate facilitation of critical thinking through AI prompts aims to assess
the intellectual stimulation provided by Gemini and ChatGPT. It also guides researchers collaboratively in refining theirresearch objectives through a more thoughtful and analytical approach.
E. Collaborative Perspec- Individual Insights and Their Reflections: The research methodology includes individual insights as a means to gain
tive Elicitation
diverse perspectives on Gemini and ChatGPT. These reflections serve as a targeted approach to understanding how eachresearcher collectively perceives the strengths and limitations of the respective AI models.

prompts a critical examination of their strengths and limitations within the context of research outline development.
While ChatGPT consistently fulfills tasks based on explicit instructions, concerns arise due to its tendency to fabricate
responses, raising questions about the reliability of generated content [11]. We often noted that the model’s word-byword rephrasing approach, coupled with a reluctance to restructure sentences autonomously, emphasizes its dependence
on precise directives. Both models occasionally struggle with LaTeX formatting, which introduces complexities that may
impact their usability in crafting research outlines (at the time of this submission). On the other hand, Gemini proves
to be a dynamic counterpart, not only completing tasks but also providing valuable suggestions and rationales. Itsability to guide users through reference searches enhances the reliability of information retrieval. Gemini’s capacity tooffer multiple word choice suggestions, each accompanied by detailed explanations, adds sophistication to its linguistic
capabilities. In tasks requiring paragraph improvement based on references, both models demonstrate a dependence
on explicit prompts as outlined by Lo [10], highlighting the importance of precise instruction in research outline
development.
In terms of articulating the potential of generative AI for research outlines, Gemini distinguishes itself with comprehensive summaries that include self-promotion. In contrast, ChatGPT tends to provide more straightforward summaries
without articulating into additional insights. We also identified ChatGPT’s potential limitations in discerning the best
method or meaning, and found its potential deficiency in common sense. This suggest that enhancing ChatGPT’s
performance could be achieved through greater customization using its customization panel, potentially infusing aform of common sense into its responses, particularly when generating content for research outlines.
In addition, Gemini impresses with its proactive and initiative-taking nature. It demonstrates a keen understanding of
the broader context and consistently suggests new phrasings or alternative options without explicit requests, showcasing
its potential to aid research outline development without waiting for specific directives. A notable distinction lies
GenAICHI: CHI 2024 Workshop on Generative AI and HCI

3

Augmenting the Author

Table 2. A preliminary summary of our collective reflection findings for ChatGPT and Gemini
Category
Strengths

ChatGPT Preliminary Findings
Consistent adherence to explicit instructions, precision, andmethodical approach.

Limitations

Tendency to fabricate responses, reliance on word-by-word
rephrasing, occasional struggle with LaTeX formatting.

Articulating AI Poten- More straightforward outlines.
tial
Handling Mistakes
May fabricate information, raising concerns about reliability.

Gemini Preliminary Findings
Dynamic, provides additional suggestions and rationales. Proficient in guiding through reference searches. Offers multiple
word choice suggestions with detailed explanations.
Reliance on explicit prompts in tasks requiring paragraph improvement based on a reference. Limited customization options
compared to ChatGPT.
Excels in comprehensive summaries with self-promotion.

Prioritizes transparency, explains confusing sources, andguides on independent information retrieval.
Human-AI Collabora- Ongoing exploration and adaptation required for maximizing Ongoing exploration and adaptation are required to maximizetionbenefits and mitigating limitations in the evolving dynamics. benefits and mitigate evolving dynamics’ limitations.
Common Sense Inte- Deficiency in common sense could be addressed through cus- Undetermined
grationtomization using its panel.
Proactivity
Undetermined
Demonstrates initiative and understanding of broader context,
delivering optimal outcomes without waiting for specific directives.
AI’s Role in Education Reflects on AI’s potential evolution into a personalized learning tool that adapts to individual learning styles.

in how these models handle mistakes. While ChatGPT at times resorts to fabricating information, raising concerns
about reliability and inadvertent alterations of intended meaning, Gemini prioritizes being helpful over maintaining
an appearance of infallibility. Gemini transparently addresses limitations, enhancing its credibility and transparency
compared to ChatGPT, particularly in the context of research outline development.
Researcher’s Antidote on Certain Concerns. Participant 1 (P1) quotes: “This worries me as a reviewer, am I ahuman validating the author’s work or AI’s work?” One notable concern is the potential shift in the role of reviewers,
who may unintentionally become the primary validators of AI-generated content rather than the authors themselves.
The human touch, creativity, and critical thinking are essential components of writing. If AI takes a prominent rolein generating content, there is a risk of diminishing these human qualities. We suggest authors utilizing AI in theirwriting should prioritize transparency by clearly disclosing the involvement of AI tools in the creation process.
P6 quotes: “After using these tools for a while, I have a bias towards a certain writing style. I can tell that AI writes thisor that, but what happens if this is just their writing style? Personally, English as a second language, I was taught to write ina formal construct.” As we explore the potential of AI writing tools, we acknowledge concerns surrounding the negative
perception of their generated styles, particularly within academia. We understand the apprehension that might stemfrom unfamiliarity with this format, potentially leading to bias against its lack of established conventions. Additionally,
we recognize the fears surrounding the potential for both quality concerns and ethical misuse of these tools. These
concerns could translate into various challenges. We might encounter resistance towards adopting AI writing tools,
creating an uneven playing field for those with access. This further highlights the need for clear guidelines to ensure
responsible and ethical usage within academic settings [8]. While addressing these concerns is crucial for successful
integration, we believe that continuous development and increasing familiarity with AI writing might eventually
alleviate the initial negative connotations.
ACKNOWLEDGMENTS
This workshop was supported by the SSHRC INSIGHT Grant (grant number:435-2022-0476), NSERC Discovery Grant(grant number: RGPIN-2023-03705) and CFI John R. Evans Leaders Fund or CFI JELF (grant number: 41844). Special
thanks to the members of the Games Institute at the University of Waterloo for all their support.
GenAICHI: CHI 2024 Workshop on Generative AI and HCI

4

Augmenting the Author

REFERENCES[1] Muhammad Yousuf Ali, Salaman Bin Naeem, and Rubina Bhatti. 2020. Artificial intelligence tools and perspectives of university librarians: An
overview. Business Information Review 37, 3 (2020), 116–124.
[2] Manal AlMarwani. 2020. Academic writing: Challenges and potential solutions. Arab World English Journal (AWEJ) Special Issue on CALL 6 (2020).
[3] David Baidoo-Anu and Leticia Owusu Ansah. 2023. Education in the era of generative artificial intelligence (AI): Understanding the potential
benefits of ChatGPT in promoting teaching and learning. Journal of AI 7, 1 (2023), 52–62.
[4] Nagadivya Balasubramaniam, Marjo Kauppinen, Antti Rannisto, Kari Hiekkanen, and Sari Kujala. 2023. Transparency and explainability of AI
systems: From ethical guidelines to requirements. Information and Software Technology 159 (2023), 107197.
[5] John N Bray. 2000. Collaborative inquiry in practice: Action, reflection, and making meaning. Sage.
[6] Sabit Ekin. 2023. Prompt engineering for ChatGPT: A quick guide to techniques, tips, and best practices. Authorea Preprints (2023).
[7] Tira Nur Fitria. 2021. Grammarly as AI-powered English writing assistant: Students’ alternative for writing English. Metathesis: Journal of English
Language, Literature, and Teaching 5, 1 (2021), 65–78.
[8] Reza Hadi Mogavi, Chao Deng, Justin Juho Kim, Pengyuan Zhou, Young D. Kwon, Ahmed Hosny Saleh Metwally, Ahmed Tlili, Simone Bassanelli,
Antonio Bucchiarone, Sujit Gujar, Lennart E. Nacke, and Pan Hui. 2024. ChatGPT in education: A blessing or a curse? A qualitative study exploring
early adopters’ utilization and perceptions. Computers in Human Behavior: Artificial Humans 2, 1 (Jan. 2024), 100027.
[9] Mary M Juzwik, Svjetlana Curcic, Kimberly Wolbers, Kathleen D Moxley, Lisa M Dimling, and Rebecca K Shankland. 2006. Writing into the 21st
century: An overview of research on writing, 1999 to 2004. Written Communication 23, 4 (2006), 451–476.
[10] Leo S Lo. 2023. The CLEAR path: A framework for enhancing information literacy through prompt engineering. The Journal of Academic
Librarianship 49, 4 (2023), 102720.
[11] OpenAI. 2023. Gpt-4 technical report. Technical Report. OpenAI. arXiv preprint arXiv:2303.08774.
[12] Arun Rai. 2020. Explainable AI: From black box to glass box. Journal of the Academy of Marketing Science 48 (2020), 137–141.
[13] Alberto D Rodriguez, Katherine R Dearstyne, and Jane Cleland-Huang. 2023. Prompts Matter: Insights and Strategies for Prompt Engineering in
Automated Software Traceability. In 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW). IEEE, 455–464.
[14] Michael Sedlmair, Miriah Meyer, and Tamara Munzner. 2012. Design study methodology: Reflections from the trenches and the stacks. IEEE
transactions on visualization and computer graphics 18, 12 (2012), 2431–2440.
[15] Mo Wang, Minjuan Wang, Xin Xu, Lanqing Yang, Dunbo Cai, and Minghao Yin. 2023. Unleashing ChatGPT’s Power: A Case Study on Optimizing
Information Retrieval in Flipped Classrooms via Prompt Engineering. IEEE Transactions on Learning Technologies (2023).
[16] Xin Zhao. 2023. Leveraging artificial intelligence (AI) technology for English writing: Introducing wordtune as a digital writing assistant for EFL
writers. RELC Journal 54, 3 (2023), 890–894.

Received 26 February 2024; accepted 13 March 2024

GenAICHI: CHI 2024 Workshop on Generative AI and HCI

5