# Chatbots in Collaborative Settings and their Impact on Virtual Teamwork

Clélie Amiot, François Charoy, Jérôme Dinet

► To cite this version:

Clélie Amiot, François Charoy, Jérôme Dinet. Chatbots in Collaborative Settings and their Impact on Virtual Teamwork. Proceedings of the ACM on Human-Computer Interaction, 2025, 9 (2), pp.CSCW047. 10.1145/3710945. hal-04841009

HAL Id: hal-04841009

https://hal.science/hal-04841009v1

Submitted on 7 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Chatbots in Collaborative Settings and their Impact on Virtual Teamwork

CLÉLIE AMIOT, Université de Lorraine, CNRS, Inria, Loria, France

FRANÇOIS CHAROY, Université de Lorraine, CNRS, Inria, Loria, France

JÉRÔME DINET, Université de Lorraine, France

Chatbots have emerged as a powerful tool for collaborative teamwork. In this paper, we investigate the influence of chatbots on virtual teamwork within the context of a collaborative online activity. To assess chatbots' impact on group dynamics and performance, we designed a novel collaborative activity with an associated online platform and a custom chatbot assistant. We recruited 72 participants divided into four-person teams, with the teams split into four conditions depending on the chatbot's assistance strategy (none, by private chat, by group chat, or both). We found that chatbot assistance improved participants' performance and reduced response times to information requests during the collaborative activity. This improvement was more pronounced for teams in the group and both chats conditions, while participants in the private chat condition reported significantly higher perceived effort. Furthermore, we observed a significant correlation between the perceived quality of the chatbot's communication skills and the cognitive workload and time pressure experienced by the participants. Our findings contribute to the field of Computer-Supported Cooperative Work by proposing a new design of experimental activity to study virtual collaboration, providing insights that can inform the design of future collaboration tools and adding to our broader understanding of human-AI synergy in group settings.

CCS Concepts: • **Human-centered computing** → **Computer supported cooperative work**; Natural language interfaces; *Laboratory experiments*.

Additional Key Words and Phrases: collaboration, chatbot, virtual teams, experimental study

## ACM Reference Format:

Clélie Amiot, François Charoy, and Jérôme Dinet. 2025. Chatbots in Collaborative Settings and their Impact on Virtual Teamwork. In *CSCW 2025: ACM SIGCHI Conference on Computer-Supported Cooperative Work & Social Computing, October 18-22, 2025, Bergen, Norway*. ACM, New York, NY, USA, 19 pages. https://doi.org/XXXXXXX.XXXXXXX

## 1 INTRODUCTION

The emergence of ChatGPT [13] and the rapid advancement of generative AI have propelled chatbots, or conversational agents, into the forefront of our daily lives. AI-powered tools are set up to revolutionize various aspects of our routines, from boosting productivity and simplifying information access to supporting decision-making across diverse domains. A growing body of research [4, 23] examines the multifaceted interactions between humans and chatbots across various contexts in one-on-one and group settings. These investigations highlight the significant impact that chatbots already have on human work practices and the importance of continuing to explore the possibilities of human-conversational agent teaming.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

CSCW 2025, October 18-22, 2025, Bergen, Norway

© 2025 Association for Computing Machinery.

ACM ISBN 978-1-4503-XXXX-X/18/06...$15.00

https://doi.org/XXXXXXX.XXXXXXX
Despite the growing use of chatbots in collaborative settings, research on their application in group dynamics remains relatively unexplored. As recent surveys and research agendas highlight [4, 16], this gap in understanding is concerning given the growing evidence that such interactions will become increasingly commonplace. Chatbots are already being integrated into group chats, educational settings, and decision-making processes, highlighting the need to investigate their impact on task-oriented group activities where their primary goal is to facilitate collaboration and goal achievement.

Our study seeks to address this knowledge gap by focusing on two critical research questions:

* How is a chatbot integrated into a group activity utilized and perceived by its members?

* How does a chatbot assistant influence team performance and individual cognitive load during collaborative activities?

We aim to shed light on the dynamics of human-chatbot interactions in collaborative contexts, paving the way for the development of more effective and user-centric chatbot tools for group settings.

To this end, we introduce a new experimental methodology and online platform designed to simulate real-world collaborative scenarios. The activity involves teams of four participants working together to solve a series of puzzle modules. These modules are solvable through actions such as button clicks or numeric entries and are designed to require effective communication and information exchange among team members. To mirror modern work environments, participants have access to a group chat for team-wide discussions and individual messaging channels for private conversations. Our custom-built chatbot, COCCO, is integrated into the platform to offer assistance. It has been designed to understand and retain task-relevant information shared by participants in group and private messages.

Our experiment investigates four distinct conditions: a control group without COCCO's presence, a group chat-only condition where COCCO solely operates within the team's group chat, a private chat-only condition where COCCO is confined to individual messaging channels, and a hybrid condition where COCCO actively participates in both group and private chats.

Through this multi-faceted approach, we aim to uncover how chatbot interventions affect team dynamics, communication patterns, and overall performance. We expect that the design of this new experimental activity and our findings in this study will inform the design of future collaborative tools and deepen our understanding of the complex interplay between humans and AI in group settings.

In the following sections, we will analyze the relevant literature, provide a detailed description of the collaborative task and COCCO's capabilities, outline our experimental methodology, and present the insights derived from our study.

## 2 CONTEXT

There have been an increasing number of studies on chatbot interactions in recent years following technological improvements and broader use [23]. However, the research on group interaction with chatbots needs to be developed as most studies are only concerned with dyadic interactions [16]. Those studies do not readily apply to group interactions with chatbots. In group conversations, chatbots aren't the sole participants; most messages are likely between humans and not meant for the chatbot. Chatbots in these settings primarily support the conversation, either by mediating or reacting to specific messages, and shouldn't hinder teamwork with excessive or irrelevant messages that disrupt the conversation's flow.

As identified by Zheng et al. [23], polyadic interactions pose different challenges to chatbots than dyadic ones, such as managing unequal participation and lack of engagement, striking a
balance between visibility and unobtrusiveness or preserving privacy, and developing new evalua-
tion metrics. This is also highlighted by Følstad et al.'s research agenda [4] which identifies the
following research challenges for collaborative chatbots: developing new models of conversational
collaboration involving agents and identifying the importance of a chatbot's social role and level of
reactivity in a collaboration.

In the literature, we identified two main subjects of interest for research on chatbots in polyadic
interaction: chatbot-assisted collaborative learning and bots in work collaboration tools. We ad-
ditionally reviewed existing experimental collaborative activities aimed at understanding group
dynamics.

## 2.1 Chatbot for collaborative learning and group discussions

One of the main research interests of studies on chatbot interactions with groups is chatbots acting as debate moderators and structuring group discussions [23]. Chatbots have been shown to help with balancing participation between participants [2, 6], argument quality [19] as well as team cohesion [1, 6].

Kim et al. study [6] presents GroupfeedBot, a chatbot that facilitates group discussions by structuring the conversation, encouraging even participation between members, and organizing group participants' opinions. Groups of 4 or 5 participants completed four tasks, two assisted by a simple chatbot and two assisted by GroupfeedBot. On tasks assisted by Groupfeedbot, groups showed increased opinion diversity and improved evenly distributed participation. Another study by Kim et al. [7] with the chatbot DebateBot also showed an increased discussion quality and balanced participation, thanks to the chatbot facilitation.

Benke [1] investigated how to leverage chatbots' capabilities to improve team emotional manage-
ment and communication. Their prototype showed improved team cohesion and communication
thanks to the chatbot social augmentation.

The delivery method of the assistance is also essential. Indeed, when interacting with a group, it
matters whether advice and directions are given to individuals or the group and whether only the
recipient or everyone can see it. Tegos et al. pilot study [19] presents their chatbot MentorChat,
which was made to support collaborative learning. They looked at differences in effectiveness
between directly prompting the weaker member of a group of two or three students or addressing
the group as a whole. Using discourse analysis, they found that directly prompting a specific student
resulted in considerably more explicitly reasoned responses. However, in the post-task survey, the
students preferred the undirected intervention strategy to the direct one.

Do et al. [2] studied possible communication strategies for chatbots to balance participation in
group ideation. The chatbot would prompt participants either as a group or individually, publicly
or privately, or ask another participant privately to prompt the least contributing member. The
private messages were displayed in the chat but only visible to the concerned participant. The
study found that participants preferred and were more reactive to being prompted individually and
privately and that the peer-mediated strategy was viewed as less intrusive and embarrassing than
other strategies.

However, these conclusions are less relevant for collaborations that do not involve structured
debates and where conversation is not the main objective. In many workplaces, balanced participa-
tion often takes a backseat to measurable performance metrics such as speed, work quality, and
error rates.
## 2.2 Chatbots integrated into work collaboration tools

Another domain of interest for studies on chatbots interaction with groups is chatbots integrated into work communication tools, such as Slack or Google Chat. Those bots have a wide range of capabilities and uses, from Git notifications to new team members onboarding [17].

Wang et al. [22] collected a dataset of instant messaging usage in a multinational IT company during the COVID-19 pandemic that provide some insights on the link between team performance and slackbot usage. They found a slight link between the number of bot messages and the team's success, but could not exclude the possibility of it being a direct consequence of the team's Git activity.

Laitinen et al. [8] looked at the impact of slackbot messages on a journalistic team, with interaction ranging from greetings and acclamations to work-related tasks like developing story ideas or coordinating work distribution and planning. They concluded that bots can facilitate team relational communication by increasing the number of socioemotional and personal interactions.

Lebeuf et al. [9] identify areas where software developers can leverage chatbots to mitigate friction in their collaborative work environment. They provide examples of chatbots implemented into collaborative tools to reduce information overload while helping developers maintain awareness of their teammates' expertise and current tasks and their team's processes and goals.

To investigate the possibility of using chatbots for task management, Toxtli et al. [20] designed and tested a chatbot for Microsoft Teams capable of sending reactive and proactive messages. After testing it with eight teams of information workers, they observed that the chatbot was sometimes used for self-communication and was challenged when multiple conversations would take place at the same time or when dealing with ambiguous messages lacking context.

Previous studies on chatbots in collaborative tools provide valuable insights but often depend on access to existing teams or datasets where chatbots are already integrated—resources we did not have for this study. Instead, we chose a more adaptable experimental approach that could be customized to address specific research questions without being limited by pre-existing team dynamics or work constraints.

## 2.3 Existing experimental collaborative activities

The previously cited studies are often based on observational data from existing teams. Conversely, there is a lack of adequate experimental platforms made to study group interactions during a collaborative task in a controlled environment. Below are the ones we identified.

The NeoCITIES simulation [12], an online activity simulating emergency dispatching in a virtual city space. It was developed to research decision-making, information overload, and teamwork. Schelbe et al. paper [15] implemented the simulation to investigate team cognition within human-agent teams of three. Humans collaborating with agents perceived significantly higher team performance, but lower team cognition than with human teammates. Additionally, humans with only agent teammates had significantly lower trust in them than when they had a human and an agent teammate.

The NeoCITIES simulation engine was then used in subsequent activities like idsNETS and TeamNETS. IdsNETS [11] aims to simulate cybersecurity intrusions to study analysts' situation awareness. In this activity, players need to monitor IDS alerts (intrusion detection system), categorize, and report. TeamNETS [10] cybersecurity simulation, teams of three each responsible for either policy management, intrusion detection, or malicious software detection. They must identify threats and either accurately categorize them or transfer them to a better-suited teammate.

Another example of an experimental activity designed to study group dynamics is the StrikeCom game [21]. It is defined as a C3ISR (Command, Control, Communication, Intelligence, Surveillance,
Reconnaissance) task designed to examine the development of shared awareness and communication in distributed groups". The game consists of a map divided into a grid of cells and is separated into multiple phases. Participants place assets on the cells to search for a target. After multiple search rounds, the group decides on which cells to strike. The game makes participants use different communication mediums, such as chat, audio, video, or in-person.

Finally, the mixed-reality game AtomicOrchid [3] was designed to study human-agent teaming using a disaster response scenario. In this game, players assume the role of field responders rescuing targets after a radioactive explosion. They navigate in real space while being guided by players in the headquarters who are the only ones able to see where the radioactive cloud is. This game was used in Ramchurn et al. study [14] which used supervised virtual agents to dispatch the players. They suggest that the adaptability of agents to users' requests is a crucial factor for their acceptability.

These experimental activities are largely designed around command-and-control frameworks with high-stakes objectives, such as emergency response or cybersecurity. Such intense, mission-critical settings are not readily applicable to studies on more routine workplace collaboration, where goals are typically less urgent, and workflows more flexible. These activities are also challenging to adapt for integrating a chatbot, which is our main research focus. This gap highlights the need for experimental platforms better suited for exploring chatbot interactions in everyday collaborative tasks.

## 2.4 Conclusion

In conclusion, while significant progress has been made in understanding chatbot interactions, especially in dyadic settings, there remains a considerable gap in knowledge about chatbot integration within group interactions. The dynamics of group conversations present unique challenges that require chatbots to operate as supportive participants rather than primary interlocutors. Our main goal is to try to understand the influence of chatbots on collaborative activities to inform their use in future settings.

## 3 METHODOLOGY

To test the influence of chatbots on collaborative activities we needed a collaborative activity that a chatbot would be able to parse and participate in. However, platforms used in the literature for studies on group interactions are often concerned with debate and learning as the sole activity or predominantly adhere to command and control structures, while requiring specific knowledge and deviating from average workplace dynamics. For this reason, we designed a new experimental activity made to investigate virtual group interactions with a scenario closer to workplace dynamics. Our focus was on creating an activity that could easily integrate a chatbot, be adaptable to a variety of protocols (duration, team size, difficulty, asymmetry of information, balance between collaboration and competition), and provide a context-independent setting that does not require participant training.

We additionally implemented a chatbot named COCCO specifically for the activity, whose primary function is to help participants by handling and relaying task-relevant information.

Finally, we designed two surveys, taken before and after the activity. The pre-activity survey collects the participants' informed consent and their demographic information, and the post-activity survey records their perception of the activity and cognitive workload using the NASA-TLX [5].

### 3.1 Collaborative activity and platform

Our experimental platform is a web-based collaborative game developed in JavaScript and hosted on a VPS. The interface comprises 16 distinct modules framed by a chat box to the left and textual
<figure><img src="image_2.png" /><figcaption>Fig. 1. Platform interface. On the left panel: the chat interface, with different tabs for all the chats available to the participant (group chat, private chat with COCCO, and three private chats with teammates). On the middle panel: the time elapsed since the start of the experiment, the participants’ pseudos, the number of modules solved, the number of errors made by the participant, and the grid of 16 modules. On the right panel: the 8 hints given to the participant at random, grouped by module type.</figcaption></figure>

hints to the right, as depicted in Figure 1. In this study, each participant is assigned 16 modules to solve, relying on hints and assistance from their teammates to complete them. The objective is to solve all modules as quickly as possible while minimizing errors. This task structure encourages collaboration, as participants must communicate effectively and support each other to optimize both speed and accuracy. To communicate, they have access to a group channel where everyone in the team can chat, and private channels where they can send direct messages to each of their teammates. The experiment ends when all participants have solved all of their modules. The platform logged every message sent by the participants and COCCO, as well as every click on the modules.

Modules are characterized by their color (red, orange, yellow, green, blue, or purple), type (checkboxes, menu, field, buttons, or grid), and associated number (ranging from 1 to 16). All modules of the same type and color have the same solution. Participants receive hints that directly or indirectly help them solve modules of a particular type and color. While some hints will be relevant to their own modules, others will benefit their teammates. Likewise, certain hints essential for solving a participant’s modules will be provided to their teammates. For each team, the modules and corresponding solutions are randomly generated, adhering to the following rules:

* Each participant has 4 field modules and 3 modules of each other type.

* No participant has two modules with the same type and color.

* No participants in the same team have a module with the same type, color, and number.

* Each participant has eight hints assigned randomly. A hint for each module type and color should be assigned to at least one participant. (There are 6x5=30 combinations of module types and colors and 8x4=32 hints assigned in a team of four, resulting in only two duplicated hints.)
<figure><img src="image_3.png" /><figcaption>Fig. 2. Examples of each module type</figcaption></figure>

The modules were designed using standard HTML elements (checkboxes, radio buttons, etc.) to reduce compatibility issues and make them easy for participants to interact with. The modules can be solved using only information available on the platform: the modules' types, colors, and numbers, and the participants' pseudos. This ensures that any participant, regardless of their background knowledge or experience, can take part in the activity. These concepts are also distinct and simple enough for a chatbot to match to predefined parameters and intents with minimal errors.

Using these simple concepts, we tried to create as much complexity in the collaboration and need for different communication strategies as possible. This was achieved by making some of the solutions dependent on additional information from one or more teammates.

Below is a breakdown of the five module types and how they are solved:

* Buttons module (2a): solved by clicking three buttons in the correct order. The hint is the solution. For example: "Purple buttons: right, center, and left".

* Grid module (2b): solved by clicking the correct button in a 3 by 3 grid. The hint is the solution. For example: "Green grid: button on the bottom right".

* Checkboxes module (2c): solved by checking the colors of the modules of a given number of all participants. The hint requires asking all teammates for the color of one of their modules. For example: "Yellow checkboxes: colors of the modules number 9 of all players". The corresponding solution would be checking "vert" (green), "orange", and "bleu" (blue).

* Menu module (2d): solved by choosing in a dropdown menu the pseudo of the teammate with a module of a given type, color, and number. The hint requires asking all teammates if they have a specific module (or deduct the answer by elimination). For example: "Red menu: player with a red grid module at number 1". The corresponding solution would be choosing "Llama" in the dropdown menu.

* Field module (2e): solved by entering the number of a specific teammate's module with a given type and color. The hint requires asking a teammate for the number of one of his modules. For example: "Blue field: player Owl's orange checkboxes module's number". The corresponding solution would be entering "14" in the field.

The different module types enable us to propose different difficulty levels regarding information sharing and encourage effective group communication. Buttons and grid modules can be solved
just by knowing their hints. Checkboxes modules can be solved by polling every teammate and remembering multiple colors. Menu modules can be solved by asking some or all of their teammates until the solution can be deduced. Finally, field modules can be solved by contacting a specific teammate. This results in a cooperative game where participants must collaborate to solve all the modules from each board to achieve their common goal.

We conducted multiple pre-tests to ensure that the experiment platform was functional and the instructions easy to understand.

## 3.2 COCCO the chatbot

3.2.1 *Design.* We developed our chatbot COCCO to be able to parse participants’ messages for task-relevant information, memorize it, and communicate it when requested.

To extract information from the messages (hints, solutions, module characteristics), we used Google Dialogflow<fnref n="1" />. The intents are defined and trained on the pre-tests’ logs. We looked at how participants exchanged information and talked with one another when doing the activity without COCCO, then created intents that would allow it to learn new information from the conversation and recognize requests for information. We then conducted additional pre-tests where COCCO was implemented in group chat and private chat to fix problems and tune its training data. The list and descriptions of the 29 implemented intents (4 conversational, 11 for new information, and 4 for requests) can be found in appendix A.

During the activity, the recognized intent dictates COCCO’s behavior. If the message includes new information, COCCO updates its database. If the message is an information request, COCCO checks its database and shares the requested information if it has it. All of COCCO’s messages to the participants use pre-written templates.

Following pre-tests done with COCCO, we added a check before saving new information in COCCO’s database to prevent it from repeating wrong information. This is a workaround for the need of the experiment as we consider miscommunication repair beyond the scope of this paper. However, even with the check, COCCO’s intent recognition is not perfect. It can still intervene at the wrong moment when it misunderstands a participant’s need. This made it similar to the participants’ messages, who rarely made outright mistakes (1.78% of answers) but sometimes would make ambiguous requests or give information relating to no clear subject.

3.2.2 *Communication strategies.* For our experiment, COCCO is designed to use different communication strategies: it can be present in the group chat, the private chat, or both.

When in the group chat, COCCO only intervenes when it recognizes an information request for which it has the correct information.

When in a private chat with a single participant, COCCO intervened similarly to when it was in the group chat. Additionally, when missing requested information, it could ask for it from other participants at the participant’s request. This prevented unnecessary clutter in the group chat and context confusion for COCCO, especially as this feature would be superfluous in a group chat where other participants could already see the information request. Moreover, in the private chat, COCCO had a small contextual memory to keep track of which question was being answered or which module a new message was about.

For some teams, COCCO was present in the group chat and the private chat. This allows it to use the information given to it in private chat for questions in the group chat and vice versa.

We provide examples of those different communication scenarios in figures 3, 4, and 5.

Figure 3 shows an interaction between participants with no intervention from COCCO, starting from player Fish’s request for the solution of purple grid modules to its answer two minutes later.

<footnote n="1">¹https://cloud.google.com/dialogflow</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 3. Excerpt from an interaction between participants with no assistance from COCCO (translated, timestamps in minutes and seconds)</figcaption></figure>

We can see that there is often cross-talk between participants and that their answers are not always clear or easy to understand without context. In this case, player Cat's last message indicates that the solution to the purple grid module is the same as the yellow grid module's solution, which is "upper center".

Figure 4 shows two interactions from different teams where COCCO provided assistance in the group chat. In scenario 4a, COCCO knows the solution to the modules thanks to previous interactions. This allows it to immediately respond to one of the two requests from player Owl (red field module). The other request (orange field module) is answered later by player Llama. We can additionally see that different participants, Owl and Turtle, request the same module solution in short succession.

In scenario 4b, player Sheep requests information for two modules (yellow and green field). COCCO only knows the hint for the yellow field module but not its solution. However, player Monkey, following COCCO's message, completes the hint by giving the necessary information (the color of his blue menu module). This sequence is repeated for the green field module's hint with players Owl and Bee and for the red field module's hint with COCCO and player Monkey.

Finally, scenario 5 shows an interaction where COCCO provides assistance in a private channel but does not have the requested information (otherwise, the interaction would be similar to those in a group chat). Instead, after confirming with player Koala, COCCO asks the other participants for the information (whether they have or do not have a purple field module at the number 13). Player Turtle states that they do not have this module, and player Monkey does not respond. When player Fox confirms that they have the module, COCCO informs player Koala.

## 3.3 Pre-activity and post-activity surveys

In addition to the collaborative activity, participants answered two surveys before and after the activity. The pre-activity survey collected participants' consent, demographic information, spoken
<figure><img src="image_5.png" /><figcaption>Fig. 4. Excerpt from two groups’ interaction with assistance from COCCO in the group chat (translated, timestamps in minutes and seconds)</figcaption></figure>

languages, and perceived ease with instant messaging tools. The post-activity survey collected answers to the NASA Task Load Index (NASA-TLX) [5], an assessment tool for subjective workload. A general score is calculated using six weighted factors (mental demand, physical demand, temporal demand, performance, effort, and frustration). We added a Likert scale on participants’ perception of the team’s performance and communication quality, and trust in the team. Participants not in the control group also evaluated COCCO’s communication quality and their trust in COCCO. Finally, all participants were invited to leave a comment about the experiment.

## 3.4 Experimental design

To study our chatbot interactions with a group, we designed three different communication strategies: group chat, private chat, and both chats. Those strategies are modeled on common workspace communication tools (Slack, Microsoft Teams) where some chatbots can be interacted with from any channel, some are only used to broadcast information in a public channel, or some have to be contacted by direct message [17, 22].

Participants were divided into teams of four according to their availability. Teams were then assigned to one of the four conditions: control, with COCCO in private chat, group chat, or both (Figure 6).

Each participant was assigned a random animal name (taken from a pre-defined list of ten) as a pseudo. The pseudo prevented participants from knowing the identities of their teammates and being influenced by possible previous familiarity or biases towards their names or genders.

Team size was determined based on preliminary test runs of the experiment. Teams of three made communication too streamlined with less cross-talk, and participants could deduce modules’
<figure><img src="image_6.png" /><figcaption>Fig. 5. Excerpt from a group interaction with assistance from COCCO in private chats (translated, timestamps in minutes and seconds)</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 6. Experimental steps</figcaption></figure>

solutions with less help from their teammates. On the other hand, teams of five lengthened the experiment too much, with a run lasting over one hour.
## 3.5 Participants

We recruited 72 participants, divided into 18 groups, for this experiment using mailing lists and word of mouth. Participants were fluent French speakers aged 17 to 68 years (m=28.6 years, sd=11.3); 39 female and 33 male.

Participants were instructed to log onto the platform at their group's scheduled time and use a confidential code communicated beforehand in email. Upon connecting, participants were presented with instructions for the activity and had to confirm their presence by sending a message in the chat. Once all four participants confirmed their presence, the interface would become interactable, and the activity could start.

All participants received a 10€ gift card after completing the activity. To encourage participants to perform to the best of their abilities, teams with better performance (time and number of errors) received a second gift card. Team performance was calculated using the slowest participant's time (to encourage teamwork) and the number of errors (to dissuade brute-forcing the tasks). The experiment was approved by our institutional ethics committee.

# 4 ANALYSIS AND RESULTS

The 18 groups produced a significant amount of data, from all the messages exchanged during the activity and the clicks on the modules to the survey answers. In the following section, we analyze the performance differences for each condition. We then examine how COCCO was used during the activity and how it was perceived in the post-activity survey.

## 4.1 Team performance

4.1.1 Module solving time. To gauge team performance, our initial metric was the duration participants took to resolve each of their respective modules (Figure 7). Participants who performed better could comprehend information and solve modules faster than others. However, this metric also reflects the overall team performance, as a quicker module resolution suggests efficient information dissemination within the team.

To compare solving times between conditions, we used a Kruskal-Wallis test. It revealed a significant difference in performance between the conditions (H=73.55, df=3, p<0.001). We conducted post-hoc pairwise comparisons using a Wilcoxon rank sum test (Holm correction) and found significant differences between all conditions, as presented in Table 1. Participants in the control condition had the longest median module solving time (≈21min), then the private chat condition (≈18min), the both chats condition (≈14min), and the group chat condition (≈12min).

<table><caption>Table 1. Comparison of module solving times between conditions using Wilcoxon rank sum test (Holm correction)</caption><thead><tr><th>Conditions</th><th>Control</th><th>Group chat</th><th>Private chat</th><th>Both chats</th></tr></thead><tbody><tr><td>Control condition (med=1265s)</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Group chat condition (med=729s)</td><td>p&lt;0.001***</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Private chat condition (med=1105s)</td><td>p=0.0354*</td><td>p&lt;0.001***</td><td>-</td><td>-</td></tr><tr><td>Both chats condition (med=840s)</td><td>p&lt;0.001***</td><td>p=0.0354*</td><td>p=0.0032**</td><td>-</td></tr></tbody></table>

4.1.2 Response delays. To ensure teams were sharing information quickly, we checked how long it took for a question to receive a response. We annotated all the messages where a participant asked for a hint or information and messages where they got a reply. Then, we calculated the delay
<figure><img src="image_8.png" /><figcaption>Fig. 7. Module solving times for each condition</figcaption></figure>

between the request and its answer. If a question did not get an answer, we did not include it in the data. This happened when a question was asked a second time before receiving an answer or when a participant deduced the answer independently.

A Kruskal-Wallis test revealed a significant difference in answer delays between conditions (H=61.71, df=3, p<0.001). We conducted post-hoc pairwise comparisons using a Wilcoxon rank sum test (Holm correction) and found significant differences between all the conditions, as presented in Table 2. Participants in the control condition had the longest median response delays (29s), then the private chat condition (23s), the group chat condition (17s), and the both chats condition (14s).

<table><caption>Table 2. Comparison of response delays between conditions using Wilcoxon rank sum test (Holm correction)</caption><thead><tr><th>Conditions</th><th>Control</th><th>Group chat</th><th>Private chat</th><th>Both chats</th></tr></thead><tbody><tr><td>Control condition (med=29s)</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Group chat condition (med=17s)</td><td>p&lt;0.001***</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Private chat condition (med=23s)</td><td>p=0.0049**</td><td>p=0.0028**</td><td>-</td><td>-</td></tr><tr><td>Both chats condition (med=14s)</td><td>p&lt;0.001***</td><td>p=0.0408*</td><td>p&lt;0.001***</td><td>-</td></tr></tbody></table>

4.1.3 *Response delays excluding COCCO.* The presence of COCCO notably influenced the variance in response times. When prompted, COCCO typically provided instant responses. To account for this, we conducted a subsequent analysis, excluding COCCO's interventions from the measured response times.

We used a Kruskal-Wallis test that revealed a significant difference in answer delays between conditions (H=31.48, df=3, p<0.001). We conducted post-hoc pairwise comparisons using a Wilcoxon rank sum test (Holm correction) and found significant differences between the control group and
the three manipulation groups, as presented in Table 3. Participants in the control condition had the longest median response delays (29s), compared to all other conditions (ranging from 18s to 22s).

<table><caption>Table 3. Comparison of response delays between conditions excluding COCCO using Wilcoxon rank sum test (Holm correction)</caption><thead><tr><th>Conditions</th><th>Control</th><th>Group chat</th><th>Private chat</th><th>Both chats</th></tr></thead><tbody><tr><td>Control condition (med=29s)</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Group-chat condition (med=18s)</td><td>p&lt;0.001***</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Private-chat condition (med=22s)</td><td>p=0.0093**</td><td>p&gt;0.05</td><td>-</td><td>-</td></tr><tr><td>Both-chats condition (med=18s)</td><td>p&lt;0.001***</td><td>p&gt;0.05</td><td>p&gt;0.05</td><td>-</td></tr></tbody></table>

## 4.2 Cognitive workload and perception

4.2.1 *Cognitive workload.* To evaluate how COCCO influenced participants’ perceptions of the activity we first calculated their cognitive workload using their answers to the NASA Task Load Index [5] in the post-activity survey.

Using a Kruskal-Wallis test, we found no significant differences in NASA-TLX scores between conditions. However, a comparison of the weighted factors of the NASA-TLX score found a significant difference between groups (H=13.19, df=3, p=0.0042) for the effort factor ("How hard did you have to work (mentally and physically) to accomplish your level of performance?"). We conducted post-hoc pairwise comparisons using Wilcoxon rank sum test (Holm correction) and found a significant difference (p=0.0032) between the control condition (med=2.0) and the private-chat condition (med=9.0).

4.2.2 *Perceived team performance, communication and trust.* We looked at how the conditions affected the participants’ perception of the activity in terms of team performance and communication quality, trust in the team, and, for non-control groups, COCCO’s communication quality and trust in it. Using a Kruskal-Wallis test, we found a significant difference in the answers to the Likert-scale between conditions (H=6.13, df=2, p=0.0467) for COCCO’s perceived communication quality. However, subsequent post-hoc comparisons using a Wilcoxon rank sum test found no significant pairwise differences.

4.2.3 *Cognitive workload and perceived team performance, communication and trust.* We then tested for correlation between the NASA-TLX score and the answers to the Likert scale. The correlation analysis using Kendall’s tau revealed a significant relationship between the NASA-TLX score and the COCCO’s perceived communication quality (τ=-0.27, p=0.0060). Participants with a good perception of COCCO’s communication capabilities experienced a lower cognitive workload.

We then tested for correlation between the NASA-TLX’s weighted factors and the answers to the Likert-scale. We found a correlation between the temporal demand factor and COCCO’s perceived communication quality (τ=-0.33, p=0.0007). Participants who had a good perception of COCCO’s communication capabilities experienced lower time pressure. We additionally found correlations between the frustration factor and the perceived team’s communication quality (τ=-0.36, p=0.0003), performance (τ=-0.26, p=0.0100), and trust in the team (τ=-0.30, p=0.0029). Participants who had a higher frustration level had an overall worse impression of their team.
# 5 DISCUSSION

Our study’s exploration of the impact of COCCO on collaborative environments led to several compelling findings that shed light on how chatbot assistance can shape team dynamics and performance.

First, the chatbot’s interventions positively influenced team performance by reducing participants’ module-solving times in non-control groups. The median time went from 21 minutes for groups in the control condition to 18 minutes for the private chat condition, 14 minutes for the both chats condition, and 12 minutes for the group chat condition.

Additionally, they improved the teams’ communication, as can be seen by participants’ faster response times to teammates’ information requests. This indicates that COCCO’s fast answers set a rhythm for the participants and possibly made the team’s exchanges more fluid by preventing unanswered information requests from piling up. This finding reinforces previous conclusions in the literature that chatbots are effective conversational facilitators. However, our study demonstrates this effectiveness in a context where debate structuring was not the chatbot’s primary goal.

Finally, participants with a good impression of the chatbot’s communication quality had a lower cognitive workload. This suggests that chatbot assistance can reduce cognitive workload. More precisely, we found a link between a participants’ perceived communication quality of COCCO and experienced time pressure. This might indicate that the chatbot reduced some of the pressure and workload experienced by the participants, possibly by speeding the rate of all the answers to information requests and answering some of the requests by itself, leaving more time for participants to solve their own modules.

Looking at the different communication strategies tested with COCCO, we observe that providing assistance via private chat was much less effective than via group chat. Indeed, while team performance in the private chat condition was still better than in the control groups, it was significantly worse than that of the teams in the group chat condition. This is also true of teams in the both chat condition, who had worse performance than group chat condition teams, showing that chatbot assistance by private chat is diminishing the effectiveness of the assistance. This results in our observation that participants interacting with COCCO in private chat felt that they had to exert more effort than control group participants. This can be explained by the fact that this communication strategy requires participants to frequently switch between channels and actively request assistance. This fits with our observation that requests to COCCO in private chat would decline with time or even stop for some teams. In private chats, COCCO required direct engagement or prompting by participants to be optimally functional, as it had only one conversational partner. In group chats, COCCO could passively monitor the team’s conversation and had more opportunities to react to information requests without needing extra effort from the participants. Additionally, non-use by one teammate would directly diminish COCCO’s usefulness for the rest of the group and increase the chances that it would be completely ignored. Indeed, out of a total number of 126 requests for information by COCCO, only 48 (38%) were answered. This is mostly explained by the fact that most of those requests were hint requests that were sent on behalf of a player to their three other teammates. This meant that participants would receive requests for information they did not have and chose to simply not answer COCCO. Some other participants might also have received successive requests and chose to answer only the most recent one. However, some of them would also not answer when they were the only ones with the requested information.

Following these insights, we suggest that chatbots should adopt a more dynamic role in team collaboration. Instead of passively absorbing information and reacting to specific requests, chatbots should take a more proactive part in the activity by, for example, noticing missing information in their model and deducting it from other information in their model or requesting it from
human teammates. This would help maintain communication with the chatbot and keep it relevant throughout the collaboration, especially for longer-term collaborations.

## 5.1 Limitations and future directions

Our study offers valuable insights into the role of chatbots in collaborative environments, but it has limitations. One significant limitation was the absence of other prevalent communication channels in our setup, such as phone calls or email. As noted by Snyder and Lee-Partridge [18], these methods are substantial competitors to newly introduced collaboration tools. Excluding these channels in our study may have unintentionally inflated the perceived effectiveness and influence of COCCO in certain collaborative contexts.

Additionally, our study focused exclusively on groups receiving assistance through a chatbot, without exploring other forms of support, such as non-chatbot or non-conversational AI-based feedback. This limitation prevents us from determining the extent to which the observed improvements are due to the unique characteristics of conversational interfaces, as opposed to general assistance.

Looking ahead, there are multiple avenues of research that can be expanded on with our collaborative activity and platform. Although disappointing in this study, chatbot assistance in private channels might be more beneficial in larger teams, where group chats could become busier and participants would have more teammates and information to manage.

As teams familiarize themselves with chatbot assistance, an analysis of how expert usage evolves becomes imperative, especially for teams where teammates have become familiar with each other and roles have cemented themselves.

Lastly, to solidify the generalizability of our findings, the study could benefit from replication involving a more extensive participant base, especially by forming teams of varied sizes and with different objectives (time limit, competition between teammates). This would provide a more comprehensive understanding of chatbots' influence on group dynamics across different collaborative scales and configurations.

In the broader context of collaborative environments, the emergence of models like ChatGPT and its variants offers new avenues for exploration. Notably, the recent development of TeamGPT<fnref n="2" />, a derivative specifically designed for groups, presents intriguing possibilities. While COCCO demonstrated significant potential in our study, models like TeamGPT, backed by state-of-the-art large language model capabilities, could further elevate the usefulness of chatbots in collaborative settings. Such models, designed with the nuances of group dynamics in mind, could provide more context-aware responses, aiding in decision-making processes, brainstorming sessions, or even conflict resolution. As these advanced models mature and become more accessible, it is invaluable for future research to delve into their impact on various collaborative scenarios and how they might redefine team interactions in the digital age.

## 6 CONCLUSION

This study explored the role of chatbots in shaping collaborative spaces. Our findings underscore the potential of chatbots to enhance team collaboration, especially when integrated into communication platforms. COCCO's noticeable impact on the group chat condition, contrasted with its less prominent role in private chats, provides valuable insights into chatbot deployment and adaptability requirements. Despite certain limitations, it is evident that the right chatbot setup can reduce response delays and cognitive burdens. Understanding these dynamics is critical as we enter an era of deepening human-AI synergy.

<footnote n="2">²https://www.teamgpt.com</footnote>
REFERENCES

[1] Ivo Benke, Michael Thomas Knierim, and Alexander Maedche. 2020. Chatbot-based emotion management for distributed teams: A participatory design study. *Proceedings of the ACM on Human-Computer Interaction* 4, CSCW2 (2020), 1–30.

[2] Hyo Jin Do, Ha-Kyung Kong, Jaewook Lee, and Brian P Bailey. 2022. How Should the Agent Communicate to the Group? Communication Strategies of a Conversational Agent in Group Chat Discussions. *Proceedings of the ACM on Human-Computer Interaction* 6, CSCW2 (2022), 1–23.

[3] Joel E Fischer, Wenchao Jiang, and Stuart Moran. 2012. AtomicOrchid: A mixed reality game to investigate coordination in disaster response. In *International Conference on Entertainment Computing*. Springer, 572–577.

[4] Asbjørn Følstad, Theo Araujo, Effie Lai-Chong Law, Petter Bae Brandtzaeg, Symeon Papadopoulos, Lea Reis, Mar-cos Baez, Guy Laban, Patrick McAllister, Carolin Ischen, et al. 2021. Future directions for chatbot research: an interdisciplinary research agenda. *Computing* 103, 12 (2021), 2915–2942.

[5] Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In *Advances in psychology*. Vol. 52. Elsevier, 139–183.

[6] Soomin Kim, Jinsu Eun, Changhoon Oh, Bongwon Suh, and Joonhwan Lee. 2020. Bot in the bunch: Facilitating group chat discussion by improving efficiency and participation with a chatbot. In *Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems*. 1–13.

[7] Soomin Kim, Jinsu Eun, Joseph Seering, and Joonhwan Lee. 2021. Moderator chatbot for deliberative discussion: Effects of discussion structure and discussant facilitation. *Proceedings of the ACM on Human-Computer Interaction* 5, CSCW1 (2021), 1–26.

[8] Kaisa Laitinen, Salla-Maaria Laaksonen, and Minna Koivula. 2021. Slacking with the bot: Programmable social bot in virtual team interaction. *Journal of Computer-Mediated Communication* 26, 6 (2021), 343–361.

[9] Carlene Lebeuf, Margaret-Anne Storey, and Alexey Zagalsky. 2017. How software developers mitigate collaboration friction with chatbots. *arXiv preprint arXiv:1702.07011* (2017).

[10] Vincent F Mancuso and Michael McNeese. 2013. TeamNETS: Scaled world simulation for distributed cyber teams. In *HCI International 2013-Posters' Extended Abstracts: International Conference, HCI International 2013, Las Vegas, NV, USA, July 21-26, 2013, Proceedings, Part II 15*. Springer, 509–513.

[11] Vincent F Mancuso, Dev Minotra, Nicklaus Giacobe, Michael McNeese, and Michael Tyworth. 2012. idsNETS: An experimental platform to study situation awareness for intrusion detection analysts. In *2012 IEEE international multi-disciplinary conference on cognitive methods in situation awareness and decision support*. IEEE, 73–79.

[12] Michael D McNeese, Priya Bains, Isaac Brewer, Cliff Brown, Erik S Connors, Tyrone Jefferson Jr, Rashaad ET Jones Jr, and Ivanna Terrell Jr. 2005. The NeoCITIES simulation: Understanding the design and experimental methodology used to develop a team emergency management simulation. In *Proceedings of the human factors and ergonomics society annual meeting*, Vol. 49. SAGE Publications Sage CA: Los Angeles, CA, 591–594.

[13] OpenAI and Josh Achiam et al. 2024. GPT-4 Technical Report. arXiv:2303.08774

[14] Sarvapali D Ramchurn, Feng Wu, Wenchao Jiang, Joel E Fischer, Steve Reece, Stephen Roberts, Tom Rodden, Chris Greenhalgh, and Nicholas R Jennings. 2016. Human-agent collaboration for disaster response. *Autonomous Agents and Multi-Agent Systems* 30 (2016), 82–111.

[15] Beau G Schelble, Christopher Flathmann, Nathan J McNeese, Guo Freeman, and Rohit Mallick. 2022. Let's think together! Assessing shared mental models, performance, and trust in human-agent teams. *Proceedings of the ACM on Human-Computer Interaction* 6, GROUP (2022), 1–29.

[16] Joseph Seering, Michal Luria, Geoff Kaufman, and Jessica Hammer. 2019. Beyond dyadic interactions: Considering chatbots as community members. In *Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems*. 1–13.

[17] Amir Shevat. 2017. *Designing bots: Creating conversational experiences*. " O'Reilly Media, Inc.".

[18] Jason Snyder and Joo Eng Lee-Partridge. 2013. Understanding communication channel choices in team knowledge sharing. *Corporate Communications: An International Journal* 18, 4 (2013), 417–431.

[19] Stergios Tegos, Stavros Demetriadis, and Thrasyvoulos Tsiatsos. 2014. A configurable conversational agent to trigger students' productive dialogue: a pilot study in the CALL domain. *International Journal of Artificial Intelligence in Education* 24 (2014), 62–91.

[20] Carlos Toxtli, Andrés Monroy-Hernández, and Justin Cranshaw. 2018. Understanding chatbot-mediated task management. In *Proceedings of the 2018 CHI conference on human factors in computing systems*. 1–6.

[21] Douglas P Twitchell, Karl Wiers, Mark Adkins, Judee K Burgoon, and Jay F Nunamaker. 2005. StrikeCOM: A multi-player online strategy game for researching and teaching group dynamics. In *Proceedings of the 38th Annual Hawaii International Conference on System Sciences*. IEEE, 45b–45b.

[22] Dakuo Wang, Haoyu Wang, Mo Yu, Zahra Ashktorab, and Ming Tan. 2022. Group chat ecology in enterprise instant messaging: How employees collaborate through multi-user chat channels on slack. *Proceedings of the ACM on Human-Computer Interaction* 6, CSCW1 (2022), 1–14.
[23] Qingxiao Zheng, Yiliu Tang, Yiren Liu, Weizi Liu, and Yun Huang. 2022. UX research on conversational human-AI interaction: A literature review of the ACM digital library. In *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems*. 1–24.

# A COCCO’S INTENTS

Received 17 January 2024; revised 16 July 2024; accepted 31 October 2024
<table><caption>Table 4. COCCO's intents (Dialogflow)</caption><thead><tr><th>Name and use case</th><th>Matched parameters</th></tr></thead><tbody><tr><td colspan="2">Conversational intents</td></tr><tr><td>Welcome: participant greets the chatbot</td><td>/</td></tr><tr><td>Help: participant asks for help</td><td>/</td></tr><tr><td>Confirmation: participant confirms or agrees</td><td>/</td></tr><tr><td>Contradiction: participant contradicts or disagrees</td><td>/</td></tr><tr><td colspan="2">Information given: COCCO's knowledge base is updated.</td></tr><tr><td>Buttons hint: participant shares the hint for a button module of a specific color.</td><td>The module's color and the correct solution composed of a sequence (left, center, right).</td></tr><tr><td>Grid hint: participant shares the hint for a field module of a specific color.</td><td>The module's color and the correct solution composed of an expression describing a position in a 3x3 grid (ex: "upper right", "center left").</td></tr><tr><td>Checkboxes hint: participant gives the hint for a field module of a specific color.</td><td>The module's color and the correct solution composed of a module's number.</td></tr><tr><td>Checkboxes solution: participant shares a checkboxes module's solution</td><td>The module's color and its solution which is a list of colors.</td></tr><tr><td>Menu hint: participant shares the hint for a field module of a specific color.</td><td>The module's color and the correct solution composed of a type of module, a color and a module's number.</td></tr><tr><td>Menu solution: participant shares a menu module's solution</td><td>The module's color and its solution which is a player's pseudo.</td></tr><tr><td>Field hint: participant shares the hint for a field module of a specific color.</td><td>The module's color and the correct solution composed of a type of module, a color and one of the players' pseudo.</td></tr><tr><td>Field solution: participant shares a field module's solution</td><td>The module's color and its solution which is a number.</td></tr><tr><td>Module color: participant shares the color of one of their module.</td><td>The module's number and color.</td></tr><tr><td>Module number: participant shares the number of one of their module.</td><td>The module's number, and optionally, the module's type and color.</td></tr><tr><td>Module information: participant confirms that they have a specific module</td><td>The module's type, color and number.</td></tr><tr><td colspan="2">Requests: COCCO checks its knowledge base and share the requested information if it has it.</td></tr><tr><td>Color request: participant requests the other participants module colors.</td><td>The color of the relevant module.</td></tr><tr><td>Hint request: participant requests the hint for a module.</td><td>The module's type and color.</td></tr><tr><td>Number request: participant requests another player's module color.</td><td>The module's type and color, and, optionally depending on the context the other player's pseudo.</td></tr><tr><td>Participant request: participant requests to know which player has a specific module.</td><td>The module's type, color and number.</td></tr></tbody></table>