<figure><img src="image_1.png" /></figure>

HAL
open science

# How to Categorize Collaboration during a Collaborative Puzzle-solving Task? Validation of Collaboration Profiles using Multimodal Data in Virtual Reality Context

Aurélien Léchappé, Cédric Fleury, Mathieu Chollet, Cédric Dumas

<figure><img src="image_2.png" /></figure>

## To cite this version:

Aurélien Léchappé, Cédric Fleury, Mathieu Chollet, Cédric Dumas. How to Categorize Collaboration during a Collaborative Puzzle-solving Task? Validation of Collaboration Profiles using Multimodal Data in Virtual Reality Context. Proceedings of the ACM on Human-Computer Interaction, 2025, 9, pp.1 - 46. 10.1145/3710996. hal-05317027

HAL Id: hal-05317027

https://imt-atlantique.hal.science/hal-05317027v1

Submitted on 16 Oct 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
How to Categorize Collaboration during a Collaborative
Puzzle-solving Task? Validation of Collaboration Profiles
using Multimodal Data in Virtual Reality Context

AURÉLIEN LÉCHAPPÉ, IMT Atlantique, LS2N, UMR CNRS 6004, Lab-STICC, UMR CNRS 6285, France

CÉDRIC FLEURY, IMT Atlantique, Lab-STICC, UMR CNRS 6285, France

MATHIEU CHOLLET, University of Glasgow, School of Computing Science, Scotland; IMT Atlantique, LS2N, UMR CNRS 6004, France

CÉDRIC DUMAS, IMT Atlantique, LS2N, UMR CNRS 6004, France

In high-stakes collaborative situations, a decline in collaboration quality can lead to adverse events with significant consequences. Analyses performed by Human factor (HF) specialists, while effective in identifying and addressing collaboration issues, are case-specific and most of the time performed a posteriori. To address these limitations, our research focuses on a real-time assessment of collaboration processes using multimodal signals collected and analyzed during the activity. Existing collaboration profiles taxonomies face limitations such as a posteriori profiles detection and the absence of quantitative behavioral indicators that can be measured during the activity. Leveraging Virtual Reality (VR), we have developed a framework for evaluating collaboration in controlled setting, testing the effectiveness of a subset of multimodal signals to detect collaboration profiles. We test our approach in a study including 11 stereotyped collaborative scenarios applied to a VR puzzle-solving task. This study reveals the effectiveness of our approach in distinguishing between non-collaborative and highly collaborative profiles. However, challenges arise in discriminating between closely related collaborative profiles. This paper also proposes some guidelines on how to improve the collaboration profile detection framework and address other collaborative situations.

CCS Concepts: • **Human-centered computing** → **Computer supported cooperative work; Empirical studies in collaborative and social computing; Virtual reality.**

Additional Key Words and Phrases: Collaboration cues and profiles, Detection approach, Multimodal signals.

# 1 Introduction

A decrease in the collaboration quality can lead to adverse events, sometimes with severe consequences for task outcomes or people in industrial or medical field [39] for example. Previous studies on robotic surgery, considering both the surgeon's and team's perspectives, indicate that issues in surgery are mostly related to miscommunication and misunderstanding among collaborators, rather than technical skills [3, 5]. These issues, coupled with a lack of non-technical skills like decision-making leadership and situation awareness, increase risk of surgical complications [39].

Human factor (HF) specialists can analyze working situations to identify and remedy such issues, but this is a long process based on observation and analysis of multimodal elements and behavior, inextricably linked to the task performed. However, applying the HF approach can be complex in many situations, such as in an operating room (OR). To overcome these limitations, our research focuses on **real-time assessment of collaboration processes using multimodal signals** (e.g. verbal and non verbal behaviors, physiological signals, etc.) analyzed during the activity.

We chose to study collaboration in virtual environments (VE) due to their advantages. Virtual Reality (VR) allows us to **control the users' environment**, removing information or distractors that can occur in real-world situations. VR devices also enable **precise behavioral data collection**,

Authors' Contact Information: Aurélien Léchappé, aurelien.lechappe@imt-atlantique.fr, IMT Atlantique, LS2N, UMR CNRS 6004, Nantes and Lab-STICC, UMR CNRS 6285, Brest, France; Cédric Fleury, cedric.fleury@imt-atlantique.fr, IMT Atlantique, Lab-STICC, UMR CNRS 6285, Brest, France; Mathieu Chollet, Mathieu.Chollet@glasgow.ac.uk, University of Glasgow, School of Computing Science, Glasgow, Scotland; and IMT Atlantique, LS2N, UMR CNRS 6004, Nantes, France; Cédric Dumas, cedric.dumas@imt-atlantique.fr, IMT Atlantique, LS2N, UMR CNRS 6004, Nantes, France.
such as eye-tracking, audio, user positions, and task-related events, which **can be harder to collect in real situations**. Thus, VR environments enable us to take a first step towards developing a framework for real-time collaboration evaluation in a controlled setting, testing the effectiveness of various indicators for detecting collaboration levels and profiles. VR environment also facilitate the exploration of strategies for re-establishing an adequate level of collaboration. This approach also addresses ethical challenges in studying real-world collaboration.

In this paper, we propose a set of collaboration profiles derived from existing schemes enhanced with an approach focused on evaluating collaboration using multimodal signals. Our contributions include (1) a set of collaboration profiles based on indicators calculated from multimodal signals, (2) a method for evaluating those profiles applied on a collaborative puzzle-solving task in VR, (3) a collection of collaboration scenarios with instructions for the puzzle-task eliciting the proposed profiles, and (4) the recorded datasets of those collaboration scenarios played out by actors.

The paper is structured as follows: Section 2 describes the related work on collaboration profiles in synchronous and co-located collaborative problem-solving tasks, followed by the description of an emergent approach for real-time evaluation of collaboration using multimodal data to compute relevant indicators; Section 3 presents the collaborative task and the data acquisition system; Section 4 details the observations of recorded collaboration sessions and adaptations of collaboration profiles for evaluation and detection during the activity; Section 5 gives the technical details on profile detection; Section 6 presents the results about the validation of profiles detection; and Section 7 discusses the limits of our approach and proposals for future studies.

## 2 Related Work

### 2.1 Collaboration profiles

Previous work explored collaboration analysis based on quantitative indicators. Shaer et al. [50] proposed to evaluate collaborative learning using video analysis, by measuring elements derived from Meier et al.'s and Collaborative Learning Mechanisms (CLM) frameworks [19, 36]: collaborators' attitudes, levels of verbal and physical participation, equality of participation, nature of discussions, and learning outcomes during neuroscience training. Based on these measures, Shaer proposed 4 dyadic collaboration profiles. **Turn-takers** refers to collaborators who engage in critical discussions, accept suggestions, and make decisions respectfully, participating equally in both verbal and physical aspects. **Driver-Navigator** includes a **Navigator** who observes and makes suggestions and a **Driver** who listens and coordinates with the **Navigator** to do most of the physical activity. **Driver-Passenger** corresponds to a **Driver** highly engaged verbally and physically, and a **Passenger** who is less focused and participates rarely. The **Driver** shares minimal information and may block or undo the **Passenger** actions. **Independent** describes collaborators who focus on their own activities, engaging in minimal communication, considering each other's actions, and executing tasks sequentially.

Extensive work has focused on evaluating collaboration on tabletops, particularly in identifying collaboration profiles. Tang et al. [53] categorized 6 dynamic collaboration profiles based on video analysis and coding of physical positioning, which task or sub-task individuals worked on, collaborators engagement, presence and type of conversation, and interference handling during a route creation task. Isenberg et al. [23] expanded on Tang et al.'s profiles, introducing 4 additional profiles based on how much information was shared among collaborators during a visual analytics task. Maquil et al. [31] adapted some collaboration profiles from Isenberg et al. to a space mining scenario, in which the work has to be distributed among three group members. They analyzed these profiles using video coding of log events, conversation, gestures, and body language.
Schneider and Pea [47] evaluated the impact of mutual gaze perception on collaboration quality and collaborative learning in a neuroscience analysis task. They used Meier et al.'s framework to evaluate collaboration quality and adopted Shaer's Driver-Passenger profile to categorize participants as Leader or Follower. The authors introduced a coding scheme to distinguish these profiles, based on discussion initiation, speaking frequency, turn-taking management, and decision-making on attention focus. They noted that individuals tend to change roles based on task/subtask activities. It emphasizes that their binary distinction may not fully capture dyad dynamics and that their coding scheme provides a global estimation of dyad collaboration over the entire activity.

All the presented profiles rely on post-experiment analyses, using mainly video traces. No study, to our knowledge, tried to automatically analyze these profiles using behavioral multimodal signals.

## 2.2 Collaboration process and quality assessment using multimodal data

Understanding the collaboration state is crucial to automatically detect real-time collaboration issues. Meier et al.'s [36] framework, outlining five key aspects (A1-5) of successful collaboration, is pertinent in this context. This five-aspect decomposition is subdivided into nine dimensions measuring different aspects of collaboration quality. **Communication** (A1) refers to the construction of the common ground [13] (i.e. a common representation of the situation) and the processes of maintaining it. Meier et al. proposed two dimensions to assess this aspect: *Mutual Understanding*, referring to the grounding process (i.e. constant updating of each collaborator's individual representations of the situation [12]), and *Dialogue Management*, including turn-taking management and coordination aspects of communication. **Joint Information Processing** (A2) involves pooling individual and complementary knowledge at the group level. The *Information Pooling* dimension assesses aspects facilitating this process, including meta-knowledge about expertise and based knowledge of each collaborator, as well as information shared at the right moment. The *Reaching Consensus* dimension reflects the decision-making process based on the shared information. **Coordination** (A3) is crucial for successful collaboration, addressing constraints arising from task nature and time-limited resource interdependencies. The *Task Division, Time Management, and Technical Coordination* dimensions evaluate how participants manage task dependencies, cope with time constraints, and handle technical interdependencies. **Interpersonal Relationship** (A4) is related to the notion of interaction and contribution symmetry mentioned by Dillenbourg [16]. The *Reciprocal Interaction* dimension underscores the importance of respectful social interaction and equitable collaboration contributions resulting from symmetrical interpersonal relationships. Finally, the **Motivation** (A5) aspect, assessed by the *Individual Task Orientation* dimension, reflects collaborators' motivation and engagement with the collaborative task.

Already adapted for Computer-Supported Collaborative Learning (CSCL) contexts [26, 45], Meier et al.'s framework serves as a reference in studies evaluating collaboration based on multimodal data [15, 33, 48]. It is employed in categorizing collaborative activity into profiles [47, 50].

Some previous studies have evaluated collaboration quality using multimodal metrics derived from physiological, audio, gaze and gesture modalities in various co-located or remote setups. They compared their results with the measures from Meier et al.'s framework or related rating scales. Schneider et al. [49] reviewed the use of multimodal data to capture collaboration aspects and called this approach Multimodal Collaboration Analytics (MMCA). Praharaj et al. [41] conducted a survey on modeling co-located collaboration. These two studies highlight the frequent use of Meier et al.'s scale when assessing collaboration with multimodal data. Praharaj et al. [42] provided an inventory of collaboration quality indicators, proposing high-level indices such as synchrony, equality, and mutual understanding derived from low-level indicators. Using the audio modality, they also introduced indicators for problem-solving context based on Meier et al.'s framework: dominance, rapport, expertise, active participation, roles, and knowledge co-construction.
Different approaches exist to assess collaboration quality and link it with multimodal signals. A first one involves evaluating collaborative activity with experts to detect issues and identify moments of good or poor collaboration quality, using methods such as observation or activity analysis [23, 31, 32, 50, 52, 53]. Another approach uses a global collaboration quality measure to assess situations and compare indicators across different collaboration levels, ranging from bad to good, without considering specific collaboration aspects or dimensions. The third approach uses a rating scheme to assess various collaboration aspects, comparing them with indicators computed using multimodal signals [9, 15, 33, 37, 42, 48]. This technique can be implemented in various ways, such as using machine learning algorithms to interpret and analyze indicators based on labeled collaboration segments [9, 37], or interpreting collaboration quality based on indicator value as linked to collaboration process dimensions in the literature [15, 33, 42, 48].

Given our aim to assess collaboration quality in real-time, we need to use all activity traces based on the knowledge of collaboration indicators. For our initial iteration, we choose the last technique, interpreting collaboration quality according to the valence of indicators presented in the literature.

## 2.3 Collaborative behavioral indicators and aspects of the collaboration process

The taxonomies of collaboration profiles presented in section 2.1 considered indicators such as discussion, consideration of the other, engagement in the task reflected by gestures and posture, visual and para-verbal indexes. Table 1 provides an overview of the various aspects and metrics related to the collaborative process. It includes the indicators presented by these taxonomies, as well as the other studies evaluating the collaboration quality with multimodal signals. It uses the dimensions introduced by Meier et al. [36] as a reference for collaboration measures. The indicators are classified into different aspects. **Dominance** reflects the balance of collaborators' contribution in the collaborative task. **Joint attention** between collaborators corresponds to “*the ability to coordinate attention toward a social partner and an object of mutual interest*” [7]. It plays a crucial role in regulating attention and establishing the common ground necessary for task realization. **Communication process management** refers to processes such as *turn-taking management* or *communication flow*. **Spatial behavior** encompasses spatial and physical indicators such as *physical proximity, gestures behaviors and postural and gesture synchrony*. **Speech content** refers to the nature of the discussion performed during the activity. Lastly, **group characteristics** encompass characteristics specific to the individual and the team composition such as individual *behavioral engagement* to realize the task [38], or relationships between collaborators.

## 2.4 Synthesis of related work

The proposed taxonomies had several limitations: (i) the profile analyses are conducted a posteriori through observations, (ii) they do not consider quantitative behavioral indicators measured during the activity and (iii) the proposed profiles are mainly limited to tasks on tabletops. The proposed profiles are thus incomplete for evaluating collaboration in real time, and studies investigating such profiles in VR have not yet been conducted.

While collaboration analysis and profile categorization has been significant topics in the CSCW and CSCL communities, no previous approach has addressed such collaboration assessment in real-time. This paper explores how advances in data acquisition setups, device capabilities, and AI research make real-time assessment possible. Our approach can be valuable to the CSCW and CSCL communities by providing new solutions for assessing the collaboration with quantitative metrics and detecting collaboration changes during activities. Real-time assessment can enable researchers to predict and address sub-optimal collaboration by providing live feedback on the collaboration state.
<table><caption>Table 1. Resume of indicators used to assess the collaboration process and differentiate collaboration profiles.</caption><thead><tr><th>Aspects</th><th>Indicators</th><th>Link with Meier et al.'s dimension</th><th>Definitions</th></tr></thead><tbody><tr><td rowspan="3">Dominance</td><td>Speech equality</td><td>Reciprocal interaction<br>Dialogue management</td><td>Speaking-time distribution [6, 16, 47, 50]</td></tr><tr><td>Task equality (physical, discussion)</td><td>Reciprocal interaction</td><td>Contribution to the task [16, 31, 50]</td></tr><tr><td>Attention level</td><td>Sustaining mutual understanding</td><td>Attention level on the continuum of Siposova and Carpenter [51]</td></tr><tr><td rowspan="4">Joint attention</td><td>Joint Visual Attention (IVA) frequency</td><td>Sustaining mutual understanding<br>Information pooling</td><td>When several people are looking a same object within a window of plus or minus 2 seconds [43]</td></tr><tr><td>Mutual gaze frequency</td><td>Sustaining mutual understanding</td><td>When several people are looking at each other at the same time [2]</td></tr><tr><td>People gaze frequency</td><td>Sustaining mutual understanding</td><td>When one individual looks at another individual. [24]</td></tr><tr><td>Who leads visual attention</td><td>Information pooling</td><td>Distribution of the roles of initiator and responder in looks [10, 47]</td></tr><tr><td rowspan="2">Communication processes management</td><td>Turn-taking, overlap and interruptions frequency</td><td>Dialogue management</td><td>Frequency and turn taking (with, without overlap, overlap, backchannels, interruption, turn-keeping) [21]</td></tr><tr><td>Verbal participation</td><td>Dialogue management<br>Information pooling</td><td>Frequency of verbal exchanges and content exchanged [28]</td></tr><tr><td rowspan="3">Spatial behaviors</td><td>Physical proximity</td><td>Sustaining mutual understanding</td><td>Physical proximity of individuals to each other (e.g. f-formation) [23, 27]</td></tr><tr><td>Synchrony in posture</td><td>Reciprocal interaction<br>Individual task orientation</td><td>Synchrony in the activity level of users for task realization, represented from the analysis of their postures [14, 46, 54]</td></tr><tr><td>Gestures</td><td>Sustaining mutual understanding<br>Information pooling</td><td>Gestures, such as deictic (i.e. pointing) gestures [4, 50]</td></tr><tr><td rowspan="3">Speech content</td><td>Type of discussion</td><td>Information pooling<br>Sustaining mutual understanding<br>Reaching consensus<br>Task division<br>Time management</td><td>We consider the nature of discussion performed by collaborators, i.e.: if collaborators make suggestions, accept or reject suggestions, explain their actions, discuss prior to decision making, coordinate for task realization [1, 34, 44, 50]</td></tr><tr><td>Behavioral engagement</td><td>Individual task orientation</td><td>Individual's level of engagement in task achievement [20, 38]</td></tr><tr><td>Rapport</td><td>Reciprocal interaction<br>Individual task orientation</td><td>Rapport between individuals [30]</td></tr><tr><td rowspan="3">Group characteristics</td><td>Expertise</td><td></td><td>Expertise and based knowledge of individual regarding the task or subtask [11]<br>Too many experts can lead to poor collaboration [22]<br>Expertise and behavioral engagement linked to group performance [55]</td></tr><tr><td>Expertise</td><td></td><td></td></tr><tr><td>Expertise</td><td></td><td></td></tr></tbody></table>
<figure><img src="image_4.png" /><figcaption>Fig. 1. Different step of the collaborative puzzle-solving task. (a) research and retrieve piece in the environment, (b) positioning piece on the puzzle, (c) coloring the puzzle pieces according to a reference figure.</figcaption></figure>

As a consequence, we designed new profiles by combining existing taxonomies and augmenting them with different indicators proposed in other previous studies. We based this design on our observations on a VR collaborative puzzle-solving task. We thus proposed 7 dyadic collaboration profiles considering the dimensions introduced by Meier et al. [36]. These profiles are distinguished using the 6 aspects presented in Table 1. These aspects can evolve during the activity, implying that we can observe different collaboration profiles during a single activity.

# 3 Experimental setup

## 3.1 Collaborative puzzle-solving task

We designed our collaborative task based on literature. Praharaj [40] highlights that problem-solving is a common scenario for studying co-located collaboration. Ladwig & Geiger [29] emphasize that assembly tasks are typical for evaluating collaboration in mixed reality. Therefore, we designed a puzzle-solving task to investigate co-located synchronous collaboration in virtual reality. This task was implemented with Unity.

After completing a brief tutorial, the two participants (P1 and P2) are transported in a virtual room that is an exact copy of the one in which they are physically co-located. Users' position in the physical space is directly mapped to their position in the virtual environment, so that they can physically move to travel in VR. They need to collaborate to retrieve pieces scattered in the environment (Fig. 1.a), then position (Fig. 1.b) and complete a puzzle based on a reference figure displayed in the virtual room (Fig. 1.c). The puzzle includes 4 colors: blue (P1's color), green (P2's color), cyan (mix of both color), white (no color). By default, all pieces are white and take on the color of the participant when placed on the puzzle. For the cyan color, both participants must select the same piece. They cannot succeed individually and must work together to complete the task.

## 3.2 Data acquisition

We used two HTC Vive Pro Eye headsets, allowing us to collect eye-tracking data using the embedded Tobii eye-tracker. Four computers connected to a local network were used. A first one collected skeletons, video and audio with an Azure Kinect. Two computers were used as clients for each participant, running the VR application. They ran a Unity client, along with a custom application for sending the participant audio to a server. The last computer ran a Unity server and a data collection server. The Unity server allows to synchronize the two clients, while the data collection server collects the synchronized data from the different computers.

Datasets are recorded using the framework Microsoft psi [8]. The dataset is split into several "partitions" that each contain specific types of data. The first partitions contain raw data such as recordings of first-person views of users, skeletons, audio, voice activity detection boolean, speech to text (French) outputs, position and rotation of users in the virtual environment. Other partitions contain raw data values for each individual (e.g. task and gaze events). Another partition provides print versions for data visualization in the Psi Studio tools. The last partitions contain individual and team metrics computing based on the raw data as well as profile confidence.
# 4 Elicitation of collaboration profiles

To obtain an initial understanding of how collaborators would behave in our experimental task, we conducted a preliminary study. We observed 42 sessions of the collaborative puzzle-solving task, involving 84 participants (56M, 28F, mean age: 26.56, sd: 9.66) from our university, including both students and professionals. Each pair of participants knew each other.

The sessions were analyzed using taxonomies (Section 2.1), signals and indicators (Section 2.2) from the literature. Table 1 summarizes the selected indicators. A subjective analysis of each session examined how individuals collaborated, based on these indicators and informed by the above collaboration theories. Key aspects considered included communication, task activity, performance, spatial behaviors, and coordination (Section 4). Observations on these aspects along each session were classified to identify the main and sub-strategies used during the collaborative activity.

## 4.1 Observations on the collaborative task

We identified two main strategies and various sub-strategies when performing the task: participants chose to work either (1) *individually* or (2) *collectively*, with various levels of task allocation among them. These strategies had a direct impact on the different collaboration aspects.

**Communication.** In terms of the *individual* strategies, most groups exhibited *minimal* or *no communication* during piece retrieval phase. They communicated for positioning and coloring pieces, but mainly in the form of instructions, questions and answers, rather than active information sharing. Other groups demonstrated *limited communication* during all the task, with exchanges occurring only when mandatory. Interactions were characterized by an *absence of mutual responses*, but featured occasional interventions for questions, suggestions, or instructions related to the task (A: "Have you got the last one?", B: "You have to take off the one on the right"). Some participants *verbalized their actions and difficulties*, sometimes followed by the cause of the problem ("Ah! Why doesn't it want to set down?", "It must be in the other direction, yeah, it's not the right one"). This situation often resulted from collaborators working individually, one of them not interacting much.

When adopting *collective* strategies, dyads maintained *continuous and active communication* throughout the activity. They verbalized actions, shared difficulties, proposed suggestions, validated actions, and asked questions continuously (A: "Deselect here", B: "Yes wait, change color, hop", "here I am putting mine down, go ahead and select there", "there you go great", "ok that's good").

**Coordination.** We observed that communication directly affects coordination between participants, leading to either *active* or *minimal coordination*. During the retrieval phase, a *minimal coordination* led to comparable piece retrieval between the collaborators without communication, or one collaborator collecting most pieces and informing the partner once in front of the puzzle. On the contrary, an active coordination was characterized by a synchronized piece collection through communication ("Go ahead and do the green stuff", "We all take it and then we think").

During the piece positioning phase, most participants tried to place the pieces without a clear communicated strategy. Given that the lower pieces had to be fitted first, they faced challenges, leading to three sub-strategies: (i) repeated attempts on all pieces until success, (ii) an individual remembering and instruction sharing, and (iii) an individual remembering, but no instruction sharing or unclear explanations to the partner. One individual sometimes also decided to take the initiative and lead the activity. Communication about retrieved pieces generally facilitated positioning, while the absence of information sharing could lead to performance drops and errors.

During the coloring phase, some participant pairs worked individually. In this case, difficulties encountered prompted either trial-and-error resolution or verbalizing difficulties to the collaborator, who then provided simple instructions, sometimes without further explanation. On the contrary,
other participant pairs worked collectively, adjusting puzzle piece colors based on the reference figure through communication ("*Now we do the Z there*", "*Yeah, the Z, go ahead*`).

**Awareness of others.** Although participants were aware of each other when they arrived in the virtual environment at the beginning, gaze between them were rare once engaged in the task. In individual strategies, participants performed the task without considering each other. For instance, if A requested something of B without receiving an immediate response, A might turn to observe B's actions. While B considered A's request, A remained unaware of B's ongoing activities and location. Awareness could also be lacking when participants focused on their own tasks, only considering the other when faced with difficulties.

In collective strategies, participants had more consideration for the collaborator. It manifested as shared focus on the same pieces. It also included instances where participants placed themselves in the place of the other or distanced themselves during puzzle positioning and coloring for a broader view of the activity and the collaborator's actions.

**Movement and proximity.** Participants did not move around much, except during the piece retrieval phase. During the positioning and coloring phases, they remained side by side facing the puzzle. This can be due to the possibility of remotely retrieving pieces, and the 2D layout of the puzzle with the reference figure visible behind. For this task, participants had thus little interest in arranging themselves other than side by side or with one of them slightly behind the other.

**Updating the common ground.** Behaviors related to common ground (CG) update were observed during effective collaboration or in reaction to the emergence of critical situations. These behaviors include coordinating, sharing of objectives during the task, and using environment elements (e.g. reference figure, spatial statements) to locate oneself in space or give information to the other collaborator. In individual strategies, participants completed the task without sharing information and updating their CG. As a consequence, they were more likely to experience problems and misunderstandings. When such problems occurred, participants had to verbalize their difficulties, discuss to develop a coherent CG and return to an appropriate level of collaboration.

In collective strategies, sharing information when carrying out actions and anticipating each other's actions were evidence of an up-to-date CG. Progressive puzzle completion by both partners maximizes the CG by focusing their attention on the same objects and steps of the task.

Finally, in both strategies, participants may impose false constraints on themselves when performing the task. These constraints, whether shared or not, can lead to divergent CGs, especially in groups applying collective strategies with no critical discussion of their understanding and actions.

## 4.2 Proposed profiles

The result of the above analysis of our preliminary study yielded a set of collaboration profiles: we propose 7 profiles to capture the different collaborative strategies with an application to our puzzle-solving task (Fig. 2).

The proposed collaboration profiles are based on existing literature (Section 2.1) and augmented by quantitative indicators reflecting aspects of the collaboration process (Section 2.2). The observational study highlighted collaboration strategies linked to some of these indicators, allowing us to apply our collaboration profiles to our task while helping to distinguish close collaboration profiles involving variations in collaboration strategies.

Individual strategies are categorized into 3 major profiles according to how collaborators allocate tasks between them: Everything-Nothing (EN), Independent Solitary (SOL), Independent Sociable (SOC). Collective strategies are decomposed in 4 profiles: Leader-Follower (LF), Teacher-Student (TS), Turn-takers Non-Accurate model (TTNA), and Turn-takers Accurate model (TTA). Shaer's Turn-takers profile [50] was divided into two profiles (TTA and TTNA) based on the relevance of
<figure><img src="image_5.png" /><figcaption>Fig. 2. Proposed collaboration profiles applied to our task: the presence of chat boxes indicates that participants discuss, and the number of hyphens indicates the intensity of the verbal participation. Red rays indicate interaction with the puzzle. The clouds represent individuals' current mental model of the situation.</figcaption></figure>

the collaborators' common ground. Similarly, the Independent profile [50] was split into two profiles (SOL and SOC), depending on whether individuals considered each other during the activity.

Based on the literature and our observations, each profile is characterized by a set of indicators (Table 2). A detailed description of the profiles is available in Appendix Part A.

# 5 Profile Assessment in Stereotypical Scenarios

Having defined our set of collaboration profiles from a set of naive participant interactions with our experimental framework, we designed several scenarios, enacted with actors, in order to record a dataset of controlled collaboration instances in our virtual reality collaborative puzzle-solving task. Two actors (2M, ages: 26 and 51) played the collaborative scenarios. They first discovered the script with instructions for a specific scenario. Then, actors played two instances for each. The first instance served as a trial to understand their roles; only the second instance was kept and analyzed.

We use this dataset to test a preliminary detection approach of the 7 dynamic collaboration profiles introduced earlier. The rationale behind the use of actors is to rapidly obtain a set of collaboration recordings with coherent, stereotyped collaborative profiles, leveraging the actors' personal understanding of collaboration styles to faithfully enact each of the 7 profiles. We expect to find stark differences in indicators between these stereotypes enactments, compared to more subtle differences using a naturalistic experiment; this enables us to rapidly evaluate a proof-of-concept of our profile assessment approach.

## 5.1 Recorded scenarios and instructions design

For each scenario, we provide actors with a sequence of instructions for each sub-task in order to elicit a particular collaboration profile (instructions in Appendix Part B). These instructions concern: (i) *interaction with the puzzle pieces* (retrieving, positioning, coloring puzzle pieces or not: which one? how many?); (ii) *consideration of the other*, such as looking at his avatar or actions; (iii) *talking to the other*, in terms of frequency of discussion, type of discourse and exchanges; (iv) *proximity to the other*; and (v) *performing specific gestures*, such as showing a specific task-related element. The instructions were designed based on strategies observed during the elicitation study (Section 4), including communication, coordination, awareness, movement, proximity, and updating the common ground. Specific indicators from observations, taxonomies, and literature on collaboration profiles (Table 2) were considered. For example, in scenarios requiring awareness of others, instructions addressed joint visual attention and gaze on peers, while scenarios with low awareness specified the absence of these indicators. Instructions also addressed dominance in speech and task participation, detailing each users' participation, to ensure appropriate levels of (in)equality in discussion and actions. Coordination strategies were tailored to promote either active or minimal coordination, and instructions are specified regarding individuals movement's in the room according to the profiles.
<table><thead><tr><th>Profile and collaborative behaviors</th><th>Dominance</th><th>Joint attention</th><th>Communication processes management</th><th>Spatial behaviors</th><th>Speech content</th><th>Group characteristics</th></tr></thead><tbody><tr><td>Everything-Nothing (EN)</td><td>Null Speech equality: Both E and N don't participate<br>Inequal Task equality: E realized most part of the activity (Disengage [53])</td><td>Ø joint visual attention → High JVA: free rider [56]<br>Low people gaze and mutual gaze</td><td>Low turn-taking, interruption and overlap frequency<br>Low verbal participation*</td><td>Ø physical proximity/formation<br>Ø physical synchrony<br>Ø gestures (referential, pointing, etc.)</td><td>Ø or low speech<br>Nature of discussion task-related and non task-related<br>E doesn't explain his actions and ignore or doesn't accept suggestions from N [50]</td><td>E actively engaged compared to N [50]<br>Rapport: E and N do not know each other and/or are not very close*<br>N doesn't have expertise while E can have it*</td></tr><tr><td>Independent society (SOL)</td><td>Null Speech equality: Both don't participate verbally<br>Tend to have Equal Task equality</td><td>Ø JVA<br>Ø or low people gaze<br>Ø mutual gaze</td><td>Ø or low turn-taking, interruption and overlap frequency<br>Ø or low verbal participation [50]</td><td>Ø physical proximity/formation<br>physical synchrony of users performed their task individually<br>Ø gestures (referential, pointing, etc.)</td><td>Ø speech<br>Ø or low verbal information sharing [50]</td><td>Both users equally engaged*<br>Rapport: Users do not know each other and/or are not very close*, not social</td></tr><tr><td>Independent society (SOC)</td><td>Null Speech equality: Both don't participate verbally<br>Tend to have Equal Task equality</td><td>JA level: Individual/Monitoring attention [51]<br>High people gaze [50]*<br>Medium JVA and mutual gaze*</td><td>Ø or low turn-taking, interruption and overlap frequency*<br>Ø or low verbal participation [50]</td><td>Possible physical proximity/f-formation<br>physical synchrony of users performed their task individually<br>Ø gestures (referential, pointing, etc.)</td><td>Ø speech<br>Ø or low verbal information sharing [50]*</td><td>Both users equally engaged*<br>Rapport: Users know each other and/or are close*, social</td></tr><tr><td>Leader-Follower (LF)</td><td>Inequal Speech equality: L speak more than F [47]<br>Inequal Task equality: L participates more than F [47] (Same problem or Discussion [53]*</td><td>JA level: Common attention [51]<br>L received more people gaze than F [10]<br>Medium JVA<br>L drives visual attention [10]<br>Low mutual gaze</td><td>Low turn-taking, interruption and overlap frequency<br>Medium verbal participation</td><td>Physical proximity / formation</td><td>L managed the task giving strategy and instructions, F answers and makes suggestions [50]</td><td>L and F are both engaged but more L<br>L have more expertise than F*</td></tr></tbody></table>
<table><caption>Table 2. Collaboration profiles with corresponding indicators for each aspect contributing for collaboration process. An * indicates a proposal based on our observations during the collaborative puzzle-solving task and derived from other behavioral indicators observed in the literature.</caption><tbody><tr><td>Teacher-Student (TS)</td><td>Inequal Speech equality: T speak more than S<br>Inequal Task equality: S realized most part of the task [50] (Same problem or Discussion [23])</td><td>JA level: Mutual attention [51]<br>High JVA*<br>High people gaze<br><b>T drives visual attention*</b><br>Possible mutual gaze</td><td>Low turn-taking, interruption and overlap frequency<br>Medium verbal participation</td><td>Physical proximity / formation<br>Referential and pointing gestures by T [50]<br>Physical synchrony (active posture) [46]</td><td>Nature of discussion mostly task-related<br>T makes suggestions and observations while S listens and realized actions [50]<br><b>Task repartition: explicit at the start of the task</b></td><td>Both users equally engaged [50]<br>T have more expertise than S*<br><b>Task repartition: explicit at the start of the task</b></td></tr><tr><td>Turn-takers Non-accurate mental model (TTNA)</td><td>Equal Speech equality</td><td>JA level: Shared attention [51]<br>Medium JVA<br>High people gaze<br>Possible mutual gaze<br>Attention not always focused on relevant task element, but considered as relevant =&gt;Misunderstanding</td><td>High verbal participation [28]<br>High turn-taking [28], and success interruption frequency</td><td>Physical proximity / formation and interaction space [23]<br>Referential and pointing gestures by both users [50]<br>Physical synchrony (active posture) [46]</td><td>Nature of discussion mostly task-related<br>Ø critical discussion*<br>Misconception [1]</td><td>Both users equally engaged<br>Rapport: Users know each other*<br>Both users have same expertise*</td></tr><tr><td>Turn-takers Accurate mental model (TTA)</td><td>Equal Speech equality</td><td>JA level: Shared attention [51]<br>High JVA<br>High people gaze<br>Possible mutual gaze</td><td>High verbal participation [28]<br>High turn-taking [28], and success interruption frequency</td><td>Physical proximity / formation and interaction space [23]<br>Referential and pointing gestures by both users [50]<br>Physical synchrony (active posture) [46]</td><td>Nature of discussion mostly task-related<br>Critical discussion on each user's understanding and actions [34]<br>Both users accept and make suggestions and observations [50]<br><b>Task repartition: explicit at the start of the task and updated</b></td><td>Both users equally engaged<br>Rapport: Users know each other*<br>Both users have same expertise*</td></tr></tbody></table>
When formulating the instructions, we avoided specifying exact behavior frequencies. We aimed to provide clear yet flexible guidelines for actors to interpret accurately the scenario while remaining flexible to reflect real collaborative situations. Specifying frequencies would have made it challenging for actors to perform the scenarios effectively. Instead, we provided general instructions by phase or sub-phase, related to strategies for each indicator. For task participation, instructions were given at specific moments to reflect observed collaborative situations. Overall, the instructions indirectly accounted for indicator occurrences for each profile during the activity phases.

Each scenario was defined in line with the profiles proposed in Section 4.2. Due to certain characteristics of the activity sub-tasks, variants have been designed for the scenarios of certain profiles. While one user could collect and position all the pieces on their own in the piece retrieval and positioning sub-tasks, the piece coloring sub-task requires both users to interact and be active in solving the task. Therefore, we have a total of 11 collaboration scenarios: (1) Everything-Nothing to Independent Solitary (EN-SOL); (2) Everything-Nothing to Independent Sociable (EN-SOC); (3) Everything-Nothing to Leader-Follower (EN-LF); (4) Independent Solitary (SOL); (5) Independent Solitary to Independent Sociable (SOL-SOC); (6) Independent Sociable (SOC); (7) Independent Sociable to Turn-Takers Accurate model (SOC-TTA); (8) Leader-Follower (LF); (9) Teacher-Student (TS); (10) Turn-Takers Non Accurate model (TTNA) and (11) Turn-Takers Accurate model (TTA). The collaboration scenarios' datasets, including the multimodal data mentioned in Section 3.2, can be found here<fnref n="1" />.

## 5.2 Profile computation

Based on the taxonomy of profiles we presented in Section 4, we propose an algorithm to detect the dyad's collaboration profile during the activity. To do this, we use for each profile certain "discriminant" indicators (red in Table 2), key elements of this particular profile. Given a particular time window of analysis, the absence of a discriminant indicator automatically implies that the corresponding collaboration profile is ruled out, even if other corresponding indicators are validated.

Currently, our profile computation algorithm is limited to indicators including: dominance, with the distribution of speech and contribution to the task; verbal coordination, with the occurrence of turn-taking with overlap and verbal participation events; and joint attention, with the occurrence of joint visual attention events. Indicators related to users' posture, gestures or speech, are being implemented. In addition, we calculate profile-specific indicators (blue in Table 2). These cues are complementary to the main cues for certain collaboration profiles and their presence can help confirm the collaboration profile detection.

**Indicators normalization.** Each indicator is normalized to a [0, 1] range. The speech and task equality indicators correspond to the Gini index of verbal and task participation of the two individuals over the same time window.

**Indicators range.** To differentiate the indicators between each profile, we define 5 categories of intervals for each indicator. These range from 0 to 1: low (0 - .25); mid-low (.25 - .50); mid-high (.50 - .75) to high (.75 - 1). A 5th category called "null" includes indicators that depend on the degree of verbal participation. If it is below the threshold of .125, we consider that individuals speak insufficiently, and that there is not enough data for the speech equality indicators to be accurate. Based on collaboration profiles, we proposed value ranges for indicators of each profile (Table 3). These ranges correspond to expected values of the collaboration indicators according to profiles.

**Profile Confidence.** Finally, our profile computation algorithm calculates a *Profile confidence* metric which provides a holistic view of proportion of indicators currently in favor of a particular

<footnote n="1">¹https://doi.org/10.57745/EXEE2Z</footnote>
<table><caption>Table 3. Normalized indexes range for collaboration profiles</caption><thead><tr><th>Profiles\Indicators</th><th>Speech Equality</th><th>Task Equality</th><th>Turn-taking with Overlap</th><th>Verbal Participation</th><th>JVA</th></tr></thead><tbody><tr><td>Everything-Nothing</td><td>null</td><td>0.50 &lt; x</td><td>0 &lt; x &lt; 0.25</td><td>0 &lt; x &lt; 0.25</td><td>0 &lt; x &lt; 0.25</td></tr><tr><td>Independent Solitary</td><td>null</td><td>0 &lt; x &lt; 0.25</td><td>0 &lt; x &lt; 0.25</td><td>0 &lt; x &lt; 0.25</td><td>0 &lt; x &lt; 0.25</td></tr><tr><td>Independent Sociable</td><td>null</td><td>0 &lt; x &lt; 0.25</td><td>0 &lt; x &lt; 0.25</td><td>0 &lt; x &lt; 0.25</td><td>0.25 &lt; x &lt; 0.50</td></tr><tr><td>Leader-Follower</td><td>0.50 &lt; x &lt; 0.75</td><td>0.50 &lt; x &lt; 0.75</td><td>0 &lt; x &lt; 0.25</td><td>0.25 &lt; x &lt; 0.50</td><td>0.25 &lt; x &lt; 0.50</td></tr><tr><td>Teacher-Student</td><td>0.25 &lt; x &lt; 0.50</td><td>0.25 &lt; x &lt; 0.50</td><td>0.25 &lt; x &lt; 0.50</td><td>0.25 &lt; x &lt; 0.50</td><td>0.50 &lt; x &lt; 0.75</td></tr><tr><td>Turn-takers: Non-Accurate Mental Model</td><td>0 &lt; x &lt; 0.25</td><td>0.25 &lt; x &lt; 0.50</td><td>0.50 &lt; x &lt; 0.75</td><td>0.25 &lt; x &lt; 0.50</td><td>0.25 &lt; x &lt; 0.50</td></tr><tr><td>Turn-takers: Accurate Mental Model</td><td>0 &lt; x &lt; 0.25</td><td>0 &lt; x &lt; 0.25</td><td>0.50 &lt; x &lt; 0.75</td><td>0.25 &lt; x &lt; 0.50</td><td>0.50 &lt; x &lt; 0.75</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>Fig. 3. Computation and update of collaboration profile confidence. Indicators in blue are available, and those in orange are being implemented.</figcaption></figure>

collaboration profile. Profile confidence is updated in two steps (Fig. 3). Firstly, the indicators ranges are updated according to the current value of the corresponding normalized indicator. Then, the confidence level of each profile is calculated for the last 20-second segment. We first check whether the discriminant indicators for any considered profile are validated. If not, the collaboration profile is ruled out. If at least one discriminating indicator is validated, we check whether the other and specific indicators are validated. If so, they will increase confidence in the collaboration profile. As an output, the level of confidence in the collaboration profile takes the form of a value from 0 to 1, with 1 indicating that all corresponding indicators are in agreement with the expected values for this profile, thus expressing a high level of confidence in the detection of this collaboration profile. We define a threshold for profile detection when any collaboration profile's confidence level equals or exceeds 0.5. This condition is met when at least one discriminating indicator is validated, along with either two other indicators or one other indicator and two specific indicators.

## 5.3 Hypothesis

We advance several hypotheses concerning the validation of our collaboration profile detection approach using this set of collaboration scenario enactments.

**H1:** The profile detection model achieves a good classification performance score (F-Score) across the different collaboration scenarios.
<table><caption>Table 4. Profile detection performance against expectations within all stereotyped collaboration scenarios. The positive and negative rows indicates the number of true and false detection regarding the expectations. Red indicate a bad performance detection while orange a medium.</caption><thead><tr><th>Stats\Profile</th><th>EN</th><th>SOL</th><th>SOC</th><th>LF</th><th>TS</th><th>TTNA</th><th>TTA</th></tr></thead><tbody><tr><th>Precision</th><td>0,50</td><td>0,36</td><td>0,31</td><td>0,17</td><td>0,28</td><td>0,19</td><td>0,65</td></tr><tr><th>Recall</th><td>0,56</td><td>0,90</td><td>0,39</td><td>0,12</td><td>0,28</td><td>0,50</td><td>0,49</td></tr><tr><th>F-Score</th><td>0,53</td><td>0,52</td><td>0,35</td><td>0,14</td><td>0,28</td><td>0,27</td><td>0,56</td></tr></tbody></table>

**H2:** The percentage of profile detection is higher when the tested profile corresponds to the expected profile in the phase of the collaboration scenario.

**H3:** The average detection confidence value for a profile is higher when the tested profile corresponds to the expected profile in the phase of the collaboration scenario.

# 6 Results

We first present the results concerning the performance of the profile detection approach for each profile, then explore the detection of each profile in the different collaborative scenarios and compared it with the other profiles.

## 6.1 Detection model performance

A detection performance index (F-Score) was calculated for each profile. Detection considered for this analysis were those with a confidence value greater than or equal to 0.5. Based on these criteria, each detection was coded as 0 (less than 0.5) or 1 (greater than or equal to 0.5). Each stereotyped collaboration scenario is characterized by known expectations in terms of profile detection during the activity. These expectations were coded 0 (detection not expected) and 1 (detection expected). Profile detection performance was then assessed by computing the Profile Detection F-Score over all scenarios. This score is calculated from the values for accuracy (percentage of correct positive detections) and recall (percentage of positives correctly predicted by the model), itself calculated from the true positives, false positives, true negatives and false negatives of detection. Results of our profile performance is display in Table 4.

Overall, the results show a low detection performance of the profiles in the different scenarios. The detection performance reflected by the F-Score indicates that the best detected profiles are the EN, SOL and TTA profiles, with detection performance ranging from 0.50 to 0.60 (EN: 0.53; SOL: 0.52; TTA: 0.56). The LF profile has the lowest performance score (0.14) and the SOC, TS and TTNA profiles have a detection performance between 0.25 and 0.35 (SOC: 0.35; TS: 0.28; TTNA: 0.27). These results does not allow us to validate **H1**.

The low detection performance may be due to several reasons, including the fact that only one collaboration profile is supposed be detected for a specific phase of the activity in the coding of the scenarios. This does not fully consider the dynamic nature of the collaboration process and therefore of the collaboration profile during the task. The interest of this approach is to demonstrate that collaboration profiles can be detected with a high level of confidence at specific moments in the activity (in this case, phases), even if they are not detected over the entire phase (see Section 6.2). Adjustment of indicator weights and ranges, as well as confidence threshold filtering, may be required to improve and accurately interpret detection results.

## 6.2 Exploration of profile detection within the collaborative scenario

Certain scenarios have specific considerations regarding their phases. For example, in the SOL scenario, individuals, who did not consider the other, initiated coloring the puzzle without verifying that all pieces were positioned. For the SOC-TTA, TTNA and TTA scenarios, participants adopted a strategy of gradually coloring the puzzle while adding pieces. Consequently, the positioning and
<figure><img src="image_7.png" /><figcaption>Fig. 4. Confidence value of each profile according to task phase for the Turn-takers Non-Accurate model scenario, for a profile detection based on a sliding average with a 20 second threshold and a profile detection each 5 second. The dotted red line at 0.5, indicates the confidence threshold at which a profile is considered to be detected. The ground truth corresponds to the profile we expect to detect.</figcaption></figure>

coloring phases of these scenarios are merged into one phase. The scenario TTNA is characterized by the presence of an additional phase, called "Breakdown". This phase corresponds to the presence of an incident disrupting collaborative activity due to a collaboration problems.

We thus explore the values of the detection confidence level for each profile through the different scenarios. The profile detection was analyzed concerning the various stereotyped collaboration scenarios and the expected detection results for these scenarios. A summary of the results for each profile detected are presented in this section. Detailed figures, detection expectation and results of profile detection within all the scenarios can be found in Appendix Part C. As an example, we presented the complete analysis for the TTNA scenario and its corresponding profile.

Fig. 4 shows the collaboration profiles detection results for the TTNA scenario. In this scenario, the TTA profile is expected to be well detected during the retrieving phase with few or no profile detection. We then expect to observe a decrease in the detection of the TTA profile during the first positioning and coloring sub-phase, with an increase in the detection of the TTNA profile. This leading to the emergence of a collaboration issue and therefore the breakdown phase. We then expect to switch to a SOL profile during the breakdown phase, with no other profiles detected. Finally, the last sub-phase should be characterized by a return to a relevant common model of the situation, associated with the detection of the TTA profile, and the absence or low detection of any other profile. In general, it is expected that a low percentage of LF, TS, EN and SOC profiles will be detected throughout the task.

**Turn-takers Non-Accurate Mental Model.** Regarding the detection results for this collaboration scenario, during the retrieving phase, the TTA profile is detected with a better max and mean confidence value than all other profiles including the SOL and LF profiles that have been detected once or few time at 0.5, but with a lower mean confidence value on the phase. Positioning and coloring sub-phase 1 is characterized by a higher mean confidence value of the TTNA profile compared to all detection on this phase. The max detected value was the same for the TS, TTA and TTNA profiles on this sub-phase, but the mean confidence value of the TS and TTA profiles detection was lower. In the breakdown sub-phase, the SOL profile have the higher max and mean confidence detection value compared to all profile. Other profile such as the SOC profile is detected
with a value of .67, but with a mean detection value near 0. Finally, in the last positioning and coloring 2 sub-phase, the TTNA and TTA profiles were the only detected, the last a higher mean confidence value of 0.6.

**Everything-Nothing.** The EN profile is best detected in the "retrieval phase" of the EN-SOL, EN-SOC, and EN-LF scenarios, with a higher mean confidence values in EN-SOL and EN-SOC. In the two following phases, the profile isn't detected or its mean confidence value is near 0. In other scenarios, the profile isn't detected or with a mean confidence value near 0.

**Independent Solitary.** The SOL profile detection is higher compared to other profiles in the SOL scenario. Its detection is sometimes associated with detections of comparable confidence for the EN and SOC profiles. The profile has a high rate of false positives in specific scenarios where SOC is expected to be detected. In the other scenarios, SOL is frequently detected in the retrieving phase except in TS, TTA, and TTNA scenarios.

**Independent Sociable.** The profile is poorly detected in relevant scenarios, compared to other profiles such as SOL. In other scenarios, the SOC profile is not detected, or with some false positives in the LF, EN-LF and EN-SOL scenarios.

**Leader-Follower.** The profile is poorly detected in relevant scenarios. Additionally, it has a high false positive detection rate in the coloring phase of the TS scenario and occasionally during phases of the TTNA scenario.

**Teacher-Student.** The TS profile is detected in its corresponding scenario, but with a lower mean confidence value in the retrieving and coloring phases, respectively compared to TTA and LF profiles. It reaches a higher max confidence value detection in the retrieving and positioning phases of its scenario. The TS profile is detected in most of the scenarios, but with a low mean confidence value (< 0.25) and is more frequently detected in some phases of the TTA and LF scenarios.

**Turn-takers Accurate Mental Model.** This profile is well detected in the different phases of its scenarios (TTA and SOC-TTA). However, the profile is often falsely detected with a low mean confidence value in scenarios where the SOC profile is expected to be detected, and with a highest mean confidence value in last phase of the LF scenario. Lastly, TTA and TTNA profiles seem to be associated because of their joint detection in various scenarios.

## 6.3 Profile detection validation

The detection of EN, SOL, and TTA profiles is accurate within their respective scenarios. Nevertheless, these profiles are occasionally detected jointly with other profiles (SOL for EN and vice versa, and TTNA and TS/LF for TTA). This complicates the differentiation of the current collaboration profile. Consequently, we cannot validate **H2**. The EN, SOL and TTA profiles are mostly detected with the higher mean confidence value than those of other profiles when they are expected to be detected. These results tend to support **H3** for these profiles.

For the SOC, LF, and TS profiles, the results indicate poor detection and lower mean confidence values compared to other profiles in expected detected phases. These findings suggest an ineffective profile detection, leading to the rejection of hypotheses **H2** and **H3**.

Finally, the detection of TTNA profiles supports **H2** regarding profile detection in the phases of the corresponding scenario. However, the mean confidence value for TTNA detection profiles is lower in expected phases than those of other detected profiles, indicating that we cannot validate **H3** for the detection of this profile.
# 7 Discussion and limits

## 7.1 Detection approach and collaboration dynamics

A first observation concerns our collaborative task and the detection of the SOL profile during the retrieval phase in most of the collaboration scenarios. We hypothesize this phase fosters an individual mode of collaboration, unlike the other phases that may necessitate collaborative and joint actions. This shift is reflected by a decrease in the detection and confidence value of EN, SOL, and SOC profiles compared to the first phase, except in scenarios corresponding to these profiles.

Concerning the detection efficiency of our proposed profiles, our results indicate a generally poor detection performance when applied to stereotyped collaboration scenarios. Specifically, LF, TS, and SOC profiles detection failed, as their detection seemed impossible with our current approach and set of indicators. However, EN, SOL, TTNA, and TTA profiles were accurately detected in the corresponding scenarios, although with reduced global accuracy scores due to false positives in phases where their detection are unexpected. Furthermore, some profiles were often jointly detected (EN, and SOL; TTA, TS, and TTNA), potentially due to their similarity (EN, SOL: non-collaborative; TTA-TTNA: collaborative) and their shared indicator sets and detection intervals (Table 3).

In scenarios involving EN, SOL, and SOC profiles exclusively, we observe minimal or no detection of other collaboration profiles, except in scenarios involving the emergence of other profiles. A similar observation in TTNA and TTA scenarios, where the detection of EN, SOL, and SOC profiles is absent or minimal. Our detection approach effectively distinguishes between non-collaborative (EN, SOL) and highly collaborative profiles (TTNA, TTA) but failed in detecting profiles situated between these extremes (SOC, LF, TS). This point aligns with findings from Martinez-Maldonado et al. [33], emphasizing that it is simpler to distinguishing collaboration levels at the extremes while the complexity often lies in evaluating collaboration between these extremes. This issue, coupled with the interconnection between profile detection may explain the presence of other collaboration profiles detection in various scenarios. Another explanation could be that the dynamic nature of collaboration during an activity, where individuals pass through multiple collaboration profiles, overshadows any attempt by actors to portray a particular type of collaboration profile.

Other reasons for the presence of false positives and joint detection can be attributed to our system: the **relevance of the indicators employed** for profile detection; the system may **lack enough indicators** to accurately identify profiles; the **intervals used might not be specific enough** for precise profile detection; or our **detection approach isn't sufficiently complete** or too simplistic to capture the current collaboration profile. The first detection iteration outlined in this paper relies on relatively high-level indicators, using their valence to differentiate between collaboration profiles using ad-hoc intervals. Relying on shared indicators introduces a limitation by increasing the risk of interconnection, leading to false-positives, as observed in various scenarios. Future research could consider an approach that incorporates only profile-specific indicators of collaboration quality. Examples include indicators related to discourse (quality, nature) [1, 34, 50], or the spatial arrangement between individuals [23], as presented in Table 2. Regarding the intervals used (Table 3), our results indicate their relevance for certain profiles (EN, SOL, TTA, TTNA) but suggest potential limitations for TS, LF, and SOC profiles due to lower detection performance. These intervals could definitely be refined using a larger and more naturalistic dataset.

Taken as a whole, the results highlight that our preliminary profile filtering approach is not sufficiently effective. Indeed, we have observed that certain profiles exhibit high-confidence detections in certain phase, but with an average confidence value for this phase close to 0. Our current approach involves calculating indicators from the last 20 seconds of the task to determine the confidence level in a collaboration profile. However, it does not consider previous detections. Incorporating temporal information, such as with an additional filtering step that considers earlier
detections could enhance the reliability of collaboration profile detection. For instance, if a profile is detected with high confidence at a specific moment but the mean confidence value of past detections for that profile is close to 0, it may be a false positive. In such cases, we consider true confidence level to be 0. However, if following detections show a high confidence level for this profile, we can confidently identify the collaboration profile. Implementing this filtering in our approach might make it possible to filter out the false detections observed currently. This approach presents an intriguing perspective for future research. Generally, for both this aspect and for the above limitations related to our set of indicators, machine learning approaches may provide an efficient alternative for indicator selection and representation, and automatic profile inference.

## 7.2 Study limitations and future research

In this study, we designed stereotyped collaborative situation datasets, reflecting collaboration profiles observed in the literature and applied them to our collaborative task. Although these scenarios are based on exhaustive observations of collaboration, one of the main limitations of this study is the **validation of collaboration profile detection on simulated rather than real collaborative situation data**. We were unable to test our profiles detection on the sessions observed due to different data recording setups, making it impossible to calculate collaboration indicators used in our approach. A next step in this research would be to test the profiles detection on non-emulated and annotated collaboration scenarios to assess the quality of our detection approach with ground truth. Another limitation is the **small sample size of profile detection for each phase**, linked to the duration of the collaborative task. This collaboration profile detection approach should be tested in collaborative tasks with longer collaboration phases. Furthermore, we chose to use a 20-second threshold to evaluate the current collaboration profile. It may be relevant to test other thresholds to study the **influence of the threshold on the quality of collaboration profiles detection**. The threshold should also be adapted to the duration of the collaborative task.

Another point to consider is the **difference between collaboration in virtual reality (VR) and physical space**, particularly the use of physical space collaboration indicators (see Section 2) for assessing co-located collaboration in VR. This raises questions about the transferability and relevance of these indicators in a VR context. Collaboration in VR presents its own set of challenges compared to physical space. According to the CSCW time-space matrix [25], collaborative virtual environments can be categorized into (a)-synchronous co-located and (a)-synchronous distributed. Asynchronous or distributed VR setups can limit verbal and nonverbal cues, affecting team effectiveness [18]. We used a synchronous co-located VR environment, enabling direct communication among collaborators in a shared virtual space. Regarding these aspects, our collaborative VR task closely simulated similar tasks than in physical space. However, factors like avatar realism can impact collaboration in CVE. Emotionally expressive avatars can influence individuals by inducing a higher sense of embodiment and enhance engagement and performance in tasks [17]. Conversely, the absence of avatar realism, especially the lack of nonverbal cues such as postural, gestural, or facial expressions, can negatively impact collaboration by reducing information about collaborators, lowering the sense of embodiment, and affecting individuals' behavior compared to real-life collaborative situations. In our task, the avatars might have influenced interest in others' avatars and consequently on the indicators relating to the gathering of information about collaborators, such as mutual gaze, and gaze on peers. Certain aspects, such as allowing physical movement and co-located audio, may have counterbalanced the low realism of our avatars, minimizing the impact on users' sense of embodiment. Future studies should control this variable by assessing individuals' sense of presence in the virtual environment.

The collaboration profiles and metrics considered in our detection approach are derived from existing literature on collaboration, which is not specific to the VR context. Therefore, it would be
highly relevant to apply this approach to similar contexts, such as physically co-located collaborative
settings. However, the applicability of our approach to real-world contexts may be limited by the
available tools and sensors for measuring certain collaboration metrics included in the approach.
While emerging hardware tools such as Meta's Aria glasses<fnref n="2" /> or software using machine learning
are promising, they are not yet reliable enough for real-time use. Capturing spatial measures,
joint visual attention, or task participation in physical collaboration contexts can be challenging
without intrusive instrumentation of individuals. A preliminary application of this approach to other
collaborative contexts could involve the use of physical devices that directly capture relevant user
behavioural data, such as interactive tables, interactive displays, or computer-based collaborative
tasks, where user activity traces can be easily captured and used to derive collaboration metrics
during the activity.

Another limitation of this study is the **generalizability** of our approach and results to **other collaborative contexts**. McGrath's circumplex model of group tasks [35] distinguishes conceptual from behavioral tasks and cooperative from conflict tasks. We focused on small group collaboration involving a physical and cognitive task requiring coordination to achieve a common goal, exemplified by a puzzle-solving task. Our task falls at the intersection of behavioral and cooperative tasks and is specifically categorized as a Planning task in McGrath's model. Considering that our detection approach focuses on a specific type of collaboration, it would be valuable to investigate its validity and applicability in collaborative contexts involving different settings such as other tasks, asynchronous or remote collaboration, or scenarios involving larger groups. Future studies should then apply this approach to collaborative tasks present on other dimensions of the McGrath model, such as production, brainstorming, decision-making, and negotiating tasks. In this way, they could explore how to adapt or extend our assessment approach to these contexts, including identifying relevant metrics to measure collaboration quality and collaboration profiles.

Additionally, our participant sample has limitations, notably that pairs knew each other before
the collaborative activity. This proximity may have influenced the type of collaboration observed,
as pre-existing relationships and attitudes likely determined participants interactions. This issue
doesn't arise when individuals don't know each other. Future studies should consider this factor
during recruitment and evaluate its impact on the collaboration activity and detected profiles.

It could also be interesting to study the relevance of detecting these dyadic profiles in the
text of a collaborative task involving more than 2 individuals, due to the differences in terms of
collaboration that can be induced by the size of the group. In this way, we could study the quality
of collaboration between individuals in pairs.

8 Conclusion

Using quantitative multimodal signals to assess collaboration and provide real-time feedback is an emerging topic of interest for the CSCW community. One method to evaluate changes in collaboration during activities is to categorize participants’ collaboration profiles. This study addresses several key contributions to this topic.

A primary contribution is an augmented taxonomy of collaboration profiles that incorporates
indicators of the collaboration process, calculated from multimodal signals collected during the
activity. We also proposed a collaboration profile detection approach based on behavioral indicators
defined from the existing literature and observation of our collaborative task. This approach is
characterized by its ease of replicability and implementation for the analysis of collaboration in
various collaborative contexts. It allowed us to explore a first version of a model that considers
multimodal behavioral indicators, for a set of collaboration profiles. Our aim is to validate that, with

<footnote n="2">²Project Aria Website: https://www.projectaria.com/</footnote>
this simple approach, differences can be identified between collaboration segments corresponding
to different profiles. This aspect was partially validated, with results indicating that extreme profiles
could be distinguished from each other more easily than more subtle, balanced profiles. Our mixed
results may be due to several factors, such as the use of shared indicators to detect profiles, the
ad-hoc mapping of profile indicator ranges, and the use of enactments of collaboration rather than
naturalistic annotated data. The results also highlight the dynamic nature of the collaboration
profiles observed throughout the activity. For a given dyad, profiles may vary according to the
sub-task solving strategy judged to be the most efficient.

In conclusion, although the results of this study must be nuanced by the use of stereotyped
collaborative scenarios, they validated the general approach that the automatic collection of multi-
modal behavior indicators enables the recognition of some collaboration profiles. An improved
set of behavioral indicators with more precise detection ranges may be sufficient to discriminate
close profiles. We aim to test other indicators in a future study with an enhanced collaboration task,
additional components considering other dimensions of collaboration to improve profile detection,
and a more naturalistic set of participant groups. Finally, we also provide a dataset of recorded
multimodal signals corresponding to various collaborative scenarios. This dataset could enable
other researchers to test detection methods and compare them with our results.

Acknowledgments

This work was conducted as part of a doctoral thesis, supported by ANR Missives (ANR-20-CE33-0007) and a grant from the Pays de la Loire region (Convention no. 2021-12881).

References

[1] Rotem Abdu. 2015. The Development of Situational-Misconceptions in Math Problem Solving. In *11th International Conference on Computer Supported Collaborative Learning, CSCL 2015, Gothenburg, Sweden, June 7-11, 2015*, Oskar Lindwall, Päivi Häkkinen, Timothy Koschmann, Pierre Tchounikine, and Sten R. Ludvigsen (Eds.). International Society of the Learning Sciences, Gothenburg, Sweden. https://repository.isls.org/handle/1/393

[2] Henny Admoni and Brian Scassellati. 2017. Social Eye Gaze in Human-Robot Interaction: A Review. *Journal of Human-Robot Interaction* 6, 1 (2017), 25. doi:10.5898/JHRI.6.1.Admoni

[3] Brittany L. Anderson-Montoya and Mark W. Scerbo. 2019. Human Factors Psychology in Surgery. In *Comprehensive Healthcare Simulation: Surgery and Surgical Subspecialties*, Dimitrios Stefanidis, James R. Korndorffer, and Robert Sweet (Eds.). Springer International Publishing, Cham, 153–167. doi:10.1007/978-3-319-98276-2_14

[4] Ferran Argelaguet, Alexander Kulik, André Kunert, Carlos Andujar, and Bernd Froehlich. 2011. See-through techniques for referential awareness in collaborative virtual reality. *International Journal of Human-Computer Studies* 69, 6 (2011), 387–400. doi:10.1016/j.ijhcs.2011.01.003

[5] Ignacio Avellino, Gilles Bailly, Geoffroy Canlorbe, Jérémie Belgihti, Guillaume Morel, and Marie-Aude Vitrani. 2019. Impacts of Telemanipulation in Robotic Assisted Surgery. In *Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19)*. Association for Computing Machinery, New York, NY, USA, 1–15. doi:10.1145/3290605.3300813

[6] Khaled Bachour, Frederic Kaplan, and Pierre Dillenbourg. 2010. An Interactive Table for Supporting Participation Balance in Face-to-Face Collaborative Learning. *IEEE Transactions on Learning Technologies* 3, 3 (2010), 203–213. doi:10.1109/TLT.2010.18

[7] Roger Bakeman and Lauren B. Adamson. 1984. Coordinating Attention to People and Objects in Mother-Infant and Peer-Infant Interaction. *Child Development* 55, 4 (1984), 1278–1289. doi:10.2307/1129997

[8] Dan Bohus, Sean Andrist, Ashley Feniello, Nick Saw, Mihai Jalobeanu, Patrick Sweeney, Anne Loomis Thompson, and Eric Horvitz. 2021. doi:10.48550/arXiv.2103.15975 Platform for Situated Intelligence.

[9] Pankaj Chejara, Luis P. Prieto, Maria Jesus Rodriguez-Triana, Reet Kasepalu, Adolfo Ruiz-Calleja, and Shashi Kant Shankar. 2023. How to Build More Generalizable Models for Collaboration Quality? Lessons Learned from Exploring Multi-Context Audio-Log Datasets Using Multimodal Learning Analytics. In *LAK23: 13th International Learning Analytics and Knowledge Conference* (, Arlington, TX, USA,,) (LAK2023). Association for Computing Machinery, New York, NY, USA, 111–121. doi:10.1145/3576050.3576144

[10] Joey T. Cheng, Fabiola H. Gerpott, Alex J. Benson, Berno Bucker, Tom Foulsham, Tessa A.M. Lansu, Oliver Schülke, and Keiko Tsuchiya. 2022. Eye gaze and visual attention as a window into leadership and followership: A review of
empirical insights and future directions. *The Leadership Quarterly* (Oct. 2022), 101654. doi:10.1016/j.leaqua.2022.101654

[11] Yi Han Victoria Chua, Preman Rajalingam, Seng Chee Tan, and Justin Dauwels. 2019. EduBrowser: A Multimodal Automated Monitoring System for Co-located Collaborative Learning. In *Learning Technology for Education Challenges (Communications in Computer and Information Science)*, Lorna Uden, Dario Liberona, Galo Sanchez, and Sara Rodríguez-González (Eds.). Springer International Publishing, Cham, 125–138. doi:10.1007/978-3-030-20798-4_12

[12] Herbert H. Clark. 1996. *Using language*. Cambridge university press.

[13] Herbert H. Clark and Susan E. Brennan. 1991. Grounding in communication. In *Perspectives on socially shared cognition*. American Psychological Association, Washington, DC, US, 127–149. doi:10.1037/10096-006

[14] Mutlu Cukurova, Rose Luckin, Eva Millán, and Manolis Mavrikis. 2018. The NISPI framework: Analysing collaborative problem-solving from students' physical interactions. *Computers & Education* 116 (2018), 93–109. doi:10.1016/j.compedu.2017.08.007

[15] Yong Dich, Joseph Reilly, and Bertrand Schneider. 2018. Using Physiological Synchrony as an Indicator of Collaboration Quality, Task Performance and Learning. In *Artificial Intelligence in Education (Lecture Notes in Computer Science. 98–110*. doi:10.1007/978-3-319-93843-1_8

[16] Pierre Dillenbourg. 1999. What do you mean by collaborative learning? *Collaborative-learning: Cognitive and computational approaches*. (1999), 1–19. https://hal.science/hal-00190240/en/

[17] Marc Fabri and David Moore. 2005. The use of emotionally expressive avatars in collaborative virtual environments. *Virtual Social Agents* 88, April (2005), 88–94.

[18] Qinna Feng, Heng Luo, Zijian Li, Jiarong Liang, Gege Li, and Yan Yi. 2024. Creating an Immersive Virtual Reality Game Space for Multiuser, Synchronous Co-Located Collaboration: Design Considerations and Influencing Factors. 14, 5 (2024), 2167. doi:10.3390/app14052167

[19] Rowanne Fleck, Yvonne Rogers, Nicola Yuill, Paul Marshall, Amanda Carr, Jochen Rick, and Victoria Bonnett. 2009. Actions speak loudly with words: unpacking collaboration around the table. doi:10.1145/1731903.1731939

[20] Joseph F. Grafsgaard, Joseph B. Wiggins, Kristy Elizabeth Boyer, Eric N. Wiebe, and James C. Lester. 2013. Automatically Recognizing Facial Indicators of Frustration: A Learning-centric Analysis. In *2013 Humaine Association Conference on Affective Computing and Intelligent Interaction*. IEEE, Geneva, Switzerland, 159–165. doi:10.1109/ACII.2013.33

[21] Agustín Gravano and Julia Hirschberg. 2011. Turn-taking cues in task-oriented dialogue. *Computer Speech & Language* 25, 3 (2011), 601–634. doi:10.1016/j.csl.2010.10.003

[22] Boris Groysberg, Jeffrey T. Polzer, and Hillary Anger Elfenbein. 2011. Too Many Cooks Spoil the Broth: How High-Status Individuals Decrease Group Effectiveness. *Organization Science* 22, 3 (2011), 722–737. doi:10.1287/orsc.1100.0547

[23] P. Isenberg, D. Fisher, S.A. Paul, M.R. Morris, K. Inkpen, and M. Czerwinski. 2012. Co-Located Collaborative Visual Analytics around a Tabletop Display. *IEEE Transactions on Visualization and Computer Graphics* 18, 5 (2012), 689–702. doi:10.1109/TVCG.2011.287

[24] Dineshbabu Jayagopi, Dairazalia Sanchez-Cortes, Kazuhiro Otsuka, Junji Yamato, and Daniel Gatica-Perez. 2012. Linking speaking and looking behavior patterns with group composition, perception, and performance. In *Proceedings of the 14th ACM international conference on Multimodal interaction*. ACM, Santa Monica California USA, 433–440. doi:10.1145/2388676.2388772

[25] Robert Johansen. 1988. *GroupWare: Computer Support for Business Teams*. The Free Press.

[26] Georgios Kahrimanis, Anne Meier, Irene-Angelica Chounta, Eleni Voyiatzaki, Hans Spada, Nikol Rummel, and Nikolaos Avouris. 2009. Assessing Collaboration Quality in Synchronous CSCL Problem-Solving Activities: Adaptation and Empirical Evaluation of a Rating Scheme. In *Learning in the Synergy of Multiple Disciplines (Lecture Notes in Computer Science. 267–272)*. doi:10.1007/978-3-642-04636-0_25

[27] Adam Kendom. 1976. The F-formation system: The spatial organization of social encounters. https://www.researchgate.net/figure/Kendons-F-formation-system_fig1_261181296

[28] Jaebok Kim, Khiet P. Truong, Vicky Charisi, Cristina Zaga, Manja Lohse, Dirk Heylen, and Vanessa Evers. 2015. Vocal turn-taking patterns in groups of children performing collaborative tasks: an exploratory study. doi:10.21437/Interspeech.2015-380 In Interspeech 2015, 1645–1649.

[29] Philipp Ladwig and Christian Geiger. 2019. A Literature Review on Collaboration in Mixed Reality. In *Smart Industry & Smart Education*, Michael E. Auer and Reinhard Langmann (Eds.). Vol. 47. Springer International Publishing, 591–600. doi:10.1007/978-3-319-95678-7_65 Series Title: Lecture Notes in Networks and Systems.

[30] Nichola Lubold and Heather Pon-Barry. 2014. Acoustic-Prosodic Entrainment and Rapport in Collaborative Learning Dialogues. In *Proceedings of the 2014 ACM Workshop on Multimodal Learning Analytics Workshop and Grand Challenge (Istanbul, Turkey) (MLA ’14)*. Association for Computing Machinery, New York, NY, USA, 5–12. doi:10.1145/2666633.2666635

[31] Valérie Maquil, Hooriefh Afkari, Béatrice Arend, Svenja Heuser, and Patrick Sunnen. 2021. Balancing Shareability and Positive Interdependence to Support Collaborative Problem-Solving on Interactive Tabletops. *Advances in Human-Computer Interaction* 2021 (2021), 1–17. doi:10.1155/2021/6632420
[32] Roberto Martinez-Maldonado, Judy Kay, Simon Buckingham Shum, and Kalina Yacef. 2019. Collocated Collaboration Analytics: Principles and Dilemmas for Mining Multimodal Interaction Data. *Human-Computer Interaction* 34, 1 (2019), 1–50. doi:10.1080/07370024.2017.1338956

[33] Roberto Martinez-Maldonado, Judy Kay, and Kalina Yacef. 2013. An Automatic Approach for Mining Patterns of Collaboration around an Interactive Tabletop. In *Artificial Intelligence in Education*, H.Chad Lane, Kalina Yacef, Jack Mostow, and Philip Pavlik (Eds.). Springer, Berlin Heidelberg, Berlin, Heidelberg, 101–110. doi:10.1007/978-3-642-39112-5_11

[34] Elizabeth McBride, Jonathan Vitale, Lauren Applebaum, and Marcia Linn. 2017. Examining the flow of ideas during critique activities in a design project. https://repository.isls.org/handle/1/296

[35] Joseph Edward McGrath. 1984. *Groups: interaction and performance*. Prentice-Hall, Englewood Cliffs, N.J. OCLC: 9083474.

[36] Anne Meier, Hans Spada, and Nikol Rummel. 2007. A rating scheme for assessing the quality of computer-supported collaboration processes. *International Journal of Computer-Supported Collaborative Learning* 2, 1 (2007), 63–86. doi:10.1007/s11412-006-9005-x

[37] René Noël, Diego Miranda, Cristian Cechinel, Fabián Riquelme, Tiago Thompsen Primo, and Roberto Munoz. 2022. Visualizing Collaboration in Teamwork: A Multimodal Learning Analytics Platform for Non-Verbal Communication. *Applied Sciences* 12, 15 (2022), 7499. doi:10.3390/app12157499

[38] Isabella Poggi. 2007. Mind, hands, face and body. A goal and belief view of multimodal communication.

[39] Jason C. Pradarelli, Steven Yule, and Douglas S. Smink. 2020. Evaluating Non-Technical Skills in Surgery. In *Health Services Research*, Justin B. Dimick and Carrie C. Lubitz (Eds.). Springer International Publishing, Cham, 125–135. doi:10.1007/978-3-030-28357-5_12

[40] Sambit Praharaj. 2022. Measuring the Unmeasurable? Towards Automatic Co-located Collaboration Analytics. doi:10.13140/RG.2.2.18216.65287

[41] Sambit Praharaj, Maren Scheffel, Hendrik Drachsler, and Marcus Specht. 2021. Literature Review on Co-Located Collaboration Modeling Using Multimodal Learning Analytics—Can We Go the Whole Nine Yards? *IEEE Transactions on Learning Technologies* 14, 3 (2021), 367–385. doi:10.1109/TLT.2021.3097766

[42] Sambit Praharaj, Maren Scheffel, Marcel Schmitz, Marcus Specht, and Hendrik Drachsler. 2022. Towards Collaborative Convergence: Quantifying Collaboration Quality with Automated Co-Located Collaboration Analytics. In *LAK22: 12th International Learning Analytics and Knowledge Conference* (Online, USA) (LAK22). Association for Computing Machinery, New York, NY, USA, 358–369. doi:10.1145/3506860.3506922

[43] Daniel C. Richardson and Rick Dale. 2005. Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension. *Cognitive Science* 29, 6 (2005), 1045–1060. doi:10.1207/s15516709cog0000_29

[44] F J Rodríguez, K M Price, and K E Boyer. 2017. Expressing and Addressing Uncertainty: A Study of Collaborative Problem-Solving Dialogues In. In *Making a Difference: Prioritizing Equity and Access in CSCL, 12th International Conference on Computer Supported Collaborative Learning*, B K Smith, M Borge, E Mercier, and K Y Lim (Eds.). Vol. 1. Philadelphia, PA: International Society of the Learning Sciences., Philadelphia, PA.

[45] Nikol Rummel, Anne Deiglmayr, Hans Spada, George Kahrimanis, and Nikolaos Avouris. 2011. Analyzing Collaborative Interactions Across Domains and Settings: An Adaptable Rating Scheme. In *Analyzing Interactions in CSCL: Methods, Approaches and Issues, Sadhana Puntambekar, Gijsbert Erkens and Cindy Hmelo-Silver* (Eds.). Springer US, Boston, MA, 367–390. doi:10.1007/978-1-4419-7710-6_17

[46] Bertrand Schneider and Paulo Blikstein. 2015. Unraveling Students' Interaction Around a Tangible Interface using Multimodal Learning Analytics. *Journal of Educational Data Mining* 7, 3 (2015), 89–116. doi:10.5281/zenodo.3554729

[47] Bertrand Schneider and Roy Pea. 2017. Real-Time Mutual Gaze Perception Enhances Collaborative Learning and Collaboration Quality. In *Educational Media and Technology Yearbook*, Michael Orey and Robert Maribe Branch (Eds.). Vol. 40. Springer International Publishing, Cham, 99–125. doi:10.1007/978-3-319-45001-8_7 Series Title: Educational Media and Technology Yearbook.

[48] Bertrand Schneider, Kshitij Sharma, Sebastien Cuendet, Guillaume Zufferey, Pierre Dillenbourg, and Roy Pea. 2018. Leveraging mobile eye-trackers to capture joint visual attention in co-located collaborative learning groups. *International Journal of Computer-Supported Collaborative Learning* 13, 3 (2018), 241–261. doi:10.1007/s11412-018-9281-2

[49] Bertrand Schneider, Gahyun Sung, Edwin Chng, and Stephanie Yang. 2021. How Can High-Frequency Sensors Capture Collaboration? A Review of the Empirical Links between Multimodal Metrics and Collaborative Constructs. *Sensors* 21, 24 (2021), 8185. doi:10.3390/s21248185

[50] Orit Shaer, Megan Strait, Consuelo Valdes, Taili Feng, Michael Lintz, and Heidi Wang. 2011. Enhancing genomic learning through tabletop interaction. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*. ACM, Vancouver BC Canada, 2817–2826. doi:10.1145/1978942.1979361
[51] Barbora Siposova and Malinda Carpenter. 2019. A new look at joint attention and common knowledge. *Cognition* 189 (2019), 260–274. doi:10.1016/j.cognition.2019.03.019

[52] Nikita Soni, Ailish Tierney, Katarina Jurczyk, Schuyler Gleaves, Elisabeth Schreiber, Kathryn A. Stofer, and Lisa Anthony. 2021. Collaboration around Multi-Touch Spherical Displays: A Field Study at a Science Museum. *Proc. ACM Hum.-Comput. Interact.* 5, CSCW2, Article 326, 34 pages. doi:10.1145/3476067

[53] Anthony Tang, Melanie Tory, Barry Po, Petra Neumann, and Sheelagh Carpendale. 2006. Collaborative Coupling over Tabletop Displays. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Montréal, Québec, Canada) (CHI '06). Association for Computing Machinery, New York, NY, USA, 1181–1190. doi:10.1145/1124772.1124950

[54] Tanya Vacharkulksemsuk and Barbara L. Fredrickson. 2012. Strangers in sync: Achieving embodied rapport through shared movements. *Journal of Experimental Social Psychology* 48, 1 (2012), 399–402. doi:10.1016/j.jesp.2011.07.015

[55] Xiangyu Ying, Huanhuan Li, Shan Jiang, Fei Peng, and Zhongxin Lin. 2014. Group laziness: The effect of social loafing on group performance. *Social Behavior and Personality: an international journal* 42, 3 (2014), 465–471. doi:10.2224/sbp.2014.42.3.465

[56] Yiqiu Zhou and Jina Kang. 2022. Characterizing joint attention dynamics during collaborative problem-solving in an immersive astronomy simulation. doi:10.5281/ZENODO.6852988
# APPENDIX

## A Written version of the proposed collaboration profiles

Those profiles have been created from observation across multiple studies on collaboration and from the literature.

### A.1 Everything – Nothing

This profile is characterized by a peer collaboration in which participants show an unequal degree of involvement in the collaborative situation. This dynamic lies somewhere between the "V" and "D" ("Disengaged") profiles [53], and corresponds to the "Driver-passenger" profile [50]. The individual currently disengaged from the task may occasionally make suggestions that are rarely taken into account by the other collaborator. The latter only calls on his partner to intervene when necessary. This form of collaboration is characterized by very little communication, whether in terms of discussions, information sharing or frequency of exchanges, as well as a very unequal distribution of contributions to task completion. Due to the disengagement of one of the collaborators and the lack of consideration of his contributions by the other actor, moments of joint attention and awareness are rare. Finally, the individuals involved in this collaboration are not in close proximity during task execution. Gestural communication regarding the task is limited, and individuals do not have a state of postural synchrony, mainly due to the disengagement of one of the actors.

### A.2 Independents: solitary and sociable

We have developed two collaboration profiles based on Shaer's "Independent" profile, describing a situation where collaborators focus on completing their own activities with a minimum of communication. These profiles have similarities with the SPDA collaboration profile described by [53]. Both profiles are characterized by the absence or very low occurrence of turn-taking, overlap and interruption, as well as limited verbal participation. There is also an unequal distribution of speech, as individuals communicate little or not at all. This dynamic also manifests itself in the low presence of verbal discourse and information exchange, as well as referential gestures used to support information sharing. Both collaborators in those profiles are actively engaged in the task, which is reflected by an equal distribution of contribution to the task and by the presence of physical synchrony. Those profiles are distinguished by how they take the other user and his or her actions into account while carrying out the activity. The first profile, named "Solitary", is characterized by an absence of consideration for the other user, observable through the absence of joint attention between users, characterized by the absence of joint visual attention events, of looking at the collaborator and of looking at each other. Collaborators also lack physical proximity, in contrast to the independent "Sociable" profile, where individuals may be in physical proximity. In the "Sociable" profile, we observe joint attention (Individual/Monitoring [51]) and consideration of the other collaborator characterized by the presence of gazes directed at the other and his/her actions. This suggests a greater occurrence of joint visual attention and mutual gaze events compared to the "Solitary" profile, although their frequency remains relatively low. The difference between those two profiles could be explained by the social proximity or relationship between the two individuals before task realization.

### A.3 Leader – Follower

This profile is inspired by the "Leader-Follower" model proposed by Schneider et al. [47], which has its origins in the "Driver-Passenger" profile by Shaer et al. [50]. Compared with the "Everything-Nothing" profile, where one collaborator performs all the actions while the other is passive, this "Leader-Follower" profile sees the "Follower" not simply as passive, but as a participant in the activity.
In this profile, the collaborator with the "Leader" profile is more strongly engaged in the task than the "Follower". This is characterized by an unequal distribution of speaking time and contribution to the task, with the "Leader" making a greater contribution. Exchanges between individuals are frequent, but there are few turn changes or interruptions from the "Follower". Moreover, exchanges are often initiated by the "Leader" and mainly concern strategies or instructions for carrying out the task, while the "Follower" tends to formulate activity-related responses and suggestions, although these are often little taken into account by the "Leader". Both participants are active in the task and have a physical proximity favorable to sharing information and taking the other into account. However, visual attention is directed by the "Leader", who also receives more gazes from the "Follower" than the other way around. Collaborators show a good level of joint attention (Common attention [51]), with many occurrences of joint visual attention events. However, few mutual gaze events are observed in this collaboration profile. It is worth noting that the "Leader" has greater expertise in the task or in the use of devices than the "Follower".

## A.4 Teacher - Student

This collaboration profile corresponds to a situation where one of the individuals performs most of the physical actions related to the task, supported by another individual who provides advice, direction and suggestions for carrying out the actions. This profile is particularly relevant in a learning context involving an expert and a novice. The profile is closely related to the "Driver-Navigator" [50] and "VE" [53] collaboration profiles, where individuals work together in discussion and only one of the collaborators physically interact to perform the task, while the other observes the actions of the first user and makes suggestions. The difference in involvement in the task between the two collaborators is reflected in an unequal distribution of speaking time and contribution to the task. The collaborator categorized as "Teacher" will make more verbalizations, while the "Student" collaborator will make a greater physical contribution to the task. Verbal coordination is characterized by a low occurrence of turn-taking, interruption and overlap (except backchannels), and average verbal participation, mainly supported by the "Teacher". Discussions focus mainly on the task and include suggestions and observations. The "Teacher" also tends to support his discourse with referential and pointing gestures. The "Student" may show verbal and physical behaviors of acquiescence and questioning. Both collaborators demonstrate a good level of joint attention (Mutual attention [51]), supported by a high frequency of joint visual attention events, gazes directed at the other collaborator, and the presence of mutual gaze. However, we observe that visual attention tends to be directed by the "Teacher" collaborator. Finally, both participants are actively engaged in the task, with physical proximity favoring exchanges and completion of the activity.

## A.5 Turn-takers Accurate and Non-Accurate model

The "Turn-takers" profile is inspired by the profile of the same name proposed by Shaer et al., [50] (see section 2.1), and also corresponds to the "SPSA" collaboration profile described by Tang et al. [53]. This profile is characterized by equal expertise and active involvement of both collaborators, reflected in a tendency towards an equal distribution of speech and contribution to the task. Individuals also show high verbal participation, with a high frequency of turn-taking and success interruption. It is important to note, however, that the use of other communication channels can influence verbal participation, with profiles that may involve a lower frequency of turn-taking, interruption and verbal involvement, while maintaining a good level of collaboration. Physical proximity between individuals can encourage the use of other communication channels, such as referential or pointing gestures, to support verbal communication but also promote awareness of the other. Physical synchrony can also be observed between collaborators, both actively engaged
in the task. The "Turn-takers" profile shows an excellent level of joint attention (Shared attention [51]), characterized by a high frequency of joint visual attention events, gazes towards the other collaborator and the presence of mutual gaze. More specifically, the "Turn-takers Non-Accurate" profile is characterized by attention that is not always focused on the relevant elements of the task or environment, but which collaborators consider essential, potentially leading to misunderstandings. These misunderstandings can lead to the sharing of information irrelevant to the task, potentially compromising collaborative efficiency and generating uncertainty. For the two "Turn-takers" profiles, the nature of discussions is mainly focused on the current task. The main distinction between the two lies in the presence of critical discussions about each collaborator's understanding and actions, which is present in the "Turn-takers Accurate" profile but absent in the "Turn-takers Non-Accurate" profile. This absence of critical discussion is linked to the presence of misconception [1], which can lead to less effective collaboration, characterized in particular by the presence of Uncertainty [44].
# B Scripts for scenario recording

## B.1 Everything-Nothing scenarios

<table><thead><tr><th>Sub-tasks</th><th>Actor 1</th><th>Actor 2</th></tr></thead><tbody><tr><td>Entry into the virtual environment (VE) and beginning of the task</td><td>(1) - 1 don't communicate with 2.<br>(2) - 1 and 2 don't coordinate at the start of the task.<br>(3) - 1 presses the start button without informing or waiting for 2.</td><td>(1) - 2 tries approaches (proximity, speech) to which 1 will not respond.<br>(2) - 1 and 2 don't coordinate at the start of the task.</td></tr><tr><td>Retrieve the puzzle pieces</td><td>(1) - 1 start to retrieve the pieces in the environment.<br>(2) - 1 retrieves most pieces in the environment.</td><td>(1) - 2 is not ready when 1 starts the task. He's late retrieving the pieces.<br>(2) - When 2 sees a piece to retrieve, 1 retrieves it in the meantime.<br>(3) - 2 successfully retrieves piece J near the door.</td></tr><tr><td>Positioning the puzzle pieces</td><td>(1) - places all his pieces.<br>(2) - 1 realizes that he is blocked from positioning the L and the Cube, as 2 has not placed his the J piece.<br>(3) - 1 explicitly asks 2 to place his piece (e.g., can you place the piece there / the J (please)?</td><td>(1) - 2 looks at 1 placing its pieces. 2 does not place his piece.<br>(4) - 2 places his piece with a short answer ("ok").</td></tr><tr><td>Case (A): Transition to Independent Solitary profile (EN-SOL)<br>Coloring the puzzle pieces without communication and without taking the other into account</td><td>(1) - 1 observes that 2 has started coloring the puzzle.<br>(2) - 1 does not communicate verbally with 2 but take 2's actions into account when coloring the pieces.<br>(3) - 1 provides 2 with clear, succinct coloring instructions.</td><td>(1) - 2 starts coloring the puzzle by referring to the reference figure, without any instructions or requests from 1.<br>(2) - 2 does not communicate verbally with 2 and does not take 1's actions into account when coloring the pieces.<br>(4) 2 agrees and performs the instructions.</td></tr><tr><td>Case (B): Transition to Independent Sociable profile (EN-SOC)<br>Coloring the puzzle pieces without communication and taking the other into account</td><td>(1) - 1 colors the puzzle according to the reference figure.<br>(2) - 1 does not communicate verbally with 2 but take 2's actions into account when coloring the pieces.<br>(3) - 1 observes that piece Z on the right should be green, but is currently blue. He selects it to remove the blue, without informing 2.<br>(5) - 1 changes remove and add their color to pieces, taking into account 2's actions.</td><td>(1) - 2 colors the puzzle according to the reference figure.<br>(2) - 2 does not communicate verbally with 1 but take 1's actions into account when coloring the piece.<br>(4) - 2 sees that 1 has just changed the color of the Z piece from blue to white. He clicks on it, the Z turns green.<br>(5) - 2 changes, remove and add their color to pieces, taking into account 1's actions.</td></tr></tbody></table>
<table><caption>Table 5. Scenarios scripts for the Everything-Nothing profile (EN). (A) presents the transition from EN to the Independent Solitary profile (EN-SOL). (B) presents the transition from EN to the Independent Sociable profile (EN-SOC). (C) presents the transition from EN to the Leader-Follower profile (EN-LF).</caption><tr><td>Case (C): Transition to Leader-Follower profile (EN-LF)</td><td>(1) - 1 realizes that he needs 2 to complete the puzzle according to the reference figure.<br/>(2) - 1 starts to look at 2.<br/>(3) - 1 engages in communication with 2. 1 communicates with 2 using very brief expressions such as: "L, click"; "L, take away the green", without argumentation.</td><td>.</td></tr><tr><td>Coloring the puzzle pieces with communication and taking the other into account</td><td>(4.1) - 1 and 2 can look several times at the same objects, as 1 directs 2's attention.<br/>(4.2) - 1 looks at 2 at certain times when 2 does not respond directly to a request from 1.<br/>(4.3) - 1 colors the puzzle to match the reference figure.</td><td>(3) - 2 colors the puzzle with 1's instructions. 2 takes initiative and changes the color of some pieces without 1's instructions.<br/>(4) - 2 doesn't look at 1, concentrating on coloring the pieces. 2 can make coloring suggestions.</td></tr><tr><td>End task</td><td>(1) - 1 or 2 judges that the puzzle is correct, and presses the validation button</td><td></td></tr></table>

## B.2 Independent Solitary scenario

<table><thead><tr><th>Sub-tasks</th><th>Actor 1</th><th>Actor 2</th></tr></thead><tbody><tr><td>Entry into VE and beginning of the task</td><td>(1) - 1 and 2 coordinate by communicating to carry out the task</td><td>(1) - 1 and 2 coordinate by communicating to carry out the task</td></tr><tr><td>Retrieve the puzzle pieces</td><td>(1) - 1 starts to retrieve the pieces close to him (L, I, Z, T) without communicating with 2.<br/>(2) - 1 don't look at 2 or what he is doing.<br/>(3) - 1 retrieves about 3/4 puzzle pieces.<br/>(4) - 1 does not share information about the number and types of pieces he has retrieved.</td><td>(1) - 2 starts retrieving pieces close to him (Cube, Z, J) without communicating with 1.<br/>(2) - 2 don't look at 1 or what he is doing.<br/>(3) - 2 retrieves about 3/4 puzzle pieces.<br/>(4) - 2 does not share information about the number and types of pieces he has retrieved.</td></tr><tr><td>Positioning the puzzle pieces</td><td>(1) - Remember to place lower pieces first.<br/>(2) - 1 starts to place his pieces without communicating with 2.<br/>(3) - 1 do not watch at 2's actions.<br/>(4) - 1 starts to place the T and the I.<br/><br/>(8) - 1 has finished placing his pieces, reasoning and starting to colorize them. He sees that the I, which is supposed to be green, is cyan; he clicks on it, and the piece becomes green. He sees that the blue T should be white, clicks on it and the T becomes white.</td><td>(1) - Remembers to place the bottom pieces first.<br/>(2) - 2 begins to place his pieces without communicating with 1.<br/>(3) - 2 do not watch at 1's actions.<br/><br/>(5) - 2 places the Z and J.<br/>(6) - 2 realizes he can't place a piece. He goes to see if he can place another piece elsewhere on the puzzle. If so, he places it. If he can't fit any more pieces, 1 starts to reason and solve the coloring of the pieces while waiting for 2 to fit his pieces.<br/>(7) - He sees that the I is blue, whereas it's supposed to be green. He clicks on it, and the piece turns cyan.</td></tr></tbody></table>
<table><caption>Table 6. Scenarios scripts for the Independent Solitary (A) and Independent Solitary to Independent Sociable (B) profiles.</caption><tr><td></td><td></td><td>(9) - 2 sees that he can place pieces. He places the J, the Cube, and then the Z.</td></tr><tr><td>Case (A): Independent Solitary profile (SOL)</td><td>(10) - 1 places the Z and the L.</td><td>(1) - 2 continues to color the puzzle according to the reference figure.</td></tr><tr><td>Coloring the puzzle pieces without communication and without taking the other into account</td><td>(1) - 1 continues to color the puzzle according to the reference figure (e.g. "1 sees that the I is cyan, he clicks on it, the I becomes green").<br>(2) - 1 does not communicate verbally with 2 and does not take 2's actions into account when coloring the pieces: 1 observes that piece Z on the right should be green, but is currently blue. He selects it to remove the blue, without informing 2.<br>(3) - "1 sees that the bottom J is green, he clicks on it, the J turns blue" (conflict with 2).<br>(4) - the Z at the bottom should be cyan but is blue. 1 selects it, and the Z becomes cyan.</td><td>(2) - 2 does not communicate verbally with 1, and does not take 1's actions into account when coloring the pieces: 2 does not see that 1 has just changed the color of a piece from cyan to blue. 2 does not respond directly by changing the color of the same piece to cyan (example applicable to both users).<br>(3) "2 sees that the J at the bottom is green, clicks on it and the J turns blue" (because 1 clicked on it at the same time).</td></tr><tr><td>Case (B): Transition to Independent Sociable profile (SOL-SOC)</td><td>(1) - 1 colors the puzzle according to the reference figure.<br>(2) - 1 does not communicate verbally with 2 but take 2's actions into account when coloring the pieces.<br>(3) - 1 observes that piece Z on the right should be green, but is currently blue. He selects it to remove the blue, without informing 2.</td><td>(1) - 2 colors the puzzle according to the reference figure.<br>(2) - 2 does not communicate verbally with 1 but take 1's actions into account when coloring the piece.<br>(4) - 2 see that 1 has just changed the color of the Z piece from blue to white. He clicks on it, the Z turns green.</td></tr><tr><td>Coloring the puzzle pieces without communication and taking the other into account</td><td>(5) 1 change remove and add their color to pieces, taking into account 2's actions.</td><td>(5) 2 change, remove and add their color to pieces, taking into account 1's actions.</td></tr><tr><td>End task</td><td>(1) - 1 or 2 judges that the puzzle is correct, and presses the validation button</td><td>(1) - 1 or 2 judges that the puzzle is correct, and presses the validation button</td></tr></table>

## B.3 Independent Sociable scenario

<table><thead><tr><th>Sub-tasks</th><th>Actor 1</th><th>Actor 2</th></tr></thead><tbody><tr><td>Entry into VE and beginning of the task</td><td>(1) - 1 and 2 coordinate by communicating to carry out the task: 1 suggests that everyone retrieve the pieces close to them.</td><td>(1) - 1 and 2 coordinate by communicating to carry out the task.</td></tr><tr><td>Retrieve the puzzle pieces</td><td>(1) - 1 starts to retrieve the pieces close to him (L, I, Z, T) without communicating with 2.<br>(2) - 1 sometimes looks at 2 (to find out where he is and what pieces he is picking up).<br>(3) - 1 retrieves about 4 puzzle pieces.</td><td>(2) - 2 accepts 1's suggestion.<br>(1) - 2 starts retrieving pieces close to him (Cube, Z, J) without communicating with 1.<br>(2) - 2 sometimes looks at 1 (to find out where he is and what pieces he is picking up).<br>(3) - 2 retrieves about 3 puzzle pieces.</td></tr></tbody></table>
<table><tr><td>Case (A): Independent Sociable profile (SOC)</td><td>(1) - Remember to place lower pieces first, but doesn't verbalize it.<br/>(2) - 1 starts to place his pieces without communicating with 2.<br/>(3) - 1 sometimes looks at 2's actions and the piece that 2 places until all the pieces have been placed.<br/>(4) - 1 starts to place the T and the I. 1 cannot place any more pieces and waits until 2 has placed his J and Z to continue the task.<br/>(6) - 1 places the Z and the L.</td><td>(1) - Remembers to place the bottom pieces first, but doesn't verbalize it.<br/>(2) - 2 begins to place his pieces without communicating with 1.<br/>(3) - 2 sometimes looks at 1's actions and the piece that 1 places until all the pieces have been placed.<br/>(5) - 2 places the Z and J.<br/>(7) - 2 sees that he can place piece. He places the Cube.</td></tr><tr><td>Case (A): Independent Sociable profile (SOC)<br/>Coloring the puzzle pieces without communication or minimal instruction communication and taking the other into account</td><td>(1) - 1 colors the puzzle according to the reference figure.<br/>(2) - 1 does not communicate verbally with 2 but take 2's actions into account when coloring the pieces.<br/>(3) - 1 observes that piece Z on the right should be green, but is currently blue. He selects it to remove the blue, without informing 2.<br/>(5) - 1 change, remove and add their color to pieces, taking into account 2's actions.</td><td>(1) - 2 colors the puzzle according to the reference figure.<br/>(2) - 2 does not communicate verbally with 1 but take 1's actions into account when coloring the piece.<br/>(4) - 2 see that 1 has just changed the color of the Z piece from blue to white. He clicks on it, the Z turns green.<br/>(5) - 2 change, remove and add their color to pieces, taking into account 1's actions.</td></tr><tr><td>Case (B): Transition to the Turn-takers Accurate Mental Model profile (SOC-TTA)</td><td>.</td><td>(1) - 2 tries to place a piece but can't and verbalizes it. He remembers that the lower pieces must be placed first and tells 1: "I can't place my piece", "Wait... it's true that we have to put the lower pieces down first! Hmmm, I don't have the T, do you?".</td></tr><tr><td>Positioning and coloring the puzzle pieces</td><td>(2) - 1 looks at his inventory and tells 2 the pieces he has collected (e.g. "So, I have a T, a L, a, I and a Z").<br/>(4) - 1 validates that he has listened to 2, suggests placing the T, then places it.<br/>(6) - 1 clicks on the T, and it turns white. He answers 2.<br/>(8) - 1 clicks on the Z, places the E on top, clicks on the piece to remove its color and tells 2 that he can click on it to add its color.</td><td>(3) - 2 listens to 1's listing and does the same (e.g. "I've got a J, a Cube and a Z too").<br/>(5) - 2 tells 1 that he can click on the T to change his color to white to match the figure.<br/>(7) - 2 validates and indicates that he's positioning the Z. He specifies that 1 can click on it to turn it cyan, and that he can then also place his Z.<br/>(9) - 2 clicks on the Z at the top.</td></tr></table>
<table><caption>Table 7. Scenarios scripts for the Independent Sociable (A) and Independent Sociable to Turn-takers Accurate Mental Model (B) profiles.</caption><tr><td></td><td>(10) - 1 recaps the pieces placed and indicates they can continue with the next pieces. He places the I and removes his color, verbalizing it.<br/>(12) - 1 says he can't place the L because the J piece is missing. 1 shows with his hand or with the hover the piece that 2 must place.<br/>(14) - 1 says he'll add his color to the piece. Then 1 places the L.<br/>(16) - 1 sees that the white cube should be blue and clicks on it.</td><td>(11) - 2 clicks on the I to add his color.<br/>(13) - 2 answers 1, indicating that he has the piece. 2 places the piece (the J).<br/>(15) - 2 has seen that the L has been placed by 1 and places the Cube. 2 notices that the cube has the wrong color and clicks on it to remove its color.</td></tr><tr><td>End task</td><td>(1) - 1 or 2 judges that the puzzle is correct, and presses the validation button</td><td>(1) - 1 or 2 judges that the puzzle is correct, and presses the validation button</td></tr></table>

## B.4 Teacher-Student scenario

<table><thead><tr><th>Sub-tasks</th><th>Actor 1 (Student)</th><th>Actor 2 (Teacher)</th></tr></thead><tbody><tr><td>Entry into VE and beginning of the task</td><td>(1) - 1 talks to 2 when arriving in the virtual environment.<br/>(2) - 1 observes 2.<br/>(4) - 1 tells 2 that he is ready to start the task.</td><td>(1) - 2 talks to 1 when arriving in the virtual environment.<br/>(2) - 2 shows more familiarity with the virtual environment (e.g. verbalization "<i>the puzzle to complete here, we should have pieces somewhere, the figure to make is here</i>").<br/>(3) - 2 asks 1 if he's ready to start the task.<br/>(5) - 2 presses the start button.</td></tr><tr><td>Retrieve the puzzle pieces</td><td>(1) - 2 looks at the number of pieces on the puzzle.<br/>(2) - 1 follows 2's gaze, which leads him to look at the puzzle.<br/>(3) - 1 doesn't retrieve the pieces directly and waits for 2's instructions<br/>(5) - 1 has trouble finding all the pieces near him</td><td>(1) - 2 instructs 1 to retrieve the puzzle pieces close to him.<br/>(4) - 2 starts to retrieve the pieces. He sees a piece and verbalizes "<i>Ah, I think there's a piece here. Okay, they look like this</i>". He clicks on it, struggles a bit, then succeeds and understands how to retrieve a piece, then verbalize it.</td></tr></tbody></table>
<table><tr><td></td><td>(8) – 1 continues to retrieve the pieces, following 2's instructions.<br/>(11) – 1 retrieves another piece, then looks at his inventory and indicates how many pieces he has.</td><td>(7) – 2 sees that 1 is having trouble finding the pieces, so he tells him what a piece looks like, and where he found one (using gestures).<br/>(9) – 2 continues to retrieve the pieces.<br/>(10) – As soon as 2 has retrieved all the pieces near him, he tells 1 how many pieces he has in his inventory (3) and asks in return how many pieces 1 has. He reminds them that in total, they must retrieve 7 pieces.</td></tr><tr><td>Positioning the puzzle pieces</td><td>(1) – 1 tries to place his pieces but cannot. He insists on a piece he can't place yet.<br/>(2) – 1 has heard 2 and places the bottom pieces if he has them. He places the I.<br/>(4) – 1 looks at his inventory to see if he has the pieces 2 asks him to place (T). He has it. 1 confirms to 2 that he has the piece ("Yes I have it, wait I'll put it in").<br/>(5) – 1 places the piece according to 2's instructions. He indicates that the piece must be white.<br/>(7) – 1 places the L.</td><td>(1) – 2 tries to place several pieces without success, then remembers that the positioning of pieces is only possible if the pieces on a lower level have been placed first. He verbalizes this.<br/>(2) – 2 places the bottom pieces if he has them. He placed the J.<br/>(3) – 2 looks at his inventory to see which pieces he has to place. Some of his pieces (Z and Cube) require 1 to place his pieces. He asks him verbally, looking at the corresponding pieces, and with gestures which piece(s) he should place.<br/>(6) – 2 confirms and indicates that he must click on them to remove his color. Finally, 2 asks 1 to place the L, so that he can place his cube.<br/>(8) – As 1 adds his pieces, 2 adds his (Cube) to continue completing the puzzle.<br/>(9) – 2 sees that all the pieces have been placed. He explains that they must now match the pieces with the right colors according to the reference figure.</td></tr><tr><td>Coloring the puzzle pieces with communication</td><td>(2) – With 2's verbalizations, 1 notices and tells 2 that the I is supposed to be green.<br/>(4) – 1 clicks on the I to remove the blue.</td><td>(1) – 2 compares the current puzzle with the reference figure. He verbalizes.<br/>(3) – 2 confirms and replies that he must click on it to remove its color.<br/>(5) – 2 then interacts with the I to turn it green</td></tr></table>
<table><caption>Table 8. Scenario script for the Teacher-Student profile (TS).</caption><tr><td></td><td>(6) - 1 starts to interact with the pieces that are supposed to be green, but has a little trouble. 1 is slower than 2.<br/>(8) - 1 selects the last green pieces as instructed by 2. 1 responds to 2 with backchannels, repetition and/or nodding gestures.<br/>(10) - 1 follows 2's instructions and manages to obtain the cyan and white pieces.<br/>(11) - 1 and 2 together check that the puzzle is correct.</td><td>(6) - 2 starts interacting with the pieces to make them match the reference figure. While verbalizing.<br/>(7) - 2 finishes interacting with the pieces supposed to be blue, then turns his attention to 1 to help him with his pieces. He helps him select the right pieces by indicating verbally, with his gaze, and by deictic gestures or hovering with the controller (e.g. "take your blue off the I so I can put it in green", "press the Cube so it's blue").<br/>(9) - For the last cyan and white pieces, 2 tells 1 which piece to select to get the right colors: e.g. "click on the J to turn it cyan", "remove your blue from the T to turn it white".<br/>(11) - 2 and 1 check together that the puzzle is correct.</td></tr><tr><td>End task</td><td>.</td><td>(1) - 2 tells 1 that he can press the button to validate the puzzle.</td></tr><tr><td></td><td>(2) - 1 presses the validation button</td><td></td></tr></table>

## B.5 Leader-Follower scenario

<table><thead><tr><th>Sub-tasks</th><th>Actor 1 (Leader)</th><th>Actor 2 (Follower)</th></tr></thead><tbody><tr><td>Entry into VE and beginning of the task</td><td>(1) - 1 and 2 talk to each other. 1 distributes the sub-tasks without giving more information on how to carry them out (e.g. "collect the pieces" - which pieces? where?).</td><td>(1) - 1 and 2 talk to each other. 2 listens to 1.<br/>(2) - 2 accepts the division of tasks proposed by 1.</td></tr><tr><td>Retrieve the puzzle pieces</td><td>(1) - 1 retrieves pieces close to him. He does not take 2 into account while collecting his pieces.<br/>(4) - 1 collects 4 pieces.<br/>(5) - 1 says he thinks he's collected the pieces on his side, that he can't see any more in the environment and that he's going to move closer to the puzzle to place his pieces.<br/>(6) - 1 suggests that 2 move closer to the table to place the pieces.</td><td>(1) - 2 doesn't directly understand what to do.<br/>(2) - He observes 1 collecting his pieces and realizes that he must do the same.<br/>(3) - 2 starts collecting pieces.<br/>(4) - 2 collects 3 pieces.<br/>(6) - 2 accepts 1's suggestion and moves to the table.</td></tr></tbody></table>
<table><caption>Table 9. Scenario script for the Leader-Follower profile (LF).</caption><tr><td>Positioning the puzzle pieces</td><td>(1) - 1 starts positioning the bottom pieces without informing 2, and without observing what 2 is doing.<br/>(4) - 1 cannot place any more pieces. He realizes that he is stuck because 2 has not placed his pieces. He looks at what 2 is doing.<br/>(6) - 1 notices that 2 is blocked at certain points in the puzzle completion. He reminds him that an upper piece can only be placed if the lower pieces have been placed.<br/>(7) - 1 gives 2 very brief instructions on how to place his pieces (e.g. "I can't place my L because the J below is not placed. Click on the J at the bottom to place the piece").<br/>(9) - 1 places his last pieces.</td><td>(2) - 2 looks at 1 and his actions, and sees that he is starting to place the bottom pieces.<br/>(3) - 2 understands that he must place the bottom pieces before the top. He starts laying his pieces. ("ah that's perfect, so I can put this piece here, the Z", 1 doesn't answer).<br/>(5) - 2 has a problem positioning some pieces. He wants to place one of the top pieces.<br/>(8) - 2 places the piece(s) requested by 1. "I wanted to place this piece here, but I'll place the J, okay".<br/>(9) - 2 places his last pieces.</td></tr><tr><td>Coloring the puzzle pieces with communication</td><td>(1) - 1 suggests that 2 start coloring the pieces of his own color.<br/>(3) - 1 begins to color the pieces of his own color.<br/>(4) - 1 does the puzzle, taking into account/controlling 2's various actions. 1 leads attention via verbal, gaze, spatial and gestural communication (deictic, hover in VR).<br/>(5) - 1 communicates with 2 whenever he needs to. He controls the colorization of the pieces (e.g. "the I in white, click on it to make it green", "the Cube in green, click under it to remove your color, so that I can color it blue").<br/>(6) - 1 listens to 2's suggestions and does them if he finds them relevant.</td><td>(2) - 2 accepts 1's suggestion.<br/>(3) - 2 observes 1 coloring the pieces, and takes the initiative to interact with other pieces.<br/>(4) - 2 follows 1's lead.<br/>(5) - 2 responds to 1's suggestions and requests.<br/>(6) Since 2 follows 1's attention, he can make suggestions. 2 makes suggestions to 1 to change certain pieces (e.g. "I think you need to click on this piece to remove the blue", "I think you can click on this piece (the J) to turn it cyan").</td></tr><tr><td>End task</td><td>(1) - 1 presses the validation button to validates the puzzle.</td><td></td></tr></table>
## B.6 Turn-takers Accurate Mental Model scenario

<table><thead><tr><th>Sub-tasks</th><th>Actor 1</th><th>Actor 2</th></tr></thead><tbody><tr><td>Entry into VE and beginning of the task</td><td>(1) – 1 asks 2 if he's ready to start the task.<br>(3) – 2 presses the start button.<br>(4) – 1 reminds them that they must first collect the pieces, then positioning them, then color the puzzle according to the reference figure.<br>(5) – 1 and 2 decide that each of them should first collect the pieces close to them, and then join them in front of the figure to complete.</td><td>(2) – 2 tells 1 that he is ready to start the task.<br>(5) – 1 and 2 decide that each of them should first collect the pieces close to them, and then join them in front of the figure to complete.</td></tr><tr><td>Retrieve the puzzle pieces</td><td>(1) – 1 retrieves the pieces close to him (J, I, Z, T). For each new piece retrieve, 1 tells 2 (e.g. "I've just retrieved a piece"; "I've retrieved the I").<br>(2) – When 1 has retrieved all his pieces, he tells 2.<br>(5) – 1 tells 2 that he has retrieves 4 pieces, and that they therefore have all the puzzle piece. 1 asks 2 to move towards the table where the puzzle is.</td><td>(1) – 2 retrieves the piece close to him (Cube, Z, L). For each new piece retrieve, 2 tells 1 (e.g. "I've just retrieved a piece"; "I've just retrieved the Cube").<br>(3) – 2 replies that he thinks too has recovered all the pieces on his side.<br>(4) – 2 consults the puzzle and counts the number of pieces supposed to be retrieved. He counts 7 pieces. 2 looks at his inventory, sees that he has recovered 3 pieces, and indicates it as 1.</td></tr><tr><td>Positioning and coloring the puzzle pieces</td><td>(1) – 2 tries to place a piece but can't and verbalizes it. He remembers that the lower pieces must be placed first and tells 1: "I can't place my piece", "Wait... it's true that we have to put the lower pieces down first! Hmmm, I don't have the T, do you ?"<br>(2) – 1 looks at his inventory and tells 2 the pieces he has collected (e.g. "So, I have a T, a L, a, I and a Z").<br>(4) – 1 validates that he has listened to 2, suggests placing the T, then places it.<br>(6) – 1 clicks on the T, and it turns white. He answers 2.<br>(8) – 1 clicks on the Z, places the E on top, clicks on the piece to remove its color and tells 2 that he can click on it to add its color.</td><td>(3) – 2 listens to 1's listing and does the same (e.g. "I've got a J, a Cube and a Z too").<br>(5) – 2 tells 1 that he can clicks on the T to change his color to white to match the figure.<br>(7) – 2 validates and indicates that he's positioning the Z. He specifies that 1 can clicks on it to turn it cyan, and that he can then also place his Z.<br>(9) – 2 clicks on the Z at the top.</td></tr></tbody></table>
<table><caption>Table 10. Scenario script for the Turn-takers Accurate Mental Model profile (TTA).</caption><tr><td></td><td>(10) - 1 recaps the pieces placed and indicates they can continue with the next pieces. He places the I and removes his color, verbalizing it.<br/>(12) - 1 says he can't place the L because the J piece is missing. 1 shows with his hand or with the hover the piece that 2 must place.<br/>(14) - 1 says he'll add his color to the piece. Then 1 places the L.<br/>(16) - 1 sees that the white cube should be blue and clicks on it.</td><td>(11) - 2 clicks on the I to add his color.<br/>(13) - 2 answers 1, indicating that he has the piece. 2 places the piece (the J).<br/>(15) - 2 has seen that the L has been placed by 1 and places the Cube. 2 notices that the cube has the wrong color and clicks on it to remove its color.</td></tr><tr><td>End task</td><td>(1) - 1 and 2 check the puzzle, see if it's correct and verbalize it. One of the 2 collaborators presses the button to validate the figure.</td><td>(1) - 1 and 2 check the puzzle, see if it's correct and verbalize it. One of the 2 collaborators presses the button to validate the figure.</td></tr></table>

## B.7 Turn-takers Non-Accurate Mental Model scenario

<table><thead><tr><th>Sub-tasks</th><th>Actor 1</th><th>Actor 2</th></tr></thead><tbody><tr><td>Entry into VE and beginning of the task</td><td>(1) - 1 and 2 confirm to each other that they are ready to start the task, one of the users presses the start button.<br/>(2) - 1 and 2 decide that each of them should first collect the pieces close to them, and then join them in front of the figure to complete.</td><td>(1) - 1 and 2 confirm to each other that they are ready to start the task, one of the users presses the start button.<br/>(2) - 1 and 2 decide that each of them should first collect the pieces close to them, and then join them in front of the figure to complete.</td></tr><tr><td>Retrieve the puzzle pieces</td><td>(1) - 1 retrieves the pieces close to him (J, I, Z, T). For each new piece retrieve, 1 tells 2 (e.g. "I've just retrieved a piece"; "I've retrieved the I").<br/>(2) - When 1 has retrieved all his pieces, he tells 2.<br/>(4) - 1 also continues to look for pieces on his own and indicates it to 2.<br/>(5) - 1 and 2 exchange and question each other about the retrieving of the pieces (e.g. "I don't know if I've got everything. Do you have everything?", "How do we know if we've got all the pieces?").</td><td>(1) - 2 retrieves the piece close to him (Cube, Z, L). For each new piece retrieve, 2 tells 1 (e.g. "I've just retrieved a piece"; "I've just retrieved the Cube").<br/>(3) - 2 doesn't indicate that he has retrieved all his pieces. 2 continues to look for pieces on his side and indicates it to 1.<br/>(5) - 2, like 1, wonders whether the pieces have been retrieved.<br/>(6) - After some time (approx. .20/30 seconds) looking for the pieces in the environment, 2 suggests 1 to move towards the table where the puzzle is.</td></tr></tbody></table>
<table><tr><td></td><td>(7) – 1 accepts 2's suggestion and moves towards the puzzle.<br/>(8) – In front of the puzzle, 1 starts counting the number of pieces supposed to be retrieved, by counting the puzzle pieces. He counts 7. 1 shares the information with 2, then looks at the number of pieces in his inventory. 1 sees that he has retrieved 4 pieces and tells 2.</td><td>.</td></tr><tr><td>Positioning the puzzle pieces</td><td>(1) – 1 starts positioning his pieces. He starts by trying to place the L. Nothing happens, the piece doesn't sit. 1 remembers that the bottom pieces must be placed first, then the top pieces. He doesn't mention this to 2 and continues to place his pieces. He places the I, the T.<br/>(3) – 1 responds by repeating the instruction to place the bottom pieces first, then the top. He comments that "that's why you weren't moving forward, and why you were trying so hard to place the pieces on top".<br/>(6) – 1 listens to 2 and responds (e.g. "I've placed the I and the T, so I've got an L and a Z left too").<br/>(7) – 1 listens to 2 suggestion to place the Z piece, and places it.<br/>(8) – 1 sees that 2 has positioned the J. He positions the L. 1 indicates with a gesture and verbally that 2 can now position the Cube.<br/>(10) – 1 and 2 note that all the pieces have been placed, and that they can now start coloring the pieces.</td><td>(9) – 2 replies to 1 that he has recovered 3 pieces, and that they therefore have all the puzzle pieces.<br/>(10) – 2 suggests started positioning the pieces.<br/>(1) – 2 starts to place his pieces. He starts to interact with the puzzle pieces, and manages to place the J. For the other pieces, nothing works, he can't place any pieces. He tries for at least 5/10 seconds.<br/>(2) – 2 tells 1 about his difficulties. He explains.<br/>(4) – 2 understands the instructions and starts to apply them with 1.<br/>(5) – 2 looks at his inventory and lists the pieces he still has to place (e.g. "I've placed the K, now I've got the Z and the Cube").<br/>(6) – 2 validates, and indicates that he can now place the Z. 2 specifies that 1 can now place his Z.<br/>(9) – 2 take into account 1's suggestions and places the Cube.<br/>(10) – 1 and 2 note that all the pieces have been placed, and that they can now start coloring the pieces.</td></tr></table>
<table><caption>Table 11. Scenario script for the Turn-takers Non-Accurate Mental Model profile (TTNA).</caption><tr><td>Coloring the puzzle pieces</td><td>(1) - 1 looks at the reference figure to see the pieces he's supposed to color.<br/>(2) - 1 notices that the Cube that should be blue is green. 1 clicks on the Cube to turn it blue, but it turns cyan. 1 understands/remembers that colors add or subtract 1 to 1. 1 does not share this information with 2.<br/>(3) - 1 clicks on the Cube, which turns green again. 1 shows the piece with his finger (gesture or hover in VR) and asks 2 to click on the piece to remove the green.<br/><br/>(5) - 1 clicks on the Cube, which turns blue.<br/><br/>(7) - 1 and 2 simultaneously press the Z piece. It turns blue.<br/><br/>(9) - 1 replies that he had already noticed this and had forgot to share it.<br/>(10) - 1 observes that the last piece supposed to be cyan (L) is green. He informs 2 and specifies that he will change it to cyan.<br/>(11) - 1 observes that the blue T should be white. He changes it to white.<br/><br/>(12) - 1 clicks on the piece shown by 2.<br/>(13) - 1 observes that the blue I is supposed to be green. He clicks on the piece to remove its color.<br/><br/>(15) - 1 and 2 compare the reference figure with the puzzle to check whether it is correct or not.</td><td>(1) - 1 looks at the reference figure to see the pieces he's supposed to color.<br/><br/>(4) - 2 confirms, clicks on the cube, which turns white.<br/><br/>(6) - 2 sees that the two cyan pieces on the reference figure are now green. He asks 2 to click on the Z at the same time as him, so that the piece becomes cyan.<br/>(7) - 1 and 2 simultaneously press the Z piece. It turns blue.<br/>(8) - 2 doesn't understand why the piece has turned blue. He decides to click on the piece. It becomes cyan. 2 understands that it is not necessary to click simultaneously to add colors to the pieces. He tells 1.<br/><br/>(11) - 2 observes that the Z which is supposed to be green is blue. He tells 1, indicating verbally and gesturally the piece concerned.<br/><br/>(14) - 2 sees that 1 has unselected I, and adds his color accordingly.<br/>(15) - 1 and 2 compare the reference figure with the puzzle to check whether it is correct or not.</td></tr><tr><td>End task</td><td>(1) - 1 and 2 check the puzzle, see if it's correct and verbalize it. One of the 2 collaborators presses the button to validate the figure.</td><td>(1) - 1 and 2 check the puzzle, see if it's correct and verbalize it. One of the 2 collaborators presses the button to validate the figure.</td></tr></table>
# C Graphics of profile confidence detection, ground truth and exploration of profile detection within all collaborative scenario

## C.1 Everything - Nothing

<figure><img src="image_8.png" /><figcaption>Fig. 5. Profile detection and confidence value per profile and phases for the Everything-Nothing to Independent Solitary scenario (EN-SOL)</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 6. Profile detection and confidence value per profile and phases for the Everything-Nothing to Independent Sociable scenario (EN-SOC)</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>Fig. 7. Profile detection and confidence value per profile and phases for the Everything-Nothing to Leader-Follower scenario (EN-LF)</figcaption></figure>

In the EN-LF, EN-SOL and EN-SOC scenarios, it is expected to have a higher max and mean confidence value for the profile EN compared to the other in the retrieving phase. In the following phases of the 3 scenarios, we expected to have a lower confidence value for detection of the EN profile, and a higher max and mean confidence value for the profile corresponding to the scenario, e.g., LF for EN-LF, etc., in the phase and the following, compared to the other profiles, including EN.

Results indicate that the EN profile has higher max confidence values in the retrieving phase of EN-LF, EN-SOL, and EN-SOC scenarios, and higher mean confidence values in EN-SOL and EN-SOC scenarios. The latter is lower than those of SOL and SOC profiles in the EN-LF scenario. In the positioning phase of the 3 scenarios, the EN mean confidence value is near 0, even if the profile was sometimes detected. Lastly, the EN profile isn't detected in the coloring phase of the 3 scenarios. In other scenarios, the EN profile isn't detected in the LF, TS, TTA, SOC and SOC-TTA with a mean confidence value of 0 and fewer detections in the retrieving phase of SOC and SOC-TTA scenarios. The profile has a max detection value of 1 in the SOL and SOL-SOC scenarios but with a lower mean confidence value than the expected profile.
## C.2 Independents: Solitary and Sociable

<figure><img src="image_11.png" /><figcaption>Fig. 8. Profile detection and confidence value per profile and phases for the Independent Solitary scenario (SOL)</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 9. Profile detection and confidence value per profile and phases for the Independent Solitary To Independent Sociable scenario (SOL-SOC)</figcaption></figure>

In the Independent solitary scenario, SOL profile detection values is expected to exceed 0.5 throughout the activity. The mean confidence value for this profile is also supposed to be higher than those of other detected profile. No other profiles are expected to be detected during the entire activity.
In the Independent Solitary To Sociable (SOL-SOC) scenario, the SOL profile is expected to be detected with the best mean confidence value in the Retrieving phase of the activity. The SOL profile is expected to be detected less in the Positioning phase and no detected in the last phase of the task (coloring phase). Conversely, we expect to see an increase in the detection percentage and mean confidence value of the SOC profile during the positioning phase, with a higher mean confidence value compared to other profile in the coloring phase. The EN, LF, TS, TTNA and TTA profiles should not be detected throughout the activity.

**Independent Solitary.** In the SOL profile scenario, detection is consistent throughout the task, with the highest mean confidence value compared to other profiles. However, the EN profile is also detected in the retrieving phase, and the SOC profile in the positioning and coloring phases, sharing the max confidence value with SOL. In the SOL-SOC scenario, contrary to expectations, SOL has the highest mean confidence value across all task phases, while SOC remains undetected. In the 9 other scenarios, SOL is frequently detected in the retrieving phase, especially in EN-LF, SOC-TTA, SOC, and LF scenarios, with rare or no detection in TS, TTA, and TTNA scenarios. Notably, SOL shows high mean confidence values and detection percentages in specific scenarios (SOL-SOC, EN-SOC, SOC-TTA) where SOC is expected to be detected rarely or never.

<figure><img src="image_13.png" /><figcaption>Fig. 10. Profile detection and confidence value per profile and phases for the Independent Sociable scenario (SOC)</figcaption></figure>

In the Sociable scenario, the SOC profile is expected to be detected throughout the task with the higher mean confidence detection value compared to other detected profile. No other profiles are expected to be detected during the entire activity.

In the Independent Sociable to Turn-Takers Accurate model (SOC-TTA) scenario, the SOC profile is expected to be best detected with a higher mean confidence value during the part retrieval phase, followed by TTA profile detection during the part positioning and coloring phase, and a decrease in SOC profile detection. The TTA profile is expected to have the best average detection value in the coloring phase, compared to all other detected profiles. A low detection percentage is expected for EN, SOL, LF, TS and TTNA profiles.

**Independent Sociable.** The SOC profile detection is concerned with various scenarios: SOC, EN-SOC, SOL-SOC, and SOC-TTA. In the SOC scenario, the profile is detected in the retrieving and
<figure><img src="image_14.png" /><figcaption>Fig. 11. Profile detection and confidence value per profile and phases for the Independent Sociable To Turn-Takers Accurate model scenario (SOC-TTA)</figcaption></figure>

positioning phases, with a lower mean confidence value than the SOL profile. In the coloring phase, its mean confidence value is lower than other profiles and below 0.5 for detection. In the SOL-SOC scenario, contrary to expectations, the profile is not detected during the positioning and coloring phases, displaying a mean confidence value near 0. In the EN-SOC scenario, the profile is weakly detected in the retrieving phase, highly detected in the positioning phase, and not detected in the coloring phase. In other scenarios, the SOC profile is detected with the highest mean confidence value in the coloring phase of EN-LF and EN-SOL scenarios, and with similar max detection values as SOL in the SOL scenario but lower mean confidence values in the coloring phase. It is less detected in the phases of the LF scenario with lower max and mean confidence values compared to SOL.
C.3 Leader - Follower

<figure><img src="image_15.png" /><figcaption>Fig. 12. Profile detection and confidence value per profile and phases for Leader-Follower Scenario (LF)</figcaption></figure>

In the Leader - Follower scenario, the LF profile is expected to be detected throughout the activity, unlike the EN, SOL, SOC, TS, TTNA and TTA profiles, which are expected to be not detected or with a low mean confidence value. The LF profile is also expected to have the higher mean confidence detection value during all phases of the task.

LF profile detection is concerned with the EN-LF and LF scenarios. In the ENLF scenario, detection occurs only once during the positioning phase alongside profiles with higher maximum and mean confidence values. In the LF scenario, the profile is detected a few times, and its mean confidence value is 0 over all phases of the scenario. Additionally, the profile is detected in the TS scenario with the highest average confidence value during the coloring phase and occasionally during phases of the TTNA scenario.
C.4 Teacher - Student

<figure><img src="image_16.png" /><figcaption>Fig. 13. Profile detection and confidence value per profile and phases for Teacher - Student scenario (TS)</figcaption></figure>

In the Teacher - Student scenario, the TS profile is expected to be detected throughout the activity, unlike the EN, SOL, SOC, LF, TTNA and TTA profiles, which are expected to not be detected or with a low mean confidence value. The TS profile is also expected to have the higher mean confidence detection value during phases of the task.

The TS profile detection concerns the TS scenario. Results on this scenario indicate that the TS profile has a lower confidence value in the retrieving and coloring phases respectively compared to the TTA and LF profiles. The profile has a higher max confidence value detection in the retrieving and positioning phases, and a lower or equal in the coloring phase compared to the SOC and LF profiles. In other scenarios, the TS profile has a higher mean confidence value in the positioning and coloring phases of the TTA scenario, and in the positioning phase of the LF scenario. It is also detected in most other scenarios but with a mean confidence value of no more than 0.25.
C.5 Turn-Takers Accurate mental model

<figure><img src="image_17.png" /><figcaption>Fig. 14. Profile detection and confidence value per profile and phases for Turn-Takers Accurate model scenario</figcaption></figure>

In the Turn-Takers Accurate mental model (TTA) scenario, it is expected that the profile will be highly detected throughout the activity, unlike the other profiles which should either not be detected or have a low mean confidence value.

The TTA profile is concerned with the TTA and SOC-TTA scenarios. The profile has a higher mean confidence value than all the other profiles in the retrieving phase, similar to the TS profile but higher than the others in the positioning and coloring phase of the TTA scenario. It also exhibits a strong mean confidence value in the positioning and coloring phase of the SOC-TTA scenario, despite a lower maximum detected confidence compared to TS and TTNA profiles in the same phase. However, in scenarios where the SOC profile is expected to be detected, the TTA profile is detected with a mean confidence value of no more than 0.25. Additionally, it is detected in the last phase of the LF scenario with the highest mean confidence value. Lastly, TTA and TTNA profiles seem to be associated because of their joint detection in various scenarios.

Received January 2024; revised July 2024; accepted October 2024