# ReflectPal: Exploring Self-Reflection on Collaborative Activities Using Voice Assistants

Eleftherios Papachristos, Dorte P. Meldgaard, Iben R. Thomsen, Mikael B. Skov

<figure><img src="image_1.png" /></figure>

## To cite this version:

Eleftherios Papachristos, Dorte P. Meldgaard, Iben R. Thomsen, Mikael B. Skov. ReflectPal: Exploring Self-Reflection on Collaborative Activities Using Voice Assistants. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.187-208, 10.1007/978-3-030-85610-6_12. hal-04215485

HAL Id: hal-04215485

https://inria.hal.science/hal-04215485v1

Submitted on 22 Sep 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# ReflectPal: Exploring Self-Reflection on Collaborative Activities using Voice Assistants

Eleftherios Papachristos¹[0000-0003-2666-9864], Dorte P. Meldgaard¹, Iben R. Thomsen¹, and Mikael B. Skov¹[0000-0003-4488-5683]

Aalborg University, Selma Lagerlöfsvej 300, Aalborg, Denmark
papachristos@cs.aau.dk, dme90@hotmail.com, ibenrosa@gmail.com,
dubois@cs.aau.dk

**Abstract.** Voice Assistants (VAs) present promising opportunities for the development of applications for the work domain. While previous research is primarily focused on aiding groups and individuals to be more productive, studies exploring the use of VAs to train and develop collaboration skills are rather limited. In this paper, we examine whether VAs can be used to help individuals improve their collaboration skills through self-reflection. We developed ReflectPal, a Google Assistant application designed to facilitate reflection sessions regarding collaboration challenges. First, we identified a list of frequently occurring challenges in a specific collaboration work environment. Then we designed ReflectPal to address a subset of those challenges and tested it in a two-week in-situ deployment with 19 participants. We found that participants benefited from the structure that the practice provided, leading to deeper and more meaningful reflections than before. However, the study also highlighted the need to design applications that take motivational aspects into account to encourage frequent engagement in self-reflection for skill development. Reflecting on insights from our study, we discuss future design directions of VAs for facilitating self-reflection in a collaborative work context.

**Keywords:** Voice Assistants · Conversational Agents · Digital Assistants · Workplace · Collaboration · Reflection.

## 1 Introduction

There is a long tradition of developing technology to support productivity in the work environment, while tools focusing on wellbeing in collaborative activities have received only a little attention [2, 25]. Although previous studies have explored reflection as a tool to increase wellbeing among employees [20, 25], few have explored designs to facilitate reflection through self-assessment [3, 44]. This study explores how reflection can be used to overcome challenges related to group collaboration without applying measures that focus on visible feedback and concrete end results. This motivation stems from a concern expressed by Baumer et al. [3] who states that most studies treat reflection as a means to
an end by measuring outcomes related to reflection and not the act of reflection itself. Contrary to the goal-oriented paradigm of measuring outcomes of reflection, this study focuses on the act of reflection itself. If through reflection, people can reach the desired outcome (e.g., performance, behavioral change, or goal-reaching), it should only be viewed as a personal achievement and not a means to an end. To conceptualize reflection in this paper, we utilize Schön's [48] definition of reflection-on-action, a retrospective act, where the practitioner reflects on an event that already has occurred while reconsidering the situation and what needs changing in the future [48].

We explore the potential of using Voice Assistants (VA) to enable reflection in a group collaboration context. According to recent research, VAs can support both behavior change and wellbeing through the act of reflection [26]. The technology supports natural ways of communicating and has proven beneficial in individuals' willingness to disclose and self-assess [33, 51]. In some situations, people can feel less vulnerable, and experience less fear of judgment when interacting with machines compared to other humans [33]. Moreover, VA technology can be advantageous in several contexts because it can support hands-free use and provide accessibility for people with visual impairments [33, 43]. The capabilities of VAs have evolved considerably over recent years, especially regarding speech-recognition [38] and in the adoption of human-like features [13, 42]. Simultaneously, problems concerning the technology have also been identified, such as usability issues and a gap between user expectation and technical capabilities [34, 42]. Although research has highlighted problems concerning VAs, the rapid improvement of the technology over the past years gives reasons to believe that shortcomings will be addressed and gradually resolved [38].

We developed ReflectPal, an application for Google Assistant, to examine whether VAs can guide and support individual reflection concerning behavioral patterns in collaborative activities. To gain a comprehensive understanding of challenges that occur in collaborative activities, we first examined related literature. We subsequently conducted thematic analysis on 126 student reports concerning university students' self-assessments of their collaborative group activities. We identified common challenges related to behavioral dynamics through the thematic analysis, such as opposing values and personality traits, underlying hierarchical structures, and unequal participation. We designed ReflectPal to address a subset of these challenges and tested it in a two-week in-situ deployment with 19 university students.

This work's contributions are: 1) A collection of challenges concerning group collaboration and activities. 2) The design of ReflectPal, an application in Google Assistant that guides and supports individual reflection on collaborative activities. 3) Findings from an in-situ deployment showing that VAs are capable of aiding reflection and that structured reflection patterns have the potential to lead to deeper and more conscious reflection. However, results also indicated that participants sometimes needed to rely on textual input, highlighting that the technology still needs improvement. Lastly, findings show that VAs supporting
reflection is a promising research area that could benefit from more examination
by the Human-Computer Interaction (HCI) community.

## 2 Related Work

A variety of technologies to support collaborative activities have been explored over the years in HCI research. Also, as quoted by Licklider in 1960: "[*...] there is a continuing interest in the idea of talking with computing machines." [31], which recently has started to become a viable alternative to traditional visual interfaces. Recent developments in VAs give reason to believe that voice interfaces could be the future of many key services [34].

### 2.1 Collaborative Activities and Technology

In collaborative activities, the concept of feedback has been used to increase individual and team performance. DiMicco et al. [12] developed a tool to indicate appropriate group behavior standards to support individual reflection. The study examined how a shared display could impact individuals' behavior in groups during collaborative tasks. Findings revealed that behavior in the extremes was affected by the presence of the display. To detect social interactions, promote behavioral change, and provide feedback to enhance group collaboration, Kim et al. [24] developed the Meeting Mediator. The aim was to bridge the gap amongst distributed groups by detecting, communicating, and visualizing social signals on group members' mobile phones [24]. Tausczik et al. [50] examined a real-time feedback system to monitor communication patterns among students in co-located groups. The results showed that the system improved group performance, but only in the groups that were dysfunctional [50]. Leshed et al. [29] aimed to stimulate reflection on language use and collaborative behavior. For this objective, they developed a chat-based system to present visual feedback to group members. Findings revealed that feedback in collaborative work settings affected social interactions and caused people to alter their communication patterns. The examples above highlight how technology can be used to support feedback in collaborative activities. However, findings also suggest that feedback alone may not be sufficient to make a difference [44]. Hence, a growing number of studies investigate whether the development of systems that encourage open-ended reflection could be more effective in creating meaningful change [41, 39, 21].

### 2.2 Supporting Reflection with Technology

Positive behavior changes can be triggered when individuals assess their own experiences to reach new understandings and appreciations [4, 26]. Reflection is a core mechanism to translate experience into learning and support personal growth has been explored in various contexts and domains (e.g., education, health, and work [26, 36]). Several technologies have been developed to inform
design on reflection for everyday practices and personal informatics, for example, through self-tracking [46], lifelogging [6], and digital diaries [32]. Isaacs et al. [22] developed Echo, an Android application designed for users to record and systematically reflect on their daily activities. Findings showed that Echo produced improvement in participants' wellbeing after only using the application for a month. Kocielnick et al. [28] developed Reflection Companion, a mobile conversational system that supported reflection on personal sensor data, specifically physical activity data collected with fitness trackers. Their findings suggested that mini-dialogues successfully triggered reflection and led to increased motivation, empowerment, and behavioral change. When designing for reflection, Baumer et al. [3] state: "[...] sometimes the goal of reflection is not only to increase self-knowledge but to take action based on this increased awareness. Systems of reflection vary as to the extent that they support taking such action.". Therefore, it is important to design systems that acknowledge reflection as an ongoing process, thus supporting an increase in self-knowledge and granting room for taking action based on the increased awareness.

## 2.3 Voice Assistants

Cho et al. [8] argue that speech-based agents need to support more core values rather than just entertainment. Today VAs are gaining popularity, and it is clear that the technology offers new and innovative opportunities for engagement in collaborative activities [38]. However, it is not entirely clear how this technology can be used to deal with challenges in collaborative work environments. In the work domain, there has been an emphasis on using agents or chatbots for personal organization, administrative tasks, or management of to-do lists [26]. Cranshaw et al. [10] presented a digital assistant that provided fast and efficient scheduling through structured workflows. Liao et al. [30] conducted a field study with a personal agent designed to help employees detect work-related information. Their findings revealed individual differences in preferences towards humanized social interactions, concluding that variability in user needs have to be considered during agent design. McGregor et al. [35] demonstrated how agents could be used to monitor spoken dialogue in group settings, and proactively detect useful actions, and carry those out without any specific commands. These examples show that speech-based technology has made its entry into the work domain by assisting workers in organizational and structural assignments. Few studies have examined how VAs can be used to support workers in processes related to personal wellbeing. Kimani et al. [25] developed Amber, a conversational agent in support of goal achievement, aiming to explore the potential of using conversational agents to improve workplace productivity and wellbeing. Findings indicated that participants enjoyed Amber's work-related suggestions. Workers became more mindful about their practices to the point where they would make changes to increase productivity and become healthier. Kocielnick et al. [26] developed an agent with chat-based communication through a personal device to examine how voice-based and chat-based interaction affected work-
ers' reflection and supported self-learning. Their findings indicated that voice
interaction might enable users to step back and reflect on their work.

3 Identification of Group Collaboration Challenges

The first step in our analysis was to identify a list of frequently occurring chal-
lenges in group collaborations. Our goal was to detect specific problems in which
self-reflection could be an appropriate method to improve teamwork skills. For
this purpose, we launched an in-depth investigation of frequently occurring col-
laboration problems in a specific context. The context we chose to investigate
was student group projects at Aalborg University. This University utilizes the
Problem-Based Learning (PBL) model, which involves a high degree of group
work and collaboration among students [14] in their semester projects. Students
work together in small groups of four to seven members attempting to solve prob-
lems relevant to their interests and learning goals. At the end of the semester,
students have to write a report describing their process, summarize their find-
ings, and provide reflections about the quality of their group collaboration. We
used these reports as input data in a thematic analysis to identify challenges stu-
dents encounter when participating in these collaborative activities. Choosing to
conduct thematic analysis on university student reports allowed us to focus on a
group of individuals with similar and well-defined requirements for group work,
who still encounter a variety of collaboration challenges. These challenges were
subsequently used to inform our application's design, as it helped us understand
what problems it should be able to address to be relevant in this specific context.

**3.1 Thematic Analysis: Procedure and Findings**

We conducted a thematic analysis on 126 reports authored by students at Aal-
borg University from a variety of educations. These reports are publicly accessi-
ble and included a description, analysis, and evaluation of the collaboration and
work processes within the group, as well as collaboration with external part-
ners and supervisors. These reports were selected after an in-depth search at
the Universities digital project repository. The selection criteria for reports to
be included in our final analysis was that they should be authored from 2015
to 2019 and contain subject headings such as 'group work', 'work processes',
and 'group collaboration'. The reports that fulfilled the selection criteria were
analyzed through a subsequent thematic analysis inspired by [5]. The analysis
was conducted in four steps. First, two of the authors read all the reports for the
first time to identify initial patterns in the data and to note explicit mentions
of collaboration challenges. Second, both authors started generating individual
inspection lists after reading and re-reading the reports while taking notes about
potential codes. The inspection lists captured all identified instances related to
collaborative problems mentioned in the reports that contributed to an initial
low-level code. Third, the two inspection lists were merged after an agreement
between the two authors, followed by a discussion about potential themes. We
identified 17 themes that were checked in relation to the coded extracts in the inspections lists and were organized into thematic maps based on their interrelationship. These themes represent commonly occurring collaboration challenges based on the reflective self-assessment of the students. Lastly, we generated definitions and names for each theme, and we calculated the frequency of mentions in our dataset of each challenge.

<table><caption>**Table 1.** Findings from thematic analysis: identified challenges.</caption><thead><tr><td>Challenges</td><td>Number of mentions</td></tr></thead><tbody><tr><td>Insufficient time management</td><td>53</td></tr><tr><td>Insufficient discussion management</td><td>44</td></tr><tr><td>Insufficient decision-making</td><td>44</td></tr><tr><td>Non-work related discussions</td><td>44</td></tr><tr><td>Unequal participation in discussions</td><td>43</td></tr><tr><td>Insufficient use of ICT-tools</td><td>30</td></tr><tr><td>Insufficient project management</td><td>29</td></tr><tr><td>Missing work ethics</td><td>28</td></tr><tr><td>Insufficient agenda management</td><td>26</td></tr><tr><td>Insufficient meetings management</td><td>26</td></tr><tr><td>Failing to match expectations in advance</td><td>25</td></tr><tr><td>Insufficient conflict management</td><td>20</td></tr><tr><td>Insufficient break management</td><td>19</td></tr><tr><td>Failing to meet as scheduled</td><td>17</td></tr><tr><td>Insufficient management of interruption</td><td>16</td></tr><tr><td>Domination and hierarchy issues</td><td>15</td></tr><tr><td>Written agreements not complied with</td><td>10</td></tr></tbody></table>

The thematic analysis revealed collaboration challenges consistent with findings from related literature (e.g., [12, 24, 42, 49]) and some unique to the specific context of project work at this University. This analysis provided a well-grounded collection of collaboration challenges relevant to the particular context we were investigating, which can be seen in Table 1. From this list, we selected challenges addressing group dynamics issues rather than practical or productivity problems such as time management or usage of appropriate collaboration software. We chose four challenges after a discussion among the authors about which of those we considered could be addressed better by personal rather than group reflection sessions. The four selected challenges are (1) Unequal participation in discussions, (2) Domination and hierarchy issues, (3) Insufficient decision-making, and (4) Non-work-related discussions. We used those four challenges to guide our design process but also the evaluation of the Voice Assistant application ReflectPal that we will present in the next section.
<figure><img src="image_4.png" /><figcaption>Fig. 1. Visualization of ReflectPal's system architecture.</figcaption></figure>

# 4 ReflectPal: an Application for Reflection

Following the thematic analysis, we started the process of designing ReflectPal to address the challenges we identified. Our objective was to develop an application that would support reflection on an individual's behavior in collaborative activities. Inspired by previous research that has demonstrated the benefits of using conversational agents in the context of facilitating reflection in the work environment [26] we developed ReflectPal as a third-party application (Action) for the Google Assistant. Our goals for developing ReflectPal were threefold: (1) provide users with questions to facilitate reflection through a guided dialogue, (2) address the four collaborative challenges identified through the thematic analysis, (3) allow interaction with multiple modalities (e.g. voice, text).

## 4.1 Design and Development

ReflectPal was developed using Google Actions and Dialogflow. This allowed us to use a variety of well-established tools such as Google's Speech to Text (STT), Natural Language Processing (NLP), and Text to Speech (TTS) technologies [1]. The core logic and the conversation structure were implemented in Dialogflow, and we used a webhook service to integrate with Firebase to store application data. We also used the Speech Synthesis Markup Language (SSML) to improve speech intelligibility by modifying TTS responses. The core architecture of ReflectPal can be seen in Figure 1.

We intended to deploy ReflectPal on devices with an embedded Google Assistant, for example, mobile devices and smart speakers. User inputs are registered through voice or text with a microphone or keyboard, and outputs are generated as audio on a speaker or text on a screen. Although we used Firebase to store basic user information, we decided that ReflectPal should not record
and store users' reflections. We considered that we would remove some of our
participants' hesitation to disclose their thoughts and feelings by not recording
the reflection sessions. The design process of ReflectPal was inspired by Google
Developers' design guidelines [11] and the design guide by Kim et al. [23] for
consistent personality manifestation. We chose a medium-to-high pitched male
voice to separate ReflectPal from the default voice in Google Assistant regarding
voice characteristics. The selected voice matched the personality traits that we
considered desirable to be mirrored by ReflectPal (e.g., calm, trustworthy, neu-
tral). In addition, we used SSML to make speech responses slower-paced, aiming
to give ReflectPal a calm personality.

To launch ReflectPal, the users had to say aloud or type "talk to ReflectPal" to the Google Assistant. This action invoked a welcome intent that introduced users to ReflectPal by explaining the purpose of the application and its benefits. Following the introduction, users were asked to select which topic they would like to focus on out of 5 predefined choices. Four out of the five topics referred to the challenges derived from the thematic analysis, while the fifth was a generic one that could be used to reflect on any challenge the user may have encountered. We included a generic topic to make ReflectPal more flexible, enabling reflection on a broader range of challenges. Although our choice of topics for reflection did not encompass challenges related to group management and productivity issues, the self-chosen topic still allowed users to reflect on these matters if they choose to do so. The five reflection topics users could select were the following: (1) Better your contribution to group discussions; (2) Decrease your domination in group work; (3) Get better at decision-making; (4) Get better at not engaging in small-talk during group activities; (5) Pick your own topic. Once users had picked a focus for the session, they were guided through six stages. The stages were explicitly designed to promote reflection. After having gone through the six stages and before ending the session, users were presented with a closing remark, a quote about reflection, and prompted to occasionally take a step back to reflect on their actions and feelings, aiming to encourage reflection in their everyday lives. Time is an essential condition for enabling reflection [28]. This was also taken into account when designing the flow of how users should interact with ReflectPal. Users were provided with the time they needed to reflect and move on whenever they felt ready. This was considered significant for the flow of the conversation to create a pleasurable and unrushed interaction.

## 4.2 Conversation Design to Enable Reflection

ReflectPal's conversation design was based on a theoretical framework, aiming to encourage reflection and guide users through a developmental process [18, 28]. The conversation structure was guided by Gibbs's reflective cycle, which proposes a design approach for reflection-on-action based on six reflective stages [18]. The six stages related to (1) Description, promoting to recall what happened in the experience; (2) Feelings, identification of thoughts and feelings; (3) Evaluation, evaluating whether the experience could be deemed good or bad; (4)
<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Sample dialogue from ReflectPal related to the reflection point ‘decrease your domination on group work’, visualizing the connection between reflection levels and conversation structure.</figcaption></figure>

Analysis, relating to what sense could be made from the experience; (5) Conclusion, relating to which actions could have been better to take; and (6) Action Plan, prompting to identify what to do in a similar experience in the future [18]. Figure 2 shows an example of how the reflection levels are related to the dialogue in ReflectPal. Additionally, we directed our focus on ways to design and phrase questions to support reflection. Here, we applied Moon's levels of learning [37], which entailed a comprehensive collection of tools for how to support reflection.

# 5 Field Deployment

To examine whether ReflectPal could support individuals in reflecting on their behavior in collaborative activities, we tested the application in a two-week in-situ deployment [47]. The purpose of the field deployment was to evaluate ReflectPal and assess the feasibility of using Voice Assistant to facilitate self-reflection on a regular basis.

## 5.1 Participants

We recruited 19 participants (M=9, F=10) through Aalborg University's social media pages and email list. Our participants were students from various departments and educations, and all of them had practical and theoretical experience working in student groups that varied in group size, scope, and purpose. They were between 19 to 30 years old, with an average age of 25.2 years. All of them reported high competence in command of the English language. Demographic data also showed that most of our participants (N=13) considered themselves experienced with technology in general but novices regarding VAs (N=15). After agreeing to participate, we provided information about the study, instructions about interacting with ReflectPal, and step-by-step installation guides for iOS
and Android. Even though we initially recruited 26 participants, seven of them
dropped out after the installation phase. Out of those seven, two reported is-
sues installing the application because they owned older iOS devices, while the
remaining five dropped out for personal reasons. Before interacting with the ap-
plication, participants were asked to sign a consent declaration and fill out a
questionnaire to collect demographic data and previous experience with tech-
nology.

## 5.2 Procedure

The study duration was two weeks, and we did not give participants a specific
schedule about when to use ReflectPal. We only instructed them to use it a
minimum of two times per week whenever they felt like it. The reason for not
forcing a more frequent use upon the participants was based on prior studies
showing that reflection is a time-consuming process that does not necessarily
happen all at once [2]. We, therefore, wanted to allow our participants to use
ReflectPal at their own pace. Immediately after the two weeks deployment pe-
riod, we invited our participants for a debriefing study. The study included a
post-deployment questionnaire and an interview, which aimed to gather feedback
from the participants’ experiences when interacting with the application. The
questionnaire contained nine questions seeking to assess user perceptions about
ReflectPal regarding interaction, usability, and overall usefulness (see table 2).
Participants provided ratings using a 7-point Likert scale with the anchors “I
totally disagree” to “I totally agree”. The questionnaire was formulated with
inspiration from agent rating scales presented by Kimani et al. [25], and Olafs-
son et al. [40], as these studies had a similar goal of encapsulating user feedback
based on VA interactions. Similarly to these studies, we aimed to explore dimen-
sions of usability, user enjoyment, trustworthiness, and whether the application
supported the primary purpose of aiding participants in reflecting. After hand-
ing in the questionnaire, we conducted a semi-structured interview focused on
participants’ experiences using ReflectPal, how it affected their reflection pat-
terns, their perception of strengths and weaknesses of the application, and their
intention of future use.

# 6 Results

In this section, we will present the findings of our post-deployment data analysis.
First, we analyzed data from the questionnaire data by calculating mean scores
and standard deviation for each of the items. Overall results from the question-
naire indicate general satisfaction with ReflectPal. Participant both liked (M =
4.32, SD = 1.64) and trusted (M = 4.74, SD = 1.6) the application. They also
gave relatively high scores to all three usability related questions (ranging from
4.43 to 5.58). However, although participants perceived the interaction with Re-
flectPal to be comfortable (M = 5.0, SD = 1.53), they gave a considerably lower
score on how natural it felt (M = 3.53, SD = 1.84), showing that there is still
<table><caption>**Table 2.** Post-deployment questionnaire items and responses.</caption><thead><tr><th>Items</th><th>Mean (SD)</th></tr></thead><tbody><tr><td><em>Usefulness</em></td><td></td></tr><tr><td>ReflectPal helped me reflect</td><td>4.58 (1.68)</td></tr><tr><td>I would like to continue reflecting with ReflectPal</td><td>1.68 (1.53)</td></tr><tr><td><em>Perceptions about ReflectPal</em></td><td></td></tr><tr><td>I like ReflectPal</td><td>4.32 (1.64)</td></tr><tr><td>I trust ReflectPal</td><td>4.74 (1.6)</td></tr><tr><td><em>Perceptions about interaction</em></td><td></td></tr><tr><td>I felt the interaction with ReflectPal was comfortable</td><td>5.0 (1.53)</td></tr><tr><td>I felt the interaction with ReflectPal was natural</td><td>3.53 (1.84)</td></tr><tr><td><em>Usability</em></td><td></td></tr><tr><td>ReflectPal offered help about how to interact with it</td><td>4.43 (1.58)</td></tr><tr><td>It was easy to interact with ReflectPal</td><td>5.27 (1.64)</td></tr><tr><td>It was clear for me how to interact with ReflectPal</td><td>5.58 (1.61)</td></tr></tbody></table>

room for improvement in this regard. Arguably the most interesting finding from the post-deployment questionnaire concerns the usefulness of ReflectPal. Based on the results, participants perceived the application useful in helping them reflect (M = 5.0, SD = 1.53), but at the same time, they would probably not continue using it (M = 1.68, SD = 1.53). This could indicate that users did not like to reflect on collaboration challenges in general or that the motivational design of ReflectPal has to be improved. We used this finding during the interview data analysis to uncover reasons and explanations that could provide answers to the discrepancy between usefulness and intention to use.

We then analyzed our qualitative data set from the interview. The interviews had an average duration of 20 minutes and were audio-recorded and fully transcribed manually by two authors to familiarize themselves with the data. Afterwards, the transcribed text was analyzed through thematic analysis inspired by [9]. We followed an inductive approach [7] by interpreting the raw data and abstracting common themes since no predefined themes were identified before the analysis. The analysis was conducted in three steps. First, two of the authors read through the transcribed text and gained an initial understanding of the data, followed by an initial coding phase. Secondly, the two authors compared and combined their coding schemes and categorized quotes into themes over several iterations. Lastly, the authors finalized the categorization of codes into themes using NVivo. Four themes emerged from our analysis: Overall Experience with ReflectPal, VA's in the context of reflection, Effect on reflection practice, and Suggestions for future usage.

## 6.1 Qualitative Findings

Our qualitative analysis showed that ReflectPal supported individual reflection on collaborative activities. The participants generally reflected more during this period, not only about collaborative activities but also about other life issues.
We will present results from the thematic analysis based on the four identified themes in the following sections.

**Overall Experience with ReflectPal** For all participants, ReflectPal managed to invoke reflection in one way or another. Most participants reported that the application supported them in reflecting on their behavior in collaborative activities. Also, many of them stated that they experienced a substantial increase in how frequently they would reflect and that the sessions led to additional reflection on other parts of their lives. Since we prioritized removing usage barriers, we did not record reflection sessions or collect usage data. However, from the interview, it becomes clear that the usage of ReflectPal varied considerably regarding what participants chose to reflect on. Some reflected on past experiences from several years ago, while others would reflect on more recent collaborative group activities. Also, the choice time for reflection sessions varied considerably among participants. Some reported that they used ReflectPal directly after a collaborative activity; others used it as an intermediate activity between different work tasks, while others used it at the end of the day as a way of debriefing. Finally, one participant reported using ReflectPal before meetings to identify how she could improve her actions compared to the previous meetings. A general finding was that the convenience of natural interaction motivated the participants to reflect more. Even though most provided positive remarks regarding ReflectPal, a few participants were reluctant to continue using it. Those comments and suggestions participants provided to make the application relevant in the long term are provided in the following sections.

**VA's in the Context of Reflection.** Several participants made comments on the context of use with the VA technology. We were particularly interested in those comments considering that most participants did not have extensive previous exposure to VA's. Some mentioned that the interaction with a VA was considerably different from any other type of technology they had encountered before and attitudes towards VA technology varied noticeably. Four participants were extremely positive, three were negative, while the rest were either neutral or had mixed feeling. On the positive side, comments revolved around how easy, fun, and joyful it was to interact with the application using voice. Some mentioned that the interaction felt more natural because there was no screen and that compared to talking to humans, the VA was perceived as more neutral. Because it felt more similar to talking to a person instead of reading text, it made them feel more guided and inclined to reflect more. Also, the act of saying thinks out loud increased the motivation of some to reflect at a deeper level:

> P19: "[...] I talked a bit more with it, and maybe that made me reflect more because I was forced to say it out loud instead of just thinking it."

Negative comments concerned a general unease with voice technologies and awkwardness talking to a machine about personal feelings. Even though the VA did not instruct participants to say anything aloud, some participants felt that
talking to a machine ruined the intimacy of reflecting. Some also thought that the interaction was too rigid and not flexible to their particular needs. Interestingly, this was mentioned more often by participants with none or only a little experience with VAs. Participants with some experience were more moderate about their expectations. One participant said:

**P1:** "[...] I feel like it is a joyful experience to talk to it in general. It is a system, and it is not that smart, and some people have made a lot of thoughts about what it will say next and how it will react to what I am saying"

The majority of participants with mixed feelings mentioned technical issues or misunderstanding of commands. This was particularly visible in the beginning but improved over time. Many noted that talking to a machine felt different and weird initially, but they become accustomed and more comfortable after using it a few times. Finally, the ability to interact both via text and speech was well-received, and many considered it one of the core strengths of ReflectPal. It is noteworthy that quite a few reported difficulties finding a private space to feel comfortable talking out loud to a VA due to their living conditions.

**Effect on Reflection Practice.** The most frequent comment in the interviews was that it provided good structure and guidance during reflection. This was mentioned even by participants who were negative about ReflectPal. Moving gradually from superficial to deep reflection, was perceived by many as a novel way of reflecting, which generally led to a deeper and more conscious reflection than usual.

**P1:** "[...]ReflectPal helped me reflect in an organized way, so with ReflectPal, I went deeper and deeper into the specific situation [...] usually when I reflect, I am not that organized about it."

Apart from the depth of reflection, many mentioned that ReflectPal helped them widen their breadth regarding reflect topics. This was evident by comments sowing surprise that one could get something out of reflecting about their own behavior in collaborative activities. A participant, for example, said:

**P15:** "[...]normally I think I reflect on other people's behavior in the group, and it is a little bit more difficult to look inwards. But I think that ReflectPal really initiates that reflection on your own behavior."

Out of the 19 participants, only three of them reported that ReflectPal did not support their reflection on collaborative activities in a profound way. One participant already had a reflection routine and did not feel that ReflectPal was creating additional value, and the other two felt that using technology to reflect was too artificial. However, even these participants still reported that they experienced a change in their reflection patterns. Exposure to ReflectPal's structured approach made them evaluate their own way of reflecting from a meta-perspective and adopting some aspects of it into their daily lives. This was mentioned by multiple times during the interviews, indicating a common pattern among participants.
**Suggestions for Future Usage.** Our interview data also illustrated why some participants would probably not use ReflectPal in the future, even though they rated it as highly useful in the questionnaire. The comments about this focused mainly on relevance and personalization. Some participants perceived the application as a learning tool that would lose its usefulness after understanding the lesson it tries to teach them. They mentioned that they had identified the pattern behind the reflection structure ReflectPal suggests, and they already tried to apply it to other aspects of their lives. Therefore, they did not see any value in continuous usage after some time of interaction. But other participants saw value in continued usage of ReflectPal as a stable guide in their reflection practice, but they stressed the need for more personalization. Some mentioned that the reflection topics were not relevant for them and their current situation and that they would use ReflectPal in the future if the topics would cover their needs:

**P16:** "[...]I would like more topics because, for example, this semester, we were having some problems in my group, but none of those problems were there."

Some participants also mentioned that they would appreciate some guidance or suggestions about appropriate action plans after reflecting on a group activity. Even though they understood that the purpose of ReflectPal was not to give them prescriptive advice about collaboration issues, they felt that challenges were left unresolved at the end of the sessions:

**P17:** "[...]I know I need to reflect myself and to think about it by myself, but maybe more tips I would have liked. [...] Guide me, but also give me solutions."

Also, some participants mentioned that they did not enjoy the fact that ReflectPal mainly focused on past negative experiences. As the reflection topics originated from common collaborative challenges that we identified in students' reports, the sessions with ReflectPal were primarily focused on problematic situations. Those participants raised the valid point that reflection about collaborative work's positive aspects would also be beneficial.

**P19:** "[...]my experience with the questions was that they kind of had a negative outlook from the start. [...] of course, I understand that reflecting is to make yourself better, but I felt that 'oh, I have done something wrong, so now I need to reflect on it,' and it is only when I have done something wrong that I can reflect on it."

Finally, three participants pointed out that they would like to see whether ReflectPal could be used in a group meeting setting. They understood the value of self-reflecting and focusing on their own behavior, but they could also see clear value in having ReflectPal guide some of their group meetings in the future.
# 7 Discussion

The field deployment results have provided insights into university students’ reflection on their behavior in collaborative group activities using a VA. The majority of our participants could see the benefits of using a VA to facilitate reflection sessions mainly because it offered support and structure. At the same time, the conversational interaction felt easy and convenient. In the next sections, we will reflect on our findings and discuss implications for design and future research directions for VA-facilitated self-reflection.

## 7.1 Reflection in a Collaborative Context

Previous research has shown that VA applications can support goal-oriented interactions and interventions to help users achieve goals through self-assessment [26, 49]. Our aim was not to examine if the reflection could lead to the desired outcome but to promote the act of reflection itself as a practice that could help develop collaboration skills. Results showed that our VA application was relatively successful in facilitating individual reflection sessions about collaborative activities. However, we also found that it would probably be challenging to regularly motivate users to initiate reflection sessions about collaborative activities.

A variety of approaches could be explored to improve the motivational design of ReflectPal, and here, we only mention some based on participant feedback. To begin with, the application could explicitly communicate the benefits of frequent reflection and provide resources to allow users to understand the possible gains of this activity. We also found that some participants would appreciate support and suggestions about how to act in future scenarios after recognizing aspects of their behavior that could be improved. Another request was to make usage data more visible as a motivational feature to use the application more often. This would be consistent with results outlined by Zhou et al. [52], which describes visible feedback as one of the most useful and effective features for users to achieve goals and maintain motivation by comparing past performances to current ones. Finally, some participants mentioned that reminders or notifications would help them remember to use the application. It could be argued that some of the improvements the participants suggested are counter to the overarching goals of our research, which was to facilitate reflection that is self-initiated and triggered by intrinsic motivations for self-improvement. However, it also becomes clear from our study results that some elements of the motivational design have to be implemented in future versions to increase the possibility of people using ReflectPal for an extended period.

Although this study has focused on individual reflection, there is no doubt that group-based reflection also plays a significant role in collaborative environments. Individual Reflection was described as beneficial for most participants, yet some also expressed an interest in applying the structured reflection plenary with their group members. We see a potential in leveraging both individual and group-based reflection in collaborative contexts to accommodate a larger area of collaborative challenges. It would also be interesting to examine how ReflectPal
could fit into existing collaborative practices and group meetings and how this
could affect motivation to use. In continuation, it could be relevant to explore the
dimension of reflecting with VAs in collaborative activities in groups compared
to individual reflection.

## 7.2 VA-facilitated Reflection

For most participants, using a VA was a new experience, and therefore, we
were able to collect feedback showing both advantages and shortcomings of the
technology. In particular, the conversational design of ReflectPal was praised for
successfully guiding the participants and showing them a new structured way of
reflecting. By presenting only small chunks of information at a time, we allowed
them to reflect on specific aspects of a challenge, thus enabling a more in-depth
reflection before moving on to the next stage. Some participants suggested that
the whole experience could be further enhanced if the six stages were split up
into several simple questions that would resemble a more natural conversation.

Findings also showed that participant opinions varied regarding using voice
as the primary mode of interaction. Some felt that it provided added value to
the experience, while some were more reluctant. It is also important to men-
tion that issues with speech recognition were discussed repeatedly during the
interview. While developing ReflectPal, we made a proactive choice of accom-
modating both voice-based and text-based inputs because we knew that our
participants would not be native English speakers. Some participants reported
that the speech recognition improved over time, while others switched to text-
based input. Giving participants the option to switch between voice-based and
text-based inputs proved beneficial in lessening frustration with the VA and
providing alternative ways to interact. Our results reconfirmed what has been
shown multiple times in previous work, that in terms of speech recognition, VAs
are still in their infancy [34, 42]. A seamless, purely voice-based interaction ap-
pears to be an ambitious but not impossible objective considering how much
commercially available devices have been improved in recent years. More than
half of our participants indicated that their living conditions made it difficult for
them to reflect using voice due to privacy reasons. This illustrates the need for
considerations of the target group demographic characteristics while designing
for VAs.

Despite technical shortcomings, voice interaction may still be more advan-
tageous over other modalities in facilitating self-reflection in the work context.
Even though we focused on individual self-reflection in future studies, we intend
to explore the use of VAs to facilitate group reflections. Shared ownership and
neutrality of the VA could prove valuable characteristics to facilitate group re-
flection sessions. Besides, we intend to develop ReflectPal further to be able to
facilitate both individual as well as group reflection sessions. We want to explore
a process in which individual members anonymously suggest reflection topics
that will be used first in individual sessions, followed by a subsequent group
session with all members.
## 7.3 Generality Versus Specificity

ReflectPal addressed only four collaboration challenges that were identified in the thematic analysis. This resulted in some participants not being able to relate to the topics presented in the application. We created an option for participants to choose a generic self-chosen topic to cover challenges that ReflectPal did not already support. Participants that mainly used the generic topic felt that ReflectPal was a learning tool that would be useful only for a limited number of times. However, it was also observed that the generic guide triggered reflection on other aspects of life that were not constrained to the work environment in contrast to the specific guides that only granted room to reflect on the one selected challenge. Designing ReflectPal to be very specific could provide a better experience but at the same time make the application too rigid. In contrast, designing it too generic could make it more useful to some and irrelevant to other participants. Our goal was to balance generality and specificity in our application's design. This has been difficult due to the participants' subjective opinions and preferences in how they preferred to facilitate their reflections sessions. What becomes apparent from the study is that to support continuous usage of the application, relevance has to be considered. This can be achieved either by including more reflection topics or by providing room for customization and flexibility. It is reasonable to believe that some domains would benefit from a generic design that supports a broader reflection, while other domains would benefit from a more specific design. We suggest that when designing for reflection, both the domain-specific and user perspectives are taken into consideration.

## 7.4 Reflecting on How to Reflect

Throughout this study, it became increasingly evident that the type of Reflection that ReflectPal supported was not only limited to the intended usage regarding reflection-on-action in collaborative experiences. We found that some participants started reflecting on their ways of reflecting and implementing the step-wise and conscious reflection into their daily lives. This use of ReflectPal we refer to as meta-reflection. This type of reflection-on-reflection was overall mentioned by several participants throughout the interviews and was an unanticipated outcome of this study. It became apparent that the meta-reflection occurred, as the sequence of the questions that followed Gibb's six stages of reflection helped some of our participants to gain more skills in personal development. For these participants, ReflectPal served as a learning tool to increase their awareness of how they reflect on their actions. The fact that VAs appear adequate to be introduced as a learning tool to meta-reflection creates several additional questions about how to design VAs to support this purpose. As for now, the field of meta-reflection is an underexplored area within the HCI field, as the body of research covering design guides for this is very limited. Research has focused on related areas such as meta-cognition [16, 17], self-reflection [19, 27], or reflective practice [15, 45]. Although overlaps are present between these research areas and meta-reflection, none of the mentioned areas entirely covers
the specific act of reflecting on one's own reflection and accustoming to deeper
and more structured reflection patterns. We see potential in further exploration
of meta-reflection, as this study has indicated that this could lead to a more
meaningful reflection by monitoring, assessing, and adjusting the act of reflec-
tion itself.

8 Limitations and Future Work

This study has some limitations that have to be noted. First, the relatively
small sample size (N=19) and the limited amount of time participants interacted
with ReflectPal (two weeks) may limit our results as well as the conclusion
that can be drawn from our findings. However, this initial exploratory study
aimed to investigate the feasibility of using VAs to facilitate reflection about
collaborative activities. In future work, a more extended deployment period and a
larger and more varied sample have to be considered to investigate the long-term
effects of VA facilitated reflection. Also, our work solely focused on challenges
with collaborative activities that occur in a specific university context. Even
though we see many parallels between this university's problem-based learning
context and many industry practices, the relevance has to be explored in future
work. Moreover, efforts should be made to uncover if similar applications as the
one presented in this study are relevant and applicable to other domains (e.g.,
everyday life) in which collaboration is important.

9 Conclusions

This study aimed to explore whether VAs could be used to facilitate reflections in the context of group collaboration. Our results show the promise of voice interaction technologies in this area, demonstrating that our VA application was more than sufficient to enable structured reflection among our participants. We developed ReflectPal to provide initial insights about how to design VAs for individual reflection on collaborative activities. We identified a collection of frequently occurring challenges in a specific collaborative context that were used to drive the design process of ReflectPal. Results from two-week in-situ deployment showed that participants generally appreciated being introduced to a structured way of reflecting on their own behavior in collaborative activities and that structured reflection in some cases led to meta-reflection. Moreover, we found that purely voice-based interaction still faces some limitations in its current state, highlighting the importance of providing text-based input as a fallback for VA applications. An important goal in sharing our findings has been to highlight trade-offs, encourage conversation about VA capabilities, and call attention to certain aspects of VAs for reflection that could benefit from further research.
References

1. Ali, M., Hassan, A.M.: Developing applications for voice enabled IoT devices to improve classroom activities. In: 2018 21st International Conference of Computer and Information Technology (ICCIT). pp. 1–4. IEEE (2018). https://doi.org/10.1109/ICCITECHN.2018.8631906

2. Baumer, E.P.: Reflective informatics: Conceptual dimensions for designing technologies of reflection. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. p. 585–594. CHI ’15, Association for Computing Machinery, New York, NY, USA (2015). https://doi.org/10.1145/2702123.2702234

3. Baumer, E.P., Khovanskaya, V., Matthews, M., Reynolds, L., Schwanda Sosik, V., Gay, G.: Reviewing reflection: On the use of reflection in interactive system design. In: Proceedings of the 2014 Conference on Designing Interactive Systems. p. 93–102. DIS ’14, Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2598510.2598598

4. Boud, D., Keogh, R., Walker, D.: Reflection:Turning experience into learning (1985)

5. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qualitative research in psychology 3(2), 77–101 (2006)

6. Byrne, D., Kelliher, A., Jones, G.J.: Life editing: Third-party perspectives on lifelog content. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. p. 1501–1510. CHI ’11, Association for Computing Machinery, New York, NY, USA (2011). https://doi.org/10.1145/1978942.1979162

7. Chandra, Y., Shang, L.: Qualitative research using R: A systematic approach. Springer (2019). https://doi.org/http://dx.doi.org/10.1007/978-981-13-3170-1

8. Cho, M., Lee, S.s., Lee, K.P.: Once a kind friend is now a thing: Understanding how conversational agents at home are forgotten. In: Proceedings of the 2019 on Designing Interactive Systems Conference. p. 1557–1569. DIS ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3322276.3322332

9. Clarke, V., Braun, V., Hayfield, N.: Thematic analysis. Qualitative psychology: A practical guide to research methods pp. 222–248 (2015)

10. Cranshaw, J., Elwany, E., Newman, T., Kocielnik, R., Yu, B., Soni, S., Teevan, J., Monroy-Hernández, A.: Calendar.help: Designing a workflow-based scheduling agent with humans in the loop. p. 2382–2393. CHI ’17, Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3025453.3025780

11. Developers, G.: Conversation design (July 2019), https://developers.google.com/assistant/actions/design

12. DiMicco, J.M., Pandolfo, A., Bender, W.: Influencing group participation with a shared display. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work. p. 614–623. CSCW ’04, Association for Computing Machinery, New York, NY, USA (2004). https://doi.org/10.1145/1031607.1031713

13. Doyle, P.R., Edwards, J., Dumbleton, O., Clark, L., Cowan, B.R.: Mapping perceptions of humanness in intelligent personal assistant interaction. In: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services. MobileHCI ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3338286.3340116

14. Fink, F.K.: Integration of engineering practice into curriculum-25 years of experience with problem based learning. In: FIE’99 Frontiers in Education. 29th Annual
Frontiers in Education Conference. Designing the Future of Science and Engineering Education. Conference Proceedings (IEEE Cat. No. 99CH37011. vol. 1, pp. 11A2-7. IEEE (1999). https://doi.org/http://dx.doi.org/10.1109/fie.1999.839084

15. Finlay, L.: Reflecting on 'reflective practice' (2008)

16. Flavell, J.H.: Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American psychologist **34**(10), 906 (1979).
https://doi.org/http://dx.doi.org/10.1037/0003-066X.34.10.906

17. Gama, C.: Helping students to help themselves: a pilot experiment on the ways of increasing metacognitive awareness in problem solving. Proceedings of CINTEC International Conference pp. 1-8 (2001)

18. Gibbs, G.: Learning by doing: A guide to teaching and learning methods. Oxford Centre for Staff and Learning Development (1988)

19. Grant, A.M., Franklin, J., Langford, P.: The self-reflection and insight scale: A new measure of private self-consciousness. Social Behavior and Personality: an international journal **30**(8), 821-835 (2002).
https://doi.org/http://dx.doi.org/10.2224/sbp.2002.30.8.821

20. Grover, T., Rowan, K., Suh, J., McDuff, D., Czerwinski, M.: Design and evaluation of intelligent agent prototypes for assistance with focus and productivity at work. In: Proceedings of the 25th International Conference on Intelligent User Interfaces. p. 390-400. IUI '20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3377325.3377507

21. Halttu, K., Oinas-Kukkonen, H.: Persuading to reflect: Role of reflection and insight in persuasive systems design for physical health. Human-Computer Interaction **32**(5-6), 381-412 (2017). https://doi.org/10.1080/07370024.2017.1283227

22. Isaacs, E., Konrad, A., Walendowski, A., Lennig, T., Hollis, V., Whittaker, S.: Echoes from the past: How technology mediated reflection improves well-being. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. p. 1071-1080. CHI '13, Association for Computing Machinery, New York, NY, USA (2013). https://doi.org/10.1145/2470654.2466137

23. Kim, H., Koh, D.Y., Lee, G., Park, J.M., Lim, Y.k.: Developing a design guide for consistent manifestation of conversational agent personalities. In: IASDR Conference. Manchester Metropolitan University, Manchester, UK. pp. 1-17 (2019)

24. Kim, T., Chang, A., Holland, L., Pentland, A.S.: Meeting mediator: Enhancing group collaboration using sociometric feedback. In: Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work. p. 457-466. CSCW '08, Association for Computing Machinery, New York, NY, USA (2008).
https://doi.org/10.1145/1460563.1460636

25. Kimani, E., Rowan, K., McDuff, D., Czerwinski, M., Mark, G.: A conversational agent in support of productivity and wellbeing at work. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). pp. 1-7. IEEE (2019). https://doi.org/http://dx.doi.org/10.1109/ACII.2019.8925488

26. Kocielnik, R., Avrahami, D., Marlow, J., Lu, D., Hsieh, G.: Designing for workplace reflection: A chat and voice-based conversational agent. In: Proceedings of the 2018 Designing Interactive Systems Conference. p. 881-894. DIS '18, Association for Computing Machinery, New York, NY, USA (2018).
https://doi.org/10.1145/3196709.3196784

27. Kocielnik, R., Maggi, F.M., Sidorova, N.: Enabling self-reflection with lifelogexplorer: Generating simple views from complex data. In: 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops. pp. 184-191. IEEE (2013).
https://doi.org/http://dx.doi.org/10.4108/icst.pervasivehealth.2013.251934
28. Kocielnik, R., Xiao, L., Avrahami, D., Hsieh, G.: Reflection companion: A conversational system for engaging users in reflection on physical activity. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2(2) (Jul 2018). https://doi.org/10.1145/3214273

29. Leshed, G., Perez, D., Hancock, J.T., Cosley, D., Birnholtz, J., Lee, S., McLeod, P.L., Gay, G.: Visualizing real-time language-based feedback on team-work behavior in computer-mediated groups. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. p. 537–546. CHI ’09, Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1518701.1518784

30. Liao, Q.V., Davis, M., Geyer, W., Muller, M., Shami, N.S.: What can you do? studying social-agent orientation and agent proactive interactions with an agent for employees. In: Proceedings of the 2016 ACM Conference on Designing Interactive Systems. p. 264–275. DIS ’16, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2901790.2901842

31. Licklider, J.C.: Man-computer symbiosis. IRE transactions on human factors in electronics (1), 4–11 (1960). https://doi.org/http://dx.doi.org/10.1109/THFE2.1960.4503259

32. Lindström, M., Ståhl, A., Höök, K., Sundström, P., Laaksolathi, J., Combetto, M., Taylor, A., Bresin, R.: Affective diary: Designing for bodily expressiveness and self-reflection. In: CHI ’06 Extended Abstracts on Human Factors in Computing Systems. p. 1037–1042. CHI EA ’06, Association for Computing Machinery, New York, NY, USA (2006). https://doi.org/10.1145/1125451.1125649

33. Lucas, G.M., Gratch, J., King, A., Morency, L.P.: It’s only a computer: Virtual humans increase willingness to disclose. Computers in Human Behavior **37**, 94 – 100 (2014). https://doi.org/https://doi.org/10.1016/j.chb.2014.04.043

34. Luger, E., Sellen, A.: "like having a really bad pa": The gulf between user expectation and experience of conversational agents. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. p. 5286–5297. CHI ’16, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2858036.2858288

35. McGregor, M., Tang, J.C.: More to meetings: Challenges in using speech-based technology to support meetings. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. p. 2208–2220. CSCW ’17, Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/2998181.2998335

36. Mols, I., van den Hoven, E., Eggen, B.: Informing design for reflection: An overview of current everyday practices. NordiCHI ’16, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2971485.2971494

37. Moon, J.A.: Reflection in learning and professional development: Theory and practice. Routledge, London UK (2013). https://doi.org/http://dx.doi.org/10.4324/9780203822296

38. Moore, R.K.: Is Spoken Language All-or-Nothing? Implications for Future Speech-Based Human-Machine Interaction, pp. 281–291. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-2585-3-22

39. Oinas-Kukkonen, H.: A foundation for the study of behavior change support systems. Personal and Ubiquitous Computing **17**(6), 1223–1235 (2013). https://doi.org/10.1007/s00779-012-0591-5

40. Olafsson, S., O'Leary, T., Bickmore, T.: Coerced change-talk with conversational agents promotes confidence in behavior change. In: Proceedings of the 13th EAI
International Conference on Pervasive Computing Technologies for Healthcare. p. 31–40. PervasiveHealth’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3329189.3329202

41. Ploderer, B., Reitberger, W., Harri Oinas-Kukkonen, H., van Gemert-Pijnen, J.: Social interaction and reflection for behaviour change. Personal and Ubiquitous Computing **18**(7), 1667–1676 (2014)

42. Porcheron, M., Fischer, J.E., McGregor, M., Brown, B., Luger, E., Candello, H., O'Hara, K.: Talking with conversational agents in collaborative action. In: Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. p. 431–436. CSCW ’17 Companion, Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3022198.3022666

43. Porcheron, M., Fischer, J.E., Reeves, S., Sharples, S.: Voice interfaces in everyday life. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. p. 1–12. CHI ’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3174214

44. Prins, F.J., Sluijsmans, D.M., Kirschner, P.A.: Feedback for general practitioners in training: Quality, styles, and preferences. Advances in Health Sciences Education **11**(3), 289 (2006). https://doi.org/http://dx.doi.org/10.1007/s10459-005-3250-z

45. Prior, J., Ferguson, S., Leaney, J.: Reflection is hard: Teaching and learning reflective practice in a software studio. In: Proceedings of the Australasian Computer Science Week Multiconference. ACSW ’16, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2843043.2843346

46. Rivera-Pelayo, V., Fessl, A., Müller, L., Pammer, V.: Introducing mood self-tracking at work: Empirical insights from call centers **24**(1) (Feb 2017). https://doi.org/10.1145/3014058

47. Rogers, Y., Connelly, K., Tedesco, L., Hazlewood, W., Kurtz, A., Hall, R.E., Hursey, J., Toscos, T.: Why it’s worth the hassle: The value of in-situ studies when designing ubicomp. In: Krumm, J., Abowd, G.D., Seneviratne, A., Strang, T. (eds.) UbiComp 2007: Ubiquitous Computing. pp. 336–353. Springer Berlin Heidelberg, Berlin, Heidelberg (2007)

48. Schon, D.A.: The reflective practitioner: How professionals think in action, vol. 5126. Basic books (1984)

49. Shamekhi, A., Liao, Q.V., Wang, D., Bellamy, R.K.E., Erickson, T.: Face value? exploring the effects of embodiment for a group facilitation agent. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. p. 1–13. CHI ’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3173574.3173965

50. Tausczik, Y.R., Pennebaker, J.W.: Improving teamwork using real-time language feedback. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. p. 459–468. CHI ’13, Association for Computing Machinery, New York, NY, USA (2013). https://doi.org/10.1145/2470654.2470720

51. Zalake, M., Lok, B.: Non-responsive virtual humans for self-report assessments. In: Proceedings of the 18th International Conference on Intelligent Virtual Agents. p. 347–348. IVA ’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3267851.3267893

52. Zhou, M., Mintz, Y., Fukuoka, Y., Goldberg, K., Flowers, E., Kaminsky, P., Castillejo, A., Aswani, A.: Personalizing mobile fitness apps using reinforcement learning. In: CEUR workshop proceedings. vol. 2068, pp. 2–21. NIH Public Access (2018)