introduced to the Pyramid flow achieved elasticity and dynamism. Dynamically growing pyramids and replication of pyramids on-demand accommodated late-comers. Dropping out from current activity did not harm the pyramid progression. The mechanisms also led to ensure that orchestration (or real time script management) aspects were meaningful, i.e. in alignment with the pedagogical structural elements of the Pyramid pattern. For example, even if every participant did not submit an initial option (e.g. if they arrive late or leave) for the given task in above cases, the mechanisms ensured every group had at least two participants and one option to discuss in the first rating level and the flow was not interrupted. The same strategy has been used at other rating stages too, assuring that the pyramid can level-up. Future work should study to what extent these mechanisms can be extrapolated to other script families (e.g. those based on the Jigsaw flow pattern).

The results also suggest that human and intelligent agent interventions could further improve the utility of the proposed mechanisms. Usage of the in-built discussion board of the PyramidApp was not satisfactory across experiments. Email notifications used in the distance mode, notifying activity status did not catch sufficient participant attention. Hence, in future discussion prompts, cues or agent technologies like learning companions can be introduced to study how they could aid in promoting higher engagement in discussions. Future improvements like allowing small groups to modify or submit new options after collaborating in the rating levels and enhancing the notification system could be beneficial. Moreover, an extended version of the monitoring dashboard that would enable human intervention by the educator could improve the options for orchestration support, e.g. by facilitating the participation of the educator in discussions when especially needed (e.g. alerted by the system) or to modify timers to regulate the activity progression if required.

## References

Davis, W.A. (2002). A comparison of pyramids versus brainstorming in a problem-based learning environment. In *Focusing on the student: Proceedings of 11th Annual Teaching Learning Forum*. Perth: ECU.

Dillenbourg, P., & Tchounikine, P. (2007). Flexibility in macro-scripts for CSCL. *Journal of Computer Assisted Learning*, 23(1), 1-13.

Dillenbourg, P., Sanna, J., & Fischer, F. (2009). The evolution of research on computer-supported collaborative learning. *Technology-Enhanced Learning*, 3-19. Netherlands: Springer.

Dillenbourg, P., & Jermann, P. (2010). Technology for Classroom Orchestration. In M.S. Khine & I.M. Saleh (Eds.), *New Science of Learning: Cognition, Computers and Collaboration in Education*, 45-52. New York: Springer.

Hernández-Leo, D., Villasclaras-Fernández, E. D., Asensio-Pérez, J. I., Dimitriadis, Y., Jorrín-Abellán, I. M., Ruiz-Requies, I., & Rubia-Avi, B. (2006). COLLAGE: A collaborative Learning Design editor based on patterns. *Educational Technology and Society*, 9(1), 58-71.

Kobbe, L., Weinberger, A., Dillenbourg, P., Harrer, A., Hämäläinen, R., Häkkinen, P., & Fischer, F. (2007). Specifying computer-supported collaboration scripts. *International Journal of Computer-Supported Collaborative Learning*, 2(2-3), 211-224.

Manathunga, K., Hernández-Leo, D. (2017). Authoring and enactment of mobile pyramid-based collaborative learning activities. *British Journal of Educational Technology*, 49(2), 262-275.

Pérez-Sanagustín, M., Burgos, J., Hernández-Leo, D., & Blat, J. (2011). CLFP intrinsic constraints-based group management of blended learning situations. *Studies in Computational Intelligence*, 350, 115-133.

Rodríguez-Triana, M. J. (2014). *Linking scripting and monitoring in blended CSCL Linking scripting & monitoring support in blended CSCL scenarios*. Unpublished Doctoral Dissertation, University of Valladolid, Valladolid, Spain.

Roschelle, J., & Teasley, S. D. (1995). The Construction of Shared Knowledge in Collaborative Problem Solving. In C. O'Malley (Eds.), *Computer Supported Collaborative Learning* (pp. 66-97). Heidelberg: Springer.

Sharples, M. (2013). Shared orchestration within and beyond the classroom. *Computers and Education*, 69, 504-506.

Twining, P., Heller, R., Nussbaum, M., & Tsai, C. (2017). Some guidance on conducting and reporting qualitative studies. *Computers and Education*, 106, A1-A19.

## Acknowledgements

DHL is a Serra Húnter fellow. This work has been partially funded by "la Caixa Foundation" (CoT project, 100010434) and the National Research Agency of the Spanish Ministry of Science, Innovations and Universities MDM-2015-0502, TIN2014-53199-C3-3-R, TIN2017-85179-C3-3-R.
# Please Don't Shoot the Messenger! Prompts in Online Learning Groups - Influences of Nudging Messages' Sender and Publicness on Recipients' Perception and Attribution

Filipa Stoyanova, University of Duisburg-Essen, filipa.stoyanova@uni-due.de
Nicole C. Krämer, University of Duisburg-Essen, nicole.kraemer@uni-due.de

**Abstract:** Co- and socially shared regulation of learning is a challenge for groups in diverse environments – offline and online (Järvelä et al., 2014). Specifically, prompting others to do their share of the work, might be challenging for interpersonal relationships. Thus, addressing the social psychological dynamics in online learning environments, an online experiment was conducted (N = 352). In a 2x(2x)2 between-subjects design the nature of the sender (group member/system), the sender's past proficiency (high/low) and publicness (private/public message) were varied to explore whether automated mediation can reduce detrimental effects and enhance impartial message perception, sender impression and causal attribution. Contrasts by sender and proficiency demonstrate improved message perception, sender impression and internal causal attribution when prompts were sent by the system instead of average or low proficient fellow students. Among all groups, public nudges indicate more negative, but also more persuasive perception, however system public and private nudges did not differ.

**Keywords:** nudging, CRL, SSRL, automated group support, mediation perception, attribution

## Introduction

High dropout-rates, low levels of participation and course satisfaction are common problems of large online courses (Erdmann et al., 2017). One didactic element that is hailed as potential solution to both, low motivation as well as low understanding is small group cooperation within large online courses. Thereby, “... the interaction process is considered to be a more important element in learning than the outcomes” (Vuopala, Hyvönen, & Järvelä, 2016, p. 26). It has indeed been demonstrated that small group work is related to higher course satisfaction (Bernard et al., 2009). However, to be successful, small group work requires participation and is dependent on group development processes (Walther & Bunz, 2005). Especially when only consisting of few members, the development and existence of small groups might be endangered by dropout-rates and delays of group activities. In order to achieve beneficial social and educational effects, it has been shown to be advantageous when groups are supervised and supported by tutors (Thorpe, 2002). However, in large online courses (e.g. MOOCs) with a high number of small groups, time and staff related capacity is unaffordable. Therefore, one solution might be automated mediation which facilitates interaction in small-groups and prevents common unproductive behaviors. “Nagging” others to do their share of the work has already been identified by educational and social psychologists as one of eight common detrimental problems in small-group collaboration (Strauß, Rummel, Stoyanova, & Krämer, 2018) that might be addressed by an automatic tutor system.

## Theoretical background

So far, “nagging” or nudging other group members is barely researched as a social psychological challenge for interpersonal relationships in small group collaboration, hence it has not been clearly defined or conceptualized. The term “nudging” has been defined as a form of soft paternalism helping to defeat cognitive or behavioral biases in decision making, as “any aspect of the choice architecture that alters people’s behavior in a predictable way without forbidding any options or significantly changing their economic incentives” (Wang et al., 2013, p. 376). Nudging is the case, for instance, when displaying driver’s current speed instead of forcing them to slow down. Such indirect means have been successfully applied in the context of health or food and for the purpose of online privacy (e.g. Acquisti et al., 2017). Nudging in the current study refers to hints in order to regulate others’ activities, i.e. unbalanced co- or socially shared regulation of learning (Isohätälä, Järvenoja, & Järvelä, 2017). Thus, in the context of learning groups, nudges can be understood as negative feedback, co- or socially shared regulation of learning, that might lead to task and person conflicts and can be assumed as either beneficial or detrimental. For instance, effective non-individual regulation of learning, as well as task conflicts are beneficial due to the facilitation of group discussions, decision making, performance and knowledge building (Janssen, Van de Vliert, & Veenstra, 1999), but failures and their effects on interpersonal relationships are yet unknown. Furthermore, socially shared regulation of learning was shown not to function well in case of disengagement,
unequal participation and contribution (Isohätälä et al., 2017). Specifically person conflict reduces teams'
effective communication and cooperation, tolerance for opposition, and openness to ideas of disliked team
members, and simultaneously improves hostile attributions of others' intentions and behaviors (Janssen et al.,
1999). Relationship conflict also was found to be linked to decreased perception of task performance (Mohammed
& Angell, 2004). While under certain circumstances the more beneficial and desirable task conflict can facilitate
group discussions, these can easily transform into more detrimental person conflicts (e.g., due to the fact that
group members can hardly distinguish between them, Janssen et al., 1999). An explanation for the
misinterpretation of tasks conflicts as person conflicts can be the need for consistency, i.e. disliked positions
regarding the task can lead to corresponding personal attitudes toward the author. Furthermore, even justified
criticism can generate reciprocal dis-liking and conflicts (Ilgen, Mitschell, & Frederickson, 1981).

Negative feedback has also been shown to have detrimental effects in groups (Gabelica, Van den
Bossche, Segers, & Gijselaers, 2012). Additionally, personal remarks and conflict hinder the success of groups'
decision making process (Walther, 1996). Socio-emotional concerns like conflicts distract from the substantive
task, and are especially time consuming in computer mediated communication (CMC), but can also easily get
restricted on purpose in CMC settings (reducing channels, synchronicity etc.). Given the fact that nudging in
learning groups occurs specifically based on missing contribution, it resembles feedback which has also been
discussed as a possible challenge for interpersonal relationships. Feedback has been defined repeatedly depending
on focus and context, as a comparison of actual performance and a desired standard (e.g. Gabelica et al., 2012).
Beyond the aforementioned social-psychological perspective on group interaction as needed but potentially
dysfunctional, peer interaction has been found to improve performance and learning outcomes – especially when
guiding and support is present to avoid less efficient learning behaviors (Diziol, Walker, Rummel, & Koedinger,
2010). Feedback could be a supportive guiding solution, however, besides many benefits, it also turns out to be
demotivating and leads to performance decreases (Kluger & DeNisi, 1996). Especially negative feedback can get
unfavorable and result in inefficient behaviors (Ilgen & Davis, 2000). Conversely to the readily accepted positive
feedback, negative feedback has been shown to be rejected and perceived as invalid and inaccurate (Ditto &
Boardman, 1995). On the other hand, it has been found that compared to positive feedback, students prefer more
negative feedback, as especially constructive if it points out a lacking goal progress instead of a decrease in
engagement (for review see Fishbach, Eyal & Finkelstein, 2010). Similarly, negative feedback was concluded to
have potential detrimental effects on members' affective reactions but may also have positive motivational effects
(Gabelica et al., 2012), for instance regarding regulation of learning. Thus, research showed contradicting results
and similar to conflicts, both detrimental and beneficial effects of negative feedback can be assumed.

Based on these findings it can be assumed that when one group member nudges another detrimental
effects occur that endanger group climate and interpersonal relationships. Especially, once negative attribution of
a certain group member or group conflicts occur, it can be more difficult to resolve conflicts and repair one's
image or the wellbeing on a group level. One potential solution can be an external envoy as a mediator of "negative
feedback" that would otherwise needed to be delivered by other group members.

Relevance of feedback source

Feedback can be based on a subjective opinion or an objective measure and derive from an internal or external
source. Here, group members might be categorized as an internal feedback source as well as providing a subjective
opinion. Automated mediated feedback, such as messages deriving from the system as an external source, might
be categorized as potentially based on more objective measures. Feedback is more likely perceived as accurate,
when deriving from more credible, powerful or knowledgeable sources (e.g. London & Sessa, 2006). This,
however, was mainly shown in the classic teaching context regarding learning processes and outcomes (e.g. Finn
et al., 2009).

Gabelica et al. (2012) conclude that feedback intervention effectiveness might be improved if feedback
is accurate, given in a timely manner, regular, given directly to teams it targets, *shared, non-threatening, and
when its distribution is fairly equal*. However, these characteristics could vary depending on the source, as notably
the last three could not be secured or held constant when feedback is given in a subjective manner by group
members, but they can potentially be adjusted explicitly when providing feedback by an external system, as
described in the next section regarding the media equation theory.

Past proficiency

Regarding the effects of feedback, it is not only important who the sender is but also his/her attributes and prior
behavior will play a role. One important aspect is past proficiency of group members as it has been described in
group dynamic theories referring to the term of idiosyncrasy (i.e., possessing unique characteristics or showing
unique behavior). In order to be allowed to derogate from a group's normative standards the individual member
must have shown high achievement in the past, i.e. must have earned a high level of “idiosyncrasy credit” (Hollander, 1958). In its origin, it considers leaders acquiring credit over time by performing continuously well and following group norms. As a credit of trust it enables one to deviate from the norms. Applied to learning groups this credit may be a basic factor in the case of the common problems of unequal contribution, social loafers and free riders.

As a recipient of a nudging message one could perceive it as inappropriate to receive any nudging remarks from a member with low past proficiency. Conversely, high past proficiency as a credit of trust and high contributor image as a kind of a group leader could turn into special rights to announce feedback within the group. This may be a fundament for the effectiveness of the message e.g. persuading the recipient to reconsider an act due to higher internal causal attribution.

## Media equation theory

Automated support is not simply an accurate and financially effective method to support high number of groups in online learning settings. In the sense of a mediator it could be perceived more impartial and can potentially prevent the occurrence of interpersonal conflicts in the case of nudging group members to do their share of the work. In case negative feedback is being sent by the system it can be less frustrating. As postulated in the media equation theory, interactions with computers and media can be perceived as real life interactions (Reeves & Nass, 1996). People have been demonstrated to treat computers similar to people, e.g. they avoid to deliver directly negative feedback. However, research in the area of information communication technology and human-robot-interaction has already shown some differences and limitations of the media equation theory. For instance, people asked to abuse a robot in an adaption of the Milgram experiment, more likely did so, when abusing a robot instead of a human (Bartneck, Rosalia, Menges, & Deckers, 2005). Furthermore, compared to other humans, virtual humans were shown to increase the willingness to disclose confidential information with them, although they are *only* computers (Lucas, Gratch, King, & Morency, 2014). Hence, it can be assumed that negative feedback from an automated system might be perceived differentially, e.g., as more impartial, since message recipients could not blame the system and its *personal* reasons for presenting feedback. Recipients could still indeed perceive the system as another human and interact with it equally, as the media equation theory postulates. However, the knowledge that “the system” has no past proficiency (neither high nor low) or any personal intentions can be significant for different perception. We assume that even if people blame the system, it would be on a level according to the missing, i.e. neutral past proficiency instead of high or low as in the case of other members’ feedback.

## Public versus private nudges

Beyond source, the way the message is presented needs to be scrutinized. One important aspect of presenting nudges in groups is whether these are targeted privately at the deviating person or displayed publicly in the group. The psychologically relevant construct in this context is ego threat. Negative feedback might pose an ego threat to one's self-image or public-image (Audia & Locke, 2003). An ego threat of self-image provides information contrary to the own beliefs about the self, whereas for public-image respectively information contrary to one's self targeted impression to others. Less or late contribution can be reasoned diversely, consciously or unconsciously, i.e. fitting or contrary to the self-image. Respectively nudges to hurry up and contribute more can be contrary to or fit the self-image too. However, since people generally tend to have a positive self-concept, prompting messages about their missing contribution could serve as a self-image threat. Indeed, threats to the self-image have already been operationalized by providing negative feedback (Leary, Terry, Allen, & Tate, 2009). Regarding the public-private distinction, the mere presence of others during feedback might be sufficient for a public-image threat since self-presentation is a basic need and ubiquitous. In the current study, it can emerge as soon as negative feedback is given in public in the group forum, i.e. others will also be informed about the presumable failure. Prior research on public and private feedback recommended it to be given in both ways in order to activate all potentially positive effects (Alvero, Bucklin, & Austin, 2001), but differences between both types may occur regarding nudging.

## Present study and hypotheses

Based on the reported theories and findings we aim to primarily explore which factors influence the perception of nudging and secondary, whether automated mediation (e.g. prompts sent by the system) can reduce potentially detrimental effects of nudging for interpersonal relationships and group climate. In an online study, in an artificial learning environment, prompt messages from the system as well as from team members will be conceptualized. Participants will imaginarily collaborate in small groups and receive prompting messages by means of visual
mock-ups. The nudging sender (group member vs. system), his proficiency (high vs. low engaged) and the publicness level (nudge in a private vs. group forum message) will be varied as independent variables.

Based on the literature summarized above, we assume differences between the experimental groups regarding causal attribution, emotional affect, sender impression and message perception. The media equation theory postulates, that people interact with machines like they do with humans. However, limitations of the theory already demonstrated different, partially more confidential interactions with machines. Hence, we assume that system nudges will be perceived differently than nudges from human team members – independent of proficiency and publicness.

**H1:** *Compared to prompts from a team member, system prompts improve a) emotional affect, b) sender impression, c) message perception and d) internal causal attribution.*

Beyond this assumption, it cannot be derived whether interaction with the system is still beneficial, when contrasted to humans who showed relevant prior engagement and therefore earned idiosyncrasy credit. We therefore pose the following research questions:

**RQ1.1:** *Compared to prompts from a low proficient team member, do system prompts improve a) emotional affect, b) sender impression, c) message perception and d) internal causal attribution?*

**RQ1.2:** *Compared to prompts from a high proficient team member, do system prompts improve a) emotional affect, b) sender impression, c) message perception and d) internal causal attribution?*

Concerning the publicness of negative feedback and public ego-threat as hostile acts, we state that publicness has an impact on the negative emotional affect, on sender impression and message perception among all experimental groups:

**H2:** *Publicness has an effect among all experimental groups, that increases a) negative emotional affect and b) negative message perception, but decreases c) persuasive message perception and d) sender impression.*

However, the main effect among all groups does not deliver details to scrutinize which publicness level of system nudges is perceived less threatening. Therefore we explore the effect of publicness in system treatment groups, exclude further influences, and state the research question:

**RQ2:** *Among system treatment conditions, is there an impact of publicness on a) negative emotional affect, b) negative message perception, c) persuasive message perception and d) sender impression?*

## Method

Participants were instructed to imagine that they participate in an online learning group, did not provide their contribution shortly before the deadline and therefore received a prompting message. The nudging sender (group member vs. system), past proficiency of sender (high vs. low engaged) and the publicness level (nudge in a private vs. group forum message) were varied as independent variables. The design is not fully crossed as proficiency of the sender can only be varied in the team member conditions, not in the system conditions. To immerse the participants we created visual vignettes based on the view of a group member account in the learning environment Moodle and adapted them to the specific conditions (Figure 1). The group context was described in advance by short texts and charts on the past progress of the group and group members' past proficiency regarding contributions' quantity and timeliness. Participants were presented a prompt message in form of a visual mock-up either from the system or from a high or low proficient teammate, and either as a private (inbox) or public (forum) message. The use of an artificial learning environment allowed us to remind participants of the group context in the main message with the aid of a Moodle tool presenting group members' past online activities.

<figure><img src="image_116.png" /><figcaption>**Figure 1.** Exemplary vignette with variable conditions (1 = sender, 2 = publicness, 3 = proficiency).</figcaption></figure>

Regarding the *sample*, the study was approved by the ethics committee of the University. A total of 444 participants were randomly assigned to one of the six conditions. They were mainly recruited through advertisements in Facebook groups and incentivized in a lottery. Additionally we used a crowdsourcing website with postpaid incentives. 92 persons were excluded from further analyses as they spent less than 10 seconds at the stimulus material pages (vignettes), their (nick)names were shorter than three letters and due to missing data. The remaining 352 participants (235 female, 117 male (33.2%)) ranged in age from 18 to 69 years (M = 29.40, SD = 10.46). Most of the participants had a university entrance degree (40.9%) or a higher degree (46.1%) and
were predominantly students (228, 64.8%) with a medium attitude towards group work participation (M = 2.95, SD = 1.04, 1-5).

Regarding *measures*, adjective item lists were adapted from various affective scales and lists for emotional affect, message perception and sender impression. Finally, for each measure, an explorative factor analysis according to Horn (1965) was conducted and the recommended factor solution was chosen.

*Emotional Affect* was measured with a list of 23 adjective items, all employing a 5-point Likert scale (1 to 5 = strongly agree). Factor analysis revealed a 3-factor solution: *Positive* affect (α = .841, 8 items, e.g. “inspired”), *negative external* affect (α = .907, 10 items, “humiliated”), and *negative internal* affect (α = .786, 5 items, “guilty”).

*Message Perception* was measured by 18 single items, employing a 5-point Likert scale (1 to 5 = strongly agree) and divided in 3 factors revealed by factor analyses: *Negative* (α = .841, 5 items, “hostile”), *positive* (α = .894, 7 items, “needed”) and *fair* (α = .827, 6 items, “impartial”). Additionally, as a further perception dimension, persuasiveness of the message was measured with an adapted version of the perceived persuasiveness scale from Orji, Vassileva and Mandryk (2014), employing a 7-point Likert scale (1 to 7 = strongly agree). An additional self-generated item regarding reluctant behavior was included. One-factor solution was applied – *persuasive* (α = .840, 5 items, “The prompt would persuade me”).

*Positive Sender Impression.* Semantic differentials were applied (e.g. 1 = unfriendly to 5 = friendly) to measure impression from the sender and taken according to factorial analysis as one-factor solution (α = .905, 9 items).

*Causal Attribution.* We measured how participants attribute why they received the nudging message, whether it was their fault (internal) or others’ (external). Based on Lefcourt (1981), 8 items with daily internal and external reasons were generated on a 5-point Likert scale (1 to 5 = strongly agree). After a factor analysis the dimensions were combined in an overall factor *Internal causal attribution* (α = .726, 5 items, “Because I was lazy”).

*Other Measures.* We employed one item measures to assess socio demographics (e.g. age, education, gender), past group work experiences (both quantity and valence), and attitude towards group work. We additionally measured self-esteem, perfectionism, and causal attribution style as traits, as well as further personality traits, which are not relevant for the analyses presented here.

## Results

Regarding *hypothesis 1*, after inspecting descriptive values (table 1), a planned comparison was conducted to test whether the system as a nudging agent compared to team members improves a) emotional affect, b) sender impression, c) message perception and d) internal causal attribution. Therefore, we compared both system-

<table><caption>Table 1: Descriptive values of dependent variables across treatment groups</caption><thead><tr><th rowspan="2">↓DV / Treatment Group→ 1. SYpr</th><th colspan="2">2. SYpu</th><th colspan="2">3. TMpr-</th><th colspan="2">4. TMpr+</th><th colspan="2">5. TMpu-</th><th colspan="2">6. TMpu+</th><th colspan="2">Total</th></tr><tr><th>M</th><th>SD</th><th>M</th><th>SD</th><th>M</th><th>SD</th><th>M</th><th>SD</th><th>M</th><th>SD</th><th>M</th><th>SD</th></tr></thead><tbody><tr><td colspan="14"><strong>Emotional Affect</strong></td></tr><tr><td>Negative internal</td><td>3.75</td><td>.77</td><td>3.72</td><td>.81</td><td>3.27</td><td>.90</td><td>3.72</td><td>.77</td><td>3.36</td><td>.86</td><td>3.78</td><td>.80</td><td>3.59</td><td>.84</td></tr><tr><td>Negative external</td><td>2.47</td><td>.96</td><td>2.66</td><td>.82</td><td>2.57</td><td>.80</td><td>2.62</td><td>.92</td><td>2.82</td><td>.94</td><td>2.88</td><td>.97</td><td>2.67</td><td>.91</td></tr><tr><td>Positive</td><td>2.22</td><td>.73</td><td>2.34</td><td>.75</td><td>2.05</td><td>.75</td><td>2.32</td><td>.76</td><td>2.16</td><td>.72</td><td>2.23</td><td>.79</td><td>2.22</td><td>.75</td></tr><tr><td colspan="14"><strong>Message Perception</strong></td></tr><tr><td>Negative</td><td>2.46</td><td>1.05</td><td>2.78</td><td>.86</td><td>2.82</td><td>.90</td><td>2.57</td><td>.98</td><td>2.90</td><td>.95</td><td>2.78</td><td>.89</td><td>2.72</td><td>.95</td></tr><tr><td>Positive</td><td>3.00</td><td>.90</td><td>2.95</td><td>.77</td><td>2.41</td><td>.90</td><td>2.97</td><td>.98</td><td>2.52</td><td>.94</td><td>2.77</td><td>.83</td><td>2.77</td><td>.92</td></tr><tr><td>Fair</td><td>3.19</td><td>.85</td><td>2.96</td><td>.81</td><td>2.68</td><td>.90</td><td>3.29</td><td>.91</td><td>2.77</td><td>.84</td><td>3.12</td><td>.71</td><td>3.00</td><td>.86</td></tr><tr><td>Persuasive</td><td>5.15</td><td>1.41</td><td>5.32</td><td>1.13</td><td>4.28</td><td>1.39</td><td>4.76</td><td>1.45</td><td>4.83</td><td>1.06</td><td>5.10</td><td>1.34</td><td>4.90</td><td>1.33</td></tr><tr><td colspan="14"><strong>Positive Sender Impression</strong></td></tr><tr><td></td><td>3.38</td><td>.66</td><td>3.21</td><td>.69</td><td>2.86</td><td>.85</td><td>3.35</td><td>.80</td><td>2.84</td><td>.90</td><td>3.23</td><td>.70</td><td>3.14</td><td>.80</td></tr><tr><td colspan="14"><strong>Internal Causal Attribution</strong></td></tr><tr><td></td><td>3.86</td><td>.55</td><td>3.88</td><td>.63</td><td>3.34</td><td>.77</td><td>3.68</td><td>.72</td><td>3.41</td><td>.66</td><td>3.85</td><td>.57</td><td>3.66</td><td>.69</td></tr></tbody></table>

message-conditions (groups 1 & 2) in planned contrasts to all the team-member-conditions (groups 3, 4, 5 & 6). The contrast revealed significant differences, indicating increased levels in the system conditions regarding internal negative emotional affect, *t*(346) = -2.20, *p* = .029, *r* = .12, but also message perception positive, *t*(346) = -2.99, *p* = .003, *r* = .16, and persuasive, *t*(346) = -3.35, *p* = .001, *r* = .18, generally positive sender impression, *t*(263.12) = -2.76, *p* = .006, *r* = .17, and internal causal attribution, *t*(346) = -4.03, *p* < .001, *r* = .13. *Hypothesis 1* is partially supported since the general comparison of all treatment groups divided by sender showed significant differences and an improvement of the positive and persuasive message perception, sender impression, as well as higher levels of internal causal attribution, whereas, contradicting to *hypothesis 1*, the internal negative emotional
affect was higher. However, the system as a sender, compared to the groups with a team member sender, did not reveal significant differences regarding fair and negative message perception.

*Research question 1.1* and *1.2* investigate whether compared to a message sent by a team member with high or low past proficiency, a message sent by the system improves a) emotional affect, b) sender impression, c) message perception and d) internal causal attribution, we conducted planned contrasts. Due to the factor proficiency which was varied in all experimental groups with a team member as a sender, we had to split the analyses to avoid conflicting high and low proficiency to be summarized. Thus, we compared the system-message-conditions (1 & 2) in planned contrasts to the team-member-conditions divided by proficiency. For *RQ1.1* we compared them to those with low past proficiency (exp. groups 3 & 5). Contrasts revealed that a system message, compared to one from a low proficient team member increased internal negative emotional affect, *t*(346) = 3.97, *p* < .001, *r* = .21, message perception, (fair, *t*(346) = 3.19, *p* = .002, *r* = .17, positive, *t*(346) = 4.38, *p* < .001, *r* = .23, and persuasive, *t*(346) = 4.04, *p* < .001, *r* = .21), positive sender impression, *t*(223.46) = 4.41, *p* < .001, *r* = .23, as well as internal causal attribution, *t*(346) = 5.81, *p* < .001, *r* = .22. There were no significant effects on external negative emotional affect and negative perception of the message. Except for these two subscales, the system-message-condition, compared to low proficient team-member-conditions, had an improving effect. For *RQ1.2* we compared both system-message-conditions (1 & 2) to those with high past proficiency team-member-conditions (exp. groups 4 & 6). However, there was no significant difference compared to high proficient team members. *Research question 1.1* revealed that messages from the system compared to those from low proficient team members improved internal causal attribution, positive sender impression, as well as fair, positive and persuasive message perception. However, on the other hand the system had detrimental effects regarding increased internal negative emotional affect. *Research question 1.2* did not reveal significant differences between the system and high proficient members.

*Hypothesis 2* was tested in a MANOVA, conducted among all treatment groups to test the influence of publicness on the experimental groups regarding increasing a) negative emotional affect and b) negative message perception, but decreasing c) persuasive message perception and d) sender impression. There was a significant effect of publicness, *V* = 0.64, *F*(5, 346) = 4.77, *p* < .000. Further separate univariate ANOVAs revealed significant effects on a) external negative emotional affect, *F*(1, 350) = 5.69, *p* = .018, η² = .016, b) negative message perception, *F*(1, 350) = 4.44, *p* = .036, η² = .013 and c) persuasive message perception, *F*(1, 350) = 5.66, *p* = .018, η² = .016. However, there was no significant effect on d) sender impression. All significant effects indicated higher levels in public, rather than in private nudges, i.e. more negative emotional affect and more negative, but also more persuasive message perception. Therefore *H2* is partly supported as in the public condition a) negative emotional affect, b) negative message perception and c) persuasive message perception increased, but there was no significant effect on d) sender impression.

*Research Question 2* addressed the publicness of a system message and whether public and private system prompts differ regarding a) negative emotional affect, b) sender impression and c) negative and persuasive message perception. Therefore a MANOVA was conducted, by comparing solely the experimental conditions with a system sender. No significant differences were revealed, *V* = 0.12, *F*(15, 99) = .893, *p* = .574.

## Discussion

In order to gain the benefits of group conflict without the costs, we focus on the common, but barely researched nudging in groups. In an online experiment we addressed the social psychological dynamics in online learning groups and explored potentially influential factors for the perception of nudging, i.e. the sender, sender's past proficiency, as well as publicness of the message. The data indicated that nudging messages sent not by a team member but by the system, were perceived more positive and persuasive, improved sender impression and internal causal attribution. In line with prior research (Lucas, Gratch, King, & Morency, 2014) this indicates that a message from a human is not always equal to a message from the system and that it can be beneficial if a system instead of a fellow human delivers unpleasant messages. Contradicting our assumptions, however, internal negative emotional affect was also higher, potentially due to the fact that participants were more likely to blame themselves when the system confronted them with negative feedback. Also against our assumptions, the system messages were not perceived more impartial. Future research will need to show whether this – in the sense of the media equation (Reeves & Nass, 1996) – actually indicates that humans and machines are subject to the same person perception mechanisms or whether this result is due to different expectations towards machines and humans.

While there was no difference between highly proficient team members and the system, the system was perceived more positive compared to low proficient team members. Keeping the idiosyncrasy credit and the media equation theory in mind, participants may think of the system differently and still attribute a level of past proficiency to the system or at least attribute sufficient competences to the system to accept that it judges oneself. Alternatively, the system might have been accepted as a truly neutral evaluator who is allowed to utter feedback
just as much as a proficient peer is. In order to address these open questions, systems' error rates should be taken into account in future studies demonstrating the systems' past proficiency. Our findings further affirm some studies in educational artificial intelligence and tutoring systems. Compared to human-tutors, intelligent computer-tutoring was shown as equally and more effective independent of time and context (for review Kulik & Fletcher, 2016) and concluded as needed only if beneficial for performance and learning (Ostrander et al., 2019). However, the mere system perception and social-psychological group dynamics were disregarded.

Regarding limitations this study so far only elucidates the effect of system nudging in the artificial context of imaginary groups. Future field studies are needed in field setting and real groups to replicate the findings. A survey analysis was applied that primarily focused on perception, but behavioral data may be more promising to investigate the topic and its links to students' learning processes and outcomes. Finally, it has to be noted that all effect sizes were small.

In conclusion, having the tutor-system deliver nudges seems to be a promising solution for a specific form of group conflicts. Future research should also include and consider the system's potential embodiment and other cues such as natural language output. The more we learn about the conditions under which negative feedback can unfold positive effects (and the messenger does not have to be shot), the better can tutoring systems be improved to support group dynamics.

## References

Acquisti, A., Adjerid, I., Balebako, R., Brandimarte, L., Cranor, L. F., Komanduri, S., ... Wang, Y. (2017). Nudges for privacy and security: Understanding and assisting users' choices online. *ACM Computing Surveys (CSUR)*, 50(3), 1–41. http://dx.doi.org/10.1145/3054926

Alvero, A. M., Bucklin, B. R., & Austin, J. (2001). An objective review of the effectiveness and essential characteristics of performance feedback in organizational settings (1985–1998). *Journal of Organizational Behavior Management*, 21(1), 3–29. https://doi.org/10.1300/J075v21n01_02

Audia, P. G., & Locke, E. A. (2003). Benefiting from negative feedback. *Human Resource Management Review*, 13(4), 631–646. https://doi.org/10.1016/j.hrmr.2003.11.006

Bartneck, C., Rosalia, C., Menges, R., & Deckers, I. (2005). Robot abuse – a limitation of the media equation. In *Proceedings of the Interact 2005 Workshop on Abuse*. Rome, IT.

Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A., & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. *Review of Educational Research*, 79(3), 1243–1289. https://doi.org/10.3102/0034654309333844

Ditto, P. H., & Boardman, A. F. (1995). Perceived accuracy of favorable and unfavorable psychological feedback. *Basic and Applied Social Psychology*, 16(1–2), 137–157. https://doi.org/10.1080/01973533.1995.9646106

Diziol, D., Walker, E., Rummel, N., & Koedinger, K. R. (2010). Using intelligent tutor technology to implement adaptive support for student collaboration. *Educational Psychology Review*, 22(1), 89–102. https://doi.org/10.1007/s10648-009-9116-9

Erdmann, J., Rummel, N., Christmann, N., Elson, M., Hecking, T., Herrmann, T., ... Wichmann, A. (2017). Challenges in implementing small group collaboration in large online courses. In B. K. Smith, B. K., M. Borge, E. Mercier, & K. Y. Lim (Eds.), *Making a difference: Prioritizing equity and access in CSCL, 12th International Conference on Computer Supported Collaborative Learning (CSCL) 2017*, Volume 2. Philadelphia, PA: International Society of the Learning Sciences.

Fishbach, A., Eyal, T., & Finkelstein, S. R. (2010). How positive and negative feedback motivate goal pursuit. *Social and Personality Psychology Compass*, 4(8), 517–530. https://doi.org/10.1111/j.1751-9004.2010.00285.x

Finn, A. N., Schrodt, P., Witt, P. L., Elledge, N., Jernberg, K. A., & Larson, L. M. (2009). A meta-analytical review of teacher credibility and its associations with teacher behaviors and student outcomes. *Communication Education*, 58(4), 516–537. https://doi.org/10.1080/03634520903131154

Gabelica, C., Van den Bossche, P., Segers, M., & Gijselaers, W. (2012). Feedback, a powerful lever in teams: A review. *Educational Research Review*, 7(2), 123–144. https://doi.org/10.1016/j.edurev.2011.11.003

Hollander, E. P. (1958). Conformity, status, and idiosyncrasy credit. *Psychological Review*, 65(2), 117–127. http://dx.doi.org/10.1037/h0042501

Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. *Psychometrika*, 30(2), 179–185. https://doi.org/10.1007/BF02289447

Ilgen, D., & Davis, C. (2000). Bearing bad news: Reactions to negative performance feedback. *Applied Psychology*, 49(3), 550–565. https://doi.org/10.1111/1464-0597.00031
Ilgen, D. R., Mitchell, T. R., & Fredrickson, J. W. (1981). Poor performers: Supervisors' and subordinates' responses. *Organizational Behavior and Human Performance, 27*(3), 386-410.
http://dx.doi.org/10.1016/0030-5073(81)90030-1

Isohätälä, J., Järvenoja, H., & Järvelä, S. (2017). Socially shared regulation of learning and participation in social interaction in collaborative learning. *International Journal of Educational Research, 81*, 11–24.
https://doi.org/10.1016/j.ijer.2016.10.006

Janssen, O., Van de Vliert, E., & Veenstra, C. (1999). How task and person conflict shape the role of positive interdependence in management teams. *Journal of Management, 25*(2), 117–141.
https://doi.org/10.1177/014920639902500201

Järvelä, S., Kirschner, P. A., Panadero, E., Malmberg, J., Phielix, C., Jaspers, J., ... Järvenoja, H. (2014). Enhancing socially shared regulation in collaborative learning groups: designing for CSCL regulation tools. *Educational Technology Research and Development, 63*(1), 125–142.
https://doi.org/10.1007/s11423-014-9358-1

Kluger, A. N., & DeNisi, A. (1996). Effects of feedback intervention on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. *Psychological Bulletin, 119*(2), 254–284.
https://doi.org/10.1037/0033-2909.119.2.254

Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: A meta-analytic review. *Review of Educational Research, 86*(1), 42–78. https://doi.org/10.3102/003465431558142

Leary, M. R., Terry, M. L., Allen, A. B., & Tate, E. B. (2009). The concept of ego threat in social and personality psychology: Is ego threat a viable scientific construct? *Personality and Social Psychology Review, 13*(3), 151–164. https://doi.org/10.1177/1088868309342595

Lefcourt, H. M. (Ed.). (1981). *Research with the locus of control construct*. Orlando, FL: Academic Press.

London, M., & Sessa, V. I. (2006). Group feedback for continuous learning. *Human Resource Development Review, 5*(3), 303–329. https://doi.org/10.1177/1534484306290226

Lucas, G. M., Gratch, J., King, A., & Morency, L. P. (2014). It's only a computer: Virtual humans increase willingness to disclose. *Computers in Human Behavior, 37*, 94–100.
https://doi.org/10.1016/j.chb.2014.04.043

Mohammed, S., & Angell, L. C. (2004). Surface- and deep-level diversity in workgroups: Examining the moderating effects of team orientation and team process on relationship conflict. *Journal of Organizational Behavior, 25*(8), 1015–1039. https://doi.org/10.1002/job.293

Orji, R., Vassileva, J., & Mandryk, R. L. (2014). Modeling the efficacy of persuasive strategies for different gamer types in serious games for health. *User Modeling and User-Adapted Interaction, 24*(5), 453–498.
https://doi.org/10.1007/s11257-014-9149-8

Ostrander, A., Bonner, D., Walton, J., Slavina, A., Ouverson, K., Kohl, A., ... & Winer, E. (2019). Evaluation of an Intelligent Team Tutoring System for a Collaborative Two-Person Problem: Surveillance. *Computers in Human Behavior* (in press). https://doi.org/10.1016/j.chb.2019.01.006

Reeves, B., & Nass, C. I. (1996). *The media equation: How people treat computers, television, and new media like real people and places*. Chicago, IL: Center for the Study of Language and Information; New York, NY: Cambridge University Press.

Strauß, S., Rummel, N., Stoyanova, F., & Krämer, N. (2018). Developing a library of typical problems for collaborative learning in online courses. In J. Kay & R. Luckin (Eds.) (2018), *Rethinking Learning in the Digital Age: Making the Learning Sciences Count, 13th International Conference of the Learning Sciences (ICLS) 2018, Volume 2* (pp. 1045-1048). London, UK: International Society of the Learning Sciences.

Thorpe, M. (2002). Rethinking Learner Support: The challenge of collaborative online learning. Open Learning: *The Journal of Open, Distance and e-Learning*, 17(2), 105–119.

Vuopala, E., Hyvönen, P., & Järvelä, S. (2016). Interaction forms in successful collaborative learning in virtual learning environments. *Active Learning in Higher Education, 17*(1), 25–38.
https://doi.org/10.1177/1469787415616730

Walther, J. B. (1996). Computer-mediated communication: Impersonal, interpersonal, and hyperpersonal interaction. *Communication Research, 23*(1), 3–43. https://doi.org/10.1177/009365096023001001

Walther, J. B., & Bunz, U. (2005). The rules of virtual groups: Trust, liking, and performance in computer-mediated communication. *Journal of Communication, 55*(4), 828–846. https://doi.org/10.1111/j.1460-2466.2005.tb03025.x

Wang, Y., Leon, P. G., Scott, K., Chen, X., Acquisti, A., & Cranor, L. F. (2013). Privacy nudges for social media: an exploratory Facebook study. In *Proceedings of the 22nd International Conference on World Wide Web* (pp. 763–770). New York, NY: ACM. https://doi.org/10.1145/2487788.2488038
# How Augmented Reality Affects Collaborative Learning of Physics:
## A Qualitative Analysis

Apittha Unahalekhaka, Iulian Radu, and Bertrand Schneider

a.unahalekhaka@gmail.com, iulian radu@gse.harvard.edu, bertrand schneider@gse.harvard.edu
Harvard University, Graduate School of Education

**Abstract:** Augmented reality (AR) is a powerful visualization tool to support learning of scientific concepts across learners of various ages. AR can make information otherwise invisible visible in the physical world in real-time. In this study, we are looking at a subset of data from a larger study (N=120), in which participant pairs interacted with an augmented sound producing speaker. We explored the learning behaviors in eight pairs of learners (N=16) who participated in an unstructured physics activity under two conditions: with or without AR. Comparing behaviors between the two experimental conditions, we found that AR affected learning in four different ways: participants in the AR condition (1) learned more about visual concepts (ex: magnetic field structures) but learned less about nonvisual content (ex: relationship between electricity and physical movement); (2) stopped exploring the system faster than NonAR participants; (3) used less aids in exploration and teaching; and (4) spent less time in teaching their collaborators. We discuss implications of those results for designing collaborative learning activities with augmented reality.

## Introduction and related work

Numerous learners find it challenging to master concepts where abstract topics cannot be explicitly seen and experienced in everyday life. For this reason, different aids have been used to make these concepts tangible. In physics education, tools such as visual representations (e.g., graphs, diagrams, icons) and physical manipulatives (e.g., blocks, magnets, compass) are frequently used as teaching aids (Zacharia & Olympiou, 2011; Savinainen et al, 2015; Suyatna et al, 2017). One issue with the usage of multiple representations is that learners must understand and integrate them (Kohl et al, 2007; Ainsworth 2008). For instance, Ainsworth (2008) has found that to use different representations effectively, learners must understand the relationship between the representation and the concept they are learning and understand what information the representation is carrying. This is especially problematic when different representations are presented in different places at different times (Bujak et al, 2013). Accordingly, using hand gestures (e.g., deictic and iconic) to connect multiple concepts together can reduce cognitive load and improve students' learning (Alibali et al, 2012; Alibali et al, 2013).

Augmented reality (AR) is an emerging technology that has become increasingly common to visualize abstract scientific learning (Wu, 2013). Augmented reality provides multiple affordances that can be beneficial for learning (Radu, 2014; Bujak et al, 2013). For example, AR can help reduce the cognitive load in learners while they are learning with multiple representations, by linking abstract representations to concrete physical representation through spatial and temporal contiguity (Radu, 2014; Bujak et al, 2013). This affordance is especially suitable for physics education, where abstract concepts accompany physical models. In one study of using AR for Newtonian force concepts, Enyedy et al (2012) found that children ages 6-8 years old were able to learn force concepts when AR was incorporated into role play. Cai et al (2017) revealed that AR-based motion-sensing tools could help junior high school students learn concepts about magnetic fields more intuitively, and that students were able to retain the concepts longer. Those findings suggest that AR can be an interesting medium for supporting collaborative learning. In the section below, we describe the larger experiment we ran and how we qualitatively analyze eight pairs from it.

## General description of the study

A larger study (Radu & Schneider, 2019) compared 60 pairs of participants (N=120) who interacted with an augmented physical model of an audio producing speaker (Figure 1). The pairs were assigned to either do the activity with or without seeing the augmented reality (AR) visuals. The NonAR condition participants were given a physical model, a compass, and informational posters on the wall (Figure 1, left), while the AR condition participants were given the same tools, and also various AR visuals in addition to what the NonAR participants were given (Figure 1, right). There were multiple representational tools to support participants in both conditions, with the main difference being that AR condition received visual representation aligned with the physical system, whereas the NonAR condition received representations not overlaid on the physical system.
For this paper we focused on analyzing video recordings of eight pairs (n=16), which were chosen according to two variables: AR versus NonAR; and overall learning gains: high learning gains versus low learning gains. This is a 2x2 design with two pairs of students in each condition.

<figure><img src="image_117.png" /><figcaption>**Figure 1.** Physical model with no AR (left); users wearing two Microsoft Hololens® and interacting with the system (middle); Physical model with AR (right).</figcaption></figure>

We found that participants in AR learned more about spatial structures (ex: shapes of magnetic fields) but learned less about non-visual concepts (ex: relationship between physical movement and electricity). Furthermore, AR groups had higher levels of engagement and improved perception of self-efficacy (Figure 2):

<figure><img src="image_118.png" /><figcaption>**Figure 2.** Group differences in relative learning gains in percentage (left); and overall attitudes in 5-point scale (right). Red= AR group, Green = NonAR group; whiskers show standard errors (for more information, please refer to (Radu & Schneider, 2019)).</figcaption></figure>

In this paper we build upon this project by using the same dataset and qualitatively analyzing the learning and teaching behaviors of participants, comparing between AR-vs-NonAR conditions, by comparing groups that had high learning vs low learning. We are interested in understanding specifically why participants learned differently between the two conditions. Our three research questions are as follows:

**Research Question 1:** How does AR impact collaboration?

**Research Question 2:** How does AR impact the use of external aids and gestures?

**Research Question 3:** How does AR impact teaching behaviors?

## Method

This paper analyzes video recording of participants learning by using the system in the AR and NonAR conditions. During the activity two participants had to work together to complete a worksheet on electromagnetism; the worksheet asked questions such as “Where is the strongest magnetic field located?” and “Draw the shape of the magnetic field when a cup is closer versus further”. At fixed times during the experience, the facilitator would ask questions to participants, such as “How is the direction and the strength of the electric current influencing the cup?”, which were meant as provocation to think about various aspects of the system, but did not require explicit answering.

We performed qualitative video analysis to understand participant behavior. A coding scheme was constructed through iterative bottom-up coding. The final coding scheme consisted of three main categories: (1)
communication type; (2) aid provided for communication; and (3) method of communication. Research study videos from the eight participant sessions were split into 30 seconds time frames and assigned one or more codes. For inter-rater reliability, one rater coded 100% of the videos, while the other rater coded 20% of each video. The eight analyzed sessions ranged between 27-33 minutes. In total, 489 30-sec. segments were coded, accounting for 4 hours of video. Inter-rater reliability reached a Cohen kappa of 0.8, which implies substantial/almost perfect agreement. The coding scheme categories that were derived from the videos are as follows:

**Communication Types:** Describes the purpose of the participants' communication: *exploring the system, teaching each other, chatting about irrelevant topics, or non-interacting* (Table 1).

**Aid Provided for Communication:** If participants were using an aid while communicating, this code describes the type of tool or representational aid used: *poster, compass, and system*. The last category is coded separately for AR vs. NonAR groups because the system was merged with holographic representations for the AR group; this resulted in two codes for our coding scheme: *using NonAR system* (only applicable for NonAR participants, Figure 1 left), and *using AR system* (only applicable for AR participants, Figure 1, right).

**Methods of Communication.** If participants were using gestures or drawings as a method of communication, this code describes that method: *deictic gesture, iconic gesture, and self-drawing*. Deictic gesture is the type of method that is the easiest to produce, it is when a participant is pointing or using gesture to make the other participant shift his/her attention to a certain location (Roth, 2001). In contrary, iconic gesture and self-drawing has a higher representation level. Iconic gesture is a symbolic gesture when one participant is using his/her hand to mimic a visual, while self-drawing is when one participant is drawing on a paper to support his/her explanation.

<table><caption>Table 1: Definition and Examples of the Communication Types</caption><thead><tr><td>Communication Types</td><td>Definition</td><td>Example</td></tr></thead><tbody><tr><td>Exploring</td><td>At least one of the two participants discuss about the activity with a clear intention to interaction with the other participant. Also includes non-verbal interaction when the participants test out the system together.</td><td>P1: “Alright, let’s see what changes when you pull it down to 10x. Anything change?”<br>P2: “I think it’s quite-er”<br>P1: “Yeah, I agree, it is definitely quite-er”</td></tr><tr><td>Teaching</td><td>At least one of the two participants tries to make the other participant understands a physics concept through explanation or clarification.</td><td>P1: “Look at the green lines over there. There are more lines inside and the magnetic field becomes bigger. The polarity is also changing north and south.”</td></tr><tr><td>Chatting irrelevant</td><td>Both participants talk on a topic unrelated to the physics concept based on the activity.</td><td>P1: “Do you know a lot of physics?”<br>P2: “I forgot most of it”</td></tr><tr><td>Non-interacting</td><td>There is no explicit intention to interact between the two participants, but at least one participant is engaging with the learning activity.</td><td>One participant played with the system, while the other sit still.</td></tr></tbody></table>

## Results

Results from the larger study (Radu & Schneider, 2019) indicate that there are differences in learning and attitudes across the AR vs NonAR conditions (Figure 2). When examining the eight pairs from this current study, we found similar results, namely that participants in the AR condition had higher learning gains in topics involving spatial structures (ex: identifying shapes of magnetic fields), while the NonAR condition participants had higher gains in topics related to physical movement (ex: relationship of magnetic field vs. movement). Additionally, similar to the overall results, participants in the AR condition had a higher tendency than NonAR participants to believe that physics is easy after completing the study. Figure 3 indicates that the eight groups we selected are representative of our sample.
<figure><img src="image_119.png" /><figcaption>**Figure 3.** Pre- and post- assessment: Difference in the attitude (curiosity and whether physics is easy) and learning scores on different areas (relationship: electricity-magnetic fields, electricity-movement, magnetic fields-movement; shapes of magnetic fields) between the 8 dyad groups.</figcaption></figure>

## RQ1: How does AR impact collaboration?

We analyzed how participants across the two conditions spent their time during the study. Figure 4 shows the distribution of participant activities over the 30 minutes of the study. Participants in the AR condition spent only 73% of their session time actively engaged, with the remaining 27% of their time conversing on non-task-relevant topics. In contrast the NonAR condition spent 90% of time actively engaged, and 10% of time conversing on non-task-relevant topics. This pattern is more prominent in groups with low learning gains – low AR groups conversed about non-task relevant topics 47% of the time while low NonAR groups conversed about non-task topics 19%; the high AR groups spent 8% of their time on non-task relevant topics, vs. 1% in high NonAR. This indicates that AR groups have an earlier tendency to believe that they are finished the activity than compared to NonAR groups, and this effect was stronger in groups with low learning gains.

### Communication

<figure><img src="image_120.png" /><figcaption>**Figure 4.** Time sequence for the communication behaviors in each 30 seconds block for the eight groups analyzed in this paper. Null=not exhibiting any of the listed communication types.</figcaption></figure>

We illustrate the phenomenon of AR participants finishing the activity more quickly in Table 2. This example shows how differently AR and NonAR participants react to the same question asked by the facilitator. After hearing the question, the low AR group pause and stare at the AR system. Both participants observe the direction of the cup then come up with their assumption on the relationship between electricity-movement-magnetic fields. They appear to be satisfied with their answers and switch back to conversing on irrelevant topics. From this observation, this AR group seems to have a positive attitude towards the activity, but do not take effort to explore the questions more deeply. In contrast, it took longer for the NonAR group to discuss the same question. Participant 2 uses a lengthy explanation with various iconic hand gestures, while Participant 1 recaps. Subsequently, they use concepts from this discussion to extend their worksheet answer.

<table><caption>**Table 2: Quotes from participants in the Low AR group and Low NonAR group figuring out a question**</caption><thead><tr><th>Low AR Group</th><th>Low NonAR Group</th></tr></thead><tbody><tr><td>Facilitator: "How is the direction and the strength of the electric current influencing the cup?"<br>[Facilitator leaves the room]<br>[P1 and P2 taking turns to press forward/backward buttons and look at the electromagnet with superimposed AR magnetic field]<br>P1: "Oh! It's like the direction is either pushing it away or pulling it closer"<br>P2: "Yeah"</td><td>Facilitator: "How is the direction and the strength of the electric current influencing the cup?"<br>[Facilitator leaves the room]<br>P1: "Is the music linked to the current, so the stronger the current, the stronger the music is?"<br>P2: "Yeah, kinda, So the music is like, little tiny current signals like saying push pull, push pull, modulating. And then it gets amplified from pushing like this, to pushing</td></tr></tbody></table>
<table><tr><td>P1: “The strength when pushing it away is less”<br/>[P1 changing the magnitude of the amplifier and looking at the AR<br/>amplifier graph]<br/>P2: “When the current is weaker, the impact on the membrane of<br/>the cup and the magnetic field is smaller”<br/>P1: “Yeah”<br/>P2: “I don’t know where to add on the paper”<br/>P1: “I think it’s all there” [P1 pointing to their existing answers on<br/>the worksheet]<br/>[Both sitting silently, then chatting on irrelevant topics]</td><td>like that” [P2 using iconic hand gesture showing<br/>different strength]. “The more windings you have, the<br/>stronger it gets. So you are shaking this magnet at a very<br/>specific pace, which then vibrates and makes sound” [P1<br/>looking at P2 while explaining and making iconic hand<br/>gestures]<br/>P1: “Do you mind answering the question, so the<br/>stronger the current, the greater amplification we hear<br/>from the cup?”<br/>[Adding answer to the question that they previously<br/>skipped, while continuing to talk about the activity]</td></tr></table>

## RQ2. How does AR impact the use of external aids and gestures?

We examined how participants in the two conditions used aids and gestures to facilitate learning, by calculating the percentage of time each aid or gesture was used while participants were involved in either exploring or teaching. In the AR condition, the main learning aid was the system, used for the majority of the time (92%). In contrast, in the NonAR condition participants focused on the system 38% of the time, and spent more time using other aids (compass 32% and poster 12%). Figure 5 illustrates the sequence of switching between different aids. Additionally, the AR condition mostly used deictic gesture (47%) with little drawing (3%) and some iconic gesture (9%) to assist with their learning. This pattern contrasts from the NonAR condition that used less deictic gestures (34%), and much more drawing (14%) and more iconic gesture (12%). This suggests that some “tunnel vision” from the AR participants: they totally neglected other resources that were at their disposal.

### Aids

<figure><img src="image_121.png" /><figcaption>Figure 5. Time sequence for aids usage in each 30 seconds block. Null=not using any of the listed aids.</figcaption></figure>

Figure 6 illustrates the sequence of gestures and drawings. These results indicate that the AR condition focused more on the AR-enhanced system and focused more on communication through pointing, while the NonAR participants used the system in concert with other external tools and communicated using iconic gestures and drawings. Again, this suggests more diverse behaviors from the NonAR participants – who used a variety of gestures and drawings to explore the concepts taught.

### Methods

<figure><img src="image_122.png" /><figcaption>Figure 6. Time sequence for communication methods usage in each 30 seconds block. Null=not using any of the listed methods.</figcaption></figure>

## RQ3: How does AR impact teaching behaviors?

While running the study, we observed differences in how participants explained concepts to each other. To further explore this question, we analyzed teaching behaviors between AR and NonAR groups. We found that in the AR condition, participants used 3.72% of the activity time for teaching each other, comparing to 27.39% in the NonAR condition.
condition (Figure 7, left). Analyzing the use of aids in teaching, we found that AR participants only taught while using the system (100% of the teaching time). In the NonAR condition, participants taught using various aids: 33% of the time by using the system, 43% of the time using compass, 11% of the time using posters, and 12% using other methods such as drawings. Aside from the differences in systems/tools used as aids to teach, the participants in the two conditions also utilized different communication methods to teach (Figure 7, right): the AR condition participants mainly used deictic gestures (67% of the time), with little iconic gestures (8%) and no drawings (0%). In contrast, the NonAR condition participants used less deictic gestures (39%), and more iconic gesture (16%), and drawings (18%). These results suggest that the AR condition was able to teach more concisely by just using the AR system and skip the process of acquiring representations from other aids or communication methods needed by the NonAR groups.

<figure><img src="image_123.png" /><figcaption>**Figure 7.** Percentage of time from the total teaching time spent on using each aids (left) and methods (right).</figcaption></figure>

As an illustration of this difference, Table 3 shows how an abstract concept (magnetic polarity) is taught more easily with AR. Without AR, participants read polarity by using a compass, whose use required learning and experimentation. Participant 1 from the NonAR group (Table 3, right) slowly taught Participant 2 on how to read polarity from a compass, whereas in the AR group (Table 3, left) Participant 1 was able to instruct the other participant to look at the polarity labels (North / South) provided in AR near the AR magnetic fields. In this scenario, the AR group is able to skip the process of understanding how to physically read the magnetic polarity, and directly talked about alternative current. In contrast, the NonAR group took a longer time to learn about current in relation to magnetic direction, and remained focused on understanding the slow motions of the magnet rather than fast oscillations resulting in music.

<table><caption>**Table 3: Quotes from participants in the AR group and NonAR group teaching**</caption><thead><tr><th>AR Group</th><th>NonAR Group</th></tr></thead><tbody><tr><td><p>[Both looking at the magnetic field AR]</p><p>P1: "This button pushes the magnet in and out..."</p><p>P2: "Um hm"</p><p>P1: "Because the magnetic field changes, you see? It's inverted. It changes between north and south." [P1 referring to the AR magnetic field]</p><p>P2: "Yea"</p><p>P1: "So it causes the push and the pull of the magnet, right?</p><p>P2: "I guess."</p><p>[P1 looking at the AR magnetic field while explaining]</p><p>P1: "The magnetic field is inverted all the time. In order to have music, they change very quickly between forward and backward."</p></td><td><p>P2: "How do I use this?" [P2 trying to use a compass]</p><p>P1: "This is north, [P1 moving a compass to the other end] this is south. Let me try one second. The moment you come outside of the membrane, it's north. When it is near the membrane, it is south. So north to south, that's how the magnet direction goes." [Using compass then iconic hand gesture]</p><p>P2: "How about this part?" [P2 pointing to the coils]</p><p>P1: "This is where they produce electricity current, it starts from the magnetic membrane, this is where the south and north pole comes in."</p><p>P1: "When I push a forward current, it starts at south pole and ends at north pole. And the backward current is north to south."</p></td></tr></tbody></table>
Discussion

In this qualitative analysis, we found that participants using augmented reality exhibited a ‘tunnel vision’ effect, where they spent most of their time utilizing the AR system without leveraging other aids available to them (ex: compass, wall poster), in contrast to the NonAR participants who frequently used external aids. This is a possible explanation for why the AR participants showed higher learning gains in concepts requiring visualization, such as understanding magnetic fields, whereas participants without AR showed higher learning gains in other physical concepts such as relationship between magnetic field and movement. It is possible that AR participants focused strongly on the visual experience and ignored physical aids, while the NonAR focused on physical aids, which in turn increased their awareness of physical effects in the learning experience. Therefore, AR experiences might be a “double-edged sword”: the spatial-temporal contiguity affordance of AR (Radu, 2014) can help participants comprehend complex co-located visual representations, but also impede them from using physical tools and learning about non-visual aspects of the system. In other words, AR representations might be beneficial for learning visual concepts, but detrimental to acquiring kinesthetic knowledge – because users’ attention is so strongly drawn to the “holograms” provided by the AR system, leading them to neglect other sources of information.

Furthermore, our qualitative analysis suggests that AR representations may create a false impression of
understanding of the concepts conveyed by the system. In our study, AR participants stopped focusing on the task
much faster than NonAR participants; additionally, this effect was stronger for AR groups with low learning gains.
This observation is supported by the findings from our larger sample, where we found that AR groups showed
higher engagement and beliefs about self-efficacy than compared to NonAR groups (Figure 1), even if learning
gains were sometimes worse than NonAR groups (Radu & Schneider, 2019). Furthermore, it may be that NonAR
groups persisted with the activity for an extended period due to their need to create representations (such as by
drawing, using iconic gestures, or using external aids). This suggests that when participants lack easily accessible
information through tools such as AR, the extra effort caused them to engage longer with the content and
potentially think more critically. In contrast, AR may give people a false sense of confidence.

Finally, our analysis shows that teaching moments were shorter in the AR groups. When participants had AR visualizations available, communication was more efficient as participants could point or simply refer to an AR visual representation. In contrast, NonAR groups (which lacked the visual representations), had to spent time describing invisible phenomena or generate their own representations by using iconic gestures or drawings. They frequently ran out of time while having to make use of various aids and methods to produce their representation. This aligns with results from Ainsworth (2008) which suggest that to use representational learning tools effectively, learners must be able to first understand the function of the representation as well as how the subject relates to the representation. However, the shorter teaching time with deictic gesture may not necessarily be superior to spending longer time teaching with iconic gesture. Alibali et al (2012) discuss how different gestures serve different purposes for learning and teaching—deictic gesture connects cognition to the physical environment, while iconic gesture represents mental images, and these may correspond to different types of learning.

In future work, we are planning to further investigate why participants in AR spent less time on specific
tasks and how different conditions influenced collaborative interactions. To achieve that, we are considering to
shrink our video observation time frames from 30 sec. to 15 sec. as we may observe more detailed behavior in
shorter intervals. We also plan to revise our coding scheme (communication types) to be able to differentiate
degrees in communication (e.g., verbal vs. non-verbal when exploring). Additionally, we are working on
augmenting qualitative observations with other process metrics collected from the study (i.e., data from
electrodermal wristbands and motion sensors).

Conclusions

We qualitatively compared video observations of students learning with and without AR visualizations. We found that AR participants learned more about visual concepts but less about non-visual content, stopped exploring the system quicker than NonAR participants, used less aids in exploration and teaching, and spent less time in teaching their collaborators. Those findings suggest that while there might be opportunities to design AR applications in education, there are drawbacks associated with the use of this technology (e.g., the “tunnel vision” effect described above). In summary, the qualitative findings presented in this paper shed new lights on the affordances of AR technology and provides a critical analysis of the use of augmented reality for co-located collaborative learning activities.
# Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant No. 1748093. We would like to thank all our study participants, the anonymous reviewers, the Harvard Decision Science Lab, and all the dedicated research assistants who helped in the research data collection, and our colleague Tonya Bryant who completed the interrater reliability for this study.

# References

Ainsworth, S. (2008). The educational value of multiple-representations when learning complex scientific concepts. In Gilbert, J. (Ed.), Reiner, M. (Ed.), & Nakleh, M. (Ed.), *Visualization: Theory and practice in science education*. (pp.191-208) Dordrecht: Springer.

Alibali, M. W., & Nathan, M. J. (2012). Embodiment in Mathematics Teaching and Learning: Evidence From Learners' and Teachers' Gestures. *Journal of the Learning Sciences*, 21(2), 247–286.
https://doi.org/10.1080/10508406.2011.611446

Alibali, M. W., Young, A. G., Crooks, N. M., Yeo, A., Wolfgram, M. S., Ledesma, I. M., ... Knuth, E. J. (2013). Students learn more when their teacher has learned to gesture effectively. * Gesture*, 13(2), 210–233.
https://doi.org/10.1075/gest.13.2.05ali

Bujak, K. R., Radu, I., Catrambone, R., MacIntyre, B., Zheng, R., & Golubski, G. (2013). A psychological perspective on augmented reality in the mathematics classroom. *Computers and Education*, 68, 536–544.
https://doi.org/10.1016/j.compedu.2013.02.017

Cai, S., Chiang, F. K., Sun, Y., Lin, C., & Lee, J. J. (2017). Applications of augmented reality-based natural interactive learning in magnetic field instruction. *Interactive Learning Environments*, 25(6), 778–791.
https://doi.org/10.1080/10494820.2016.1181094

Enyedy, N., Danish, J. A., Delacruz, G., & Kumar, M. (2012). Learning physics through play in an augmented reality environment. *International Journal of Computer-Supported Collaborative Learning* (Vol. 7).
https://doi.org/10.1007/s11412-012-9150-3

Hill, M., & Sharma, M. D. (2015). Students' representational fluency at university: A cross-sectional measure of how multiple representations are used by physics students Using the representational fluency survey. *Eurasia Journal of Mathematics, Science and Technology Education*, 11(6), 1633–1655.
https://doi.org/10.12973/eurasia.2015.1427a

Kohl, P. B., & Finkelstein, N. D. (2007). Expert and novice use of multiple representations during physics problem solving. *AIP Conference Proceedings*, 951, 132–135. https://doi.org/10.1016/j.apm.2016.02.027

Radu, I. (2014). Augmented reality in education: A meta-review and cross-media analysis. Personal and Ubiquitous Computing, 18(6), 1533–1543. https://doi.org/10.1007/s00779-013-0747-y

Radu, I. & Schneider, B. (2019). What Can We Learn from Augmented Reality (AR)? Benefits and Drawbacks of AR for Inquiry-based Learning of Physics. In *2019 CHI Conference on Human Factors in Computing Systems Proceedings* (CHI 2019), May 4–9, 2019, Glagsow, Scotland, UK. ACM, New York, NY, USA. 12 pages. https://doi.org/10.1145/3290605.3300774

Roth, W.-M. (2001). Gestures: Their Role in Teaching and Learning. *Review of Educational Research*, 71(3), 365-392. https://doi.org/10.3102/00346543071003365

Suyatna, A., Anggraini, D., Agustina, D., & Widyastuti, D. (2017). The role of visual representation in physics learning: Dynamic versus static visualization. *Journal of Physics: Conference Series*, 909(1).
https://doi.org/10.1088/1742-6596/909/1/012048

Wu, H. K., Lee, S. W. Y., Chang, H. Y., & Liang, J. C. (2013). Current status, opportunities and challenges of augmented reality in education. *Computers and Education*, 62, 41–49.
https://doi.org/10.1016/j.compedu.2012.10.024

Zacharia, Z. C., & Olympiou, G. (2011). Physical versus virtual manipulative experimentation in physics learning. *Learning and Instruction*, 21(3), 317–331. https://doi.org/10.1016/j.learninstruc.2010.03.001
# Developing a Model of Collaborative Learning With Minecraft for Social Studies Classrooms Using Role-play Theory and Practice

Anders I. Mørch, University of Oslo, Anders.Morch@iped.uio.no

Louise Mifsud, Oslo Metropolitan University, Louise.Mifsud@oslomet.no

Siv Eie, Oslo Metropolitan University, Siv.Eie@oslomet.no

**Abstract:** Minecraft is a multi-user block-building game and 3D virtual world for creating visual structures. We report on two three-week collaborative teaching experiences in a teacher education program in social studies where we experimented with different ways of intertwining block building and role-playing in Minecraft. We created a model for integrating collaborative learning and virtual worlds in social studies classrooms, referred to as collaborative knowledge adaptation. The model consists of three levels: 1) introduction (teacher introducing topic for learning), 2) reconstruction (building in Minecraft and creating a script for role-play), and 3) transformation (acting out role-play and producing a video). Data were collected through interviews, an open-ended questionnaire, and product (video) analysis. We focus our analysis on generic skills interwoven with domain-specific skills and three levels of intersubjectivity toward a shared knowledge object.

## Introduction

Minecraft has been rated the most popular digital game among 9–14-year-old children in Norway (girls/boys 9–11 and in the top five for girls/boys 12–14), but teachers’ perception of Minecraft as a game and not a learning tool poses a challenge. This challenge is addressed by a new model for using Minecraft in teacher education, which manages to meet the students in their own arena while fulfilling the requirements of the curriculum. We took advantage of the popularity of the game, opportunities for collaboration and creativity enabled by block building and role-playing, and imposed constraints through knowledge-based activities before, after and in parallel with using Minecraft. Our case is a teacher education program in social studies. Research indicates that three-dimensional (3D) virtual worlds can be integrated into a teacher education program to provide preservice teachers with the experiences needed to apply teaching skills in real school settings (Muir, Allen, Rayner, & Cleland, 2013; Mørch, Hartley & Caruso, 2015).

Minecraft is a sandbox game, which means that users in the game interact by creating visual structures using basic blocks (modeled after 1m³ physical construction blocks and in analogy to a child playing in a sandbox) and communicate in parallel, mainly by chat but also by voice via other applications. Children are attracted to Minecraft for several reasons; they mine building blocks and craft tools, combat monsters, and collaborate with peers and allies in one of four different gameplay modes. We used the version of the game called Minecraft Education Edition in creative mode, with many of the gaming elements turned off to emphasize the elementary design acts of placing and breaking blocks in order to accomplish more advanced design tasks, such as reconstructing symbolic buildings of a society (e.g., governmental, industrial, cultural, religious, historical). Role-playing was the final stage of our learning activity and occurred inside or outside the buildings.

Engagement and motivation in learning are often listed as the strengths of using games in education (Gros, 2007; Pivec, 2007). Taking advantage of this potential could be the first step toward helping teachers to bridge the gap between students’ life worlds and schools’ curricular goals. Next, collaboration and problem solving are important behaviors associated with children’s digital game playing (Danby et al., 2018), which have been highlighted as important skills for the 21st century that we refer to as generic skills (Mørch, Eie, & Mifsud, 2018). Students combine generic and domain-specific skills when they learn in Minecraft. The aim of our research is to understand how student teachers organize and perceive the value of generic skills in specific subject areas, and how both types of skills can be practiced in the same lesson and in the same digital learning environment. The most prominent generic skills practiced in our study are information seeking, collaboration, and creativity. Other types of generic skills are communication, critical thinking, decision-making, reading, writing, computational skills, adaptability, personal development, and group effectiveness (Binkley et al., 2012).

Minecraft's technological and pedagogical affordances support a broad range of domain-specific skills, including mathematics, chemistry, computer science, history, and social studies (Nebel, Schneider, & Rey, 2016). The students can both visualize and concretize concepts. For example, the constructive play features in Minecraft inherited from wooden blocks, jigsaw puzzles, and Lego bricks can improve children's spatial abilities (Caldera et al., 1999). Furthermore, Minecraft includes tools that enable children to learn programming by introducing it as one of several techniques for solving practical problems in the game environment and
enhancing the game experience. Finally, research has shown that virtual worlds can motivate children to learn specific concepts through immersion and play, which means that they become involved in a subjective experience of concept understanding that leads to a feeling of participating in a comprehensive, realistic experience (Dede, 2009). The domain (subject area) we focus on in this study is social studies.

Minecraft and related virtual worlds and block-building games are suitable for a variety of teaching methods, including scenario-based learning, incremental and iterative activities, and role-playing (Prasolova-Førland et al., 2013; Westera, Nadolski, Hummel, & Wopereis, 2008). In role-play activities, participants are involved in “as-if” or simulated actions to approximate aspects of a real-life situation that is problematic, impractical, impossible, expensive, or risky to carry out in the real world (Yardley-Matwiejczuk, 1997). Students often blur the distinction between themselves and avatars when they immerse themselves in a 3D learning environment, which makes 3D virtual worlds potential sites for embodied and extended cognition (Pasfield-Neofitou, Huang & Grant, 2015). Educators have used role-play in many different application domains, including healthcare, therapy, organizational change, crisis management, military training, and education (Mørch et al., 2015; Prasolova-Førland et al., 2013; Westera et al., 2008). For example, Mørch et al. (2015) studied collaborative learning and role-play in special education in Second Life and found that combining abstract (conceptual) and concrete (hands-on) learning activities in multiple rounds and debriefing sessions were useful for grounding theoretical concepts in concrete experience. We address the combination of abstract and concrete learning activities in our study. We chose Minecraft instead of Second Life, as Minecraft is better adapted to schools.

Role-play and virtual worlds offer interesting opportunities for computer supported collaborative learning (CSCL) research (Jamaludin, Chee, & Ho, 2009). For example, embodied, enactive, extended, and embedded (4E) learning has been proposed as the theme of the CSCL 2019 conference, combining these elements in collaborative settings. Historical re-enactment is an educational or entertainment activity in which people follow a plan to recreate aspects of a historical event or period. In the study presented here this historical enactment is transferred to Minecraft. Our approach builds on ideas of collaborative knowledge construction (Roschelle & Teasley, 1995; Stahl, 2006), particularly knowledge construction achieved by intersubjectivity and meaning-making (Arnseth & Solheim, 2002; Baker et al., 1999; Suthers, 2006), which can be regarded as a dynamically changing context (Suthers, 2006) toward a shared object (Fugelli, Lahn, & Mørch, 2013), in our case a domain-specific knowledge object. Achieving and maintaining intersubjectivity requires a repertoire of conversational acts like “uptake” to reuse and modify prior contributions in conversation (Medina & Suthers, 2013), as well as models for researchers to analyze and design CSCL tasks at different levels of intersubjectivity (Baker et al., 1999) and guidelines for educators to organize their teaching designs (Matusov, 2000).

This study will therefore address the following research questions:

* How are generic and domain-specific skills practiced in the different learning activities?

* How are intersubjectivity and role-play relevant in learning social studies?

The rest of the paper is organized as follows. First, we describe our theoretical framework and the model we partially derived from the framework. Next, we present the methods we have used for data collection and analysis. Then, we present, analyze, and discuss our data and findings and show how the empirical findings informed the model in important ways.

## Theoretical framework: Role-play and prolepsis-driven intersubjectivity

The idea of role-play in learning is to foreground envisionment and exploration of alternative courses of action (Yardley-Matwiejczuk, 1997). For example, a child will envision different scenarios for how she will act in an upcoming school presentation by imagining her audience in front of her in the mirror or in her dreams. When this inner and private (non-observable) hypothetical situation is turned into an observable and organized activity, we have a role-play situation, and new opportunities for teaching and learning occur. Vygotsky (1978) suggested that play is a leading factor in human development, stating that “In play a child always behaves beyond his average age, above his daily behavior; in play it is though he were a head taller than himself” (p. 102). The imaginary situations that children engage in when they play are concrete while involving rules and roles, thus providing a means to iterate between concrete and abstract behavior, which we argue is essential for the effective use of role-play in education. Furthermore, Mead (1932) developed a philosophy of experience, with concepts such as emergence, temporality, and consciousness, in which the past and future are viewed through the lens of the present in interactions with others. Mead (1932) emphasized the novel character latent in both the present and the past (e.g., that historical understandings may involve anticipatory cues that may lead to transformations that in turn may lead to new insights).
Role-play conversation is different from naturally occurring dialog in that it is scripted and staged (Yardley-Matwiejczuk, 1997). CSCL researchers have adopted the term script for organizing collaborative learning activities (Cesareni, Cacciamani, & Fujita, 2016). We use script in a slightly different way to mean the written text of a roleplay in film production and connected with terms such as scenes, scripts, props, and prolepsis. We draw on the work of Rommetveit (1976) who developed a prolepsis-driven approach to intersubjectivity. According to Rommetveit, intersubjectivity is a temporarily sustained and partially shared social world that depends on access to historical information (common pre-understanding). Participants in conversation collaboratively construct knowledge by expanding a joint space of intersubjectivity, and a technique for this is to issue anticipatory cues (shared prolepsis), such as subtle utterances, incomplete sentences, or deliberately inserted cues. An incomplete utterance invites the listener to actively participate in the co-construction of an expanded intersubjective space to fill in the missing parts (Rommetveit, 1976; Fugelli, Lahn & Mørch, 2013). An example from our case is providing the role-players with an opportunity to understand a historical event in a contemporary context through an utterance, which points toward a future event outside the context that is shared by the participants, thus making the learning episode more meaningful to them.

The theoretical framework helped us construct a model of collaborative knowledge adaptation for the implementation of virtual worlds in social studies classrooms (Table 1). The columns in the table are connected through three crosscutting themes (introduction, reconstruction, transformation) and are either theoretically motivated (columns 2&3), based on empirical findings (columns 1&5), or informed by both theory and data (column 4).

<table><caption>Table 1: Collaborative knowledge adaptation model for integrating virtual worlds in classroom practice</caption><thead><tr><th></th><th>Activity</th><th>Rationale (metaphor of learning)</th><th>Temporal orientation (Mead, 1932)</th><th>Shared know-ledge object (intersubjectivity)</th><th>Type of skill (primary / secondary)</th></tr></thead><tbody><tr><td>Introduction</td><td>Teacher lecturing; students gathering information</td><td>Acquisition (Sfard, 1998)</td><td>Past: Retrospective views of topic to be learned</td><td>Vague: Teacher centered; students with different prior experiences</td><td>Domain-specific (primary)</td></tr><tr><td>Reconstruction</td><td>Collaborative design in Minecraft and roleplay scriptwriting</td><td>Participation (Sfard, 1998)</td><td>Present: Create new experiences together</td><td>Fragmented: Minecraft building vs. role-play script</td><td>Generic (primary) and Domain-specific (secondary)</td></tr><tr><td>Transformation</td><td>Role-playing and video recording</td><td>Knowledge creation (Paavola & Hakkarainen, 2005)</td><td>Emergent: Potential to connect an historical event with an upcoming event</td><td>Focused: Enacting an historical event while immersed in a virtual world learning activity (video recorded)</td><td>Domain-specific (primary) and Generic (secondary)</td></tr></tbody></table>

The model is the result of the first iteration of a design experiment. We used part of the model to organize the second iteration; and data we analyze in the next section provide contents for the last two columns (RQ1&2). The rationale for the model is three metaphors of learning (Sfard, 1998; Paavola & Hakkarainen, 2005), which suggest a sequence of increased complexity of collaborative knowledge construction. We call it collaborative knowledge adaptation, as the first level is domain-specific knowledge introduced by teachers.

## Methods

The study was organized as an experimental teaching activity between two educational institutions in Norway (A and B) over a three-week period over two semesters (2017 & 2018). Institution A provided educational resources (15 MA students majoring in education acted as tutors) and institution B organized the design experiment as part of a teacher education course (60 BA student teachers majoring in social studies).

Our interventionist approach follows design-based research principles (DBR) (Hoadley, 2002), in which a teacher and two researchers (the authors) collaborate to ‘engineer innovative educational environments and simultaneously conduct experimental studies of those innovations’ (Brown, 1992). DBR as a methodology allows us to focus on contextual aspects that become ‘relevant in the students’ interactions’ (Krange & Ludvigsen, 2009). This means we do not limit ourselves to the local situation of interaction but take the whole
into account in considering how the technical solution provides a new context for interaction. In the first experiment, the activities were building in Minecraft and role-playing with minimal teacher instructions, and in the second experiment the three-level model presented in Table 1 was used to inform our learning design.

The assignment in the first iteration (case 1) was to create a model of the Norwegian parliament and to engage in collaborative learning by role-playing a political decision-making process inside the building based on a script collaboratively created in small groups (Figure 1). The assignment of the second experiment (case 2) was to create a model of one of the historical buildings along the main river of Oslo and create a role-play of a historical event that is said to have happened in and around the building by its owners, tenants, and workers during industrialization, focusing on the working and living conditions of the 19th century (Figure 2).

<figure><img src="image_124.png" /></figure>

<figure><img src="image_125.png" /><figcaption>**Figure 1.** Two video snapshots of Minecraft parliament building (outside and inside, respectively) from case 1.</figcaption></figure>

In both cases, the students were bachelor students, co-located, and worked in groups of three to five individuals. We collected data through an open-ended questionnaire informed by our research questions (N1=37/60; N2=25/40), interviews (in the first case), observation notes, and 25 role-play films (3–5 minutes, both cases). We asked questions about experiences with Minecraft, what activities they found more or less interesting, how they organized collaborative work, the availability of support, would they use Minecraft for own school learning, etc. We thematically coded the answers, following an abductive approach that was partly data-driven and partly theory-driven by our model and research questions (Guest, 2012). We summarize our findings in the next section.

<figure><img src="image_126.png" /></figure>

<figure><img src="image_127.png" /><figcaption>**Figure 2.** Two video snapshots of Minecraft reconstruction of a textile factory building from case 2.</figcaption></figure>

We identified the following themes in our data: 1) time issues, 2) play vs. learning, 3) technical challenges, 4) generic skills interwoven with domain-specific skills, and 5) knowledge sharing (intersubjectivity). We focus on themes 4 and 5 in this paper.

## Results and analysis

A higher percentage of students reported a positive experience using Minecraft in case 2 compared to case 1. Thirteen students (52%) who had not played Minecraft previously indicated that they had satisfactory
knowledge of Minecraft after the three weeks (44% reported previous knowledge of Minecraft). Only one student (4%) claimed that s/he did not master the game after the three-week period (19% in case 1). Findings from interviews in case 1 (Mørch et al., 2018) showed that the student teachers perceive the use of Minecraft in their teaching as a possible threat to domain-specific (social studies) skills practice. However, they consider generic skills important and see the potential in Minecraft as a new way of reaching their students. Our experiences from case 1 made clear to us that the two types of skill (domain specific and generic) were not well integrated in the learning design.

We took the lessons learned from case 1 and started with a domain-specific context on which to base building in Minecraft. The teacher gave a lecture to introduce the social studies topic to the student teachers: the industrial breakthrough in Oslo in the mid-19th century. They were asked to search for information in two ways: through a given list of sources and encouraged to search for additional sources independently. The student teachers were also given an introduction to Minecraft and the generic skills emphasized in the curriculum (information seeking, communication, collaboration). This was followed by an excursion to the river and the industrial area.

## Thematic analysis

To address the two research questions, we give examples from our data showing how generic skills are interwoven with domain-specific skills and how intersubjectivity (knowledge sharing) is achieved during the course of the activity at three levels we have labeled introduction, reconstruction, and transformation.

### Introduction

The students worked in groups after the initial orientation by the teacher. They searched for information through books and the Internet to understand the task. One student elaborated on the process: "We collaborated on finding images of the building from all angles and had a discussion at the physical site" (Student 9). During the excursion to the river and the industrial area, the students measured the perimeter of the building and took photos as preparatory work for the reconstruction to follow. This part of the activity was mainly about acquiring domain-specific (social studies, historical) knowledge and using information seeking as the main generic skill (excluding the basic skills of reading and writing). One student said, "[It took us] around three to four hours to find information about the building, its history, and activity" (Student 18).

Intersubjectivity was not achieved at this level, which we call the "vague object." Based on our observations, the students, despite knowing of or about each other from previous classes and course assignments, were just beginning to work together on this task. The group efforts included gathering information from various sources (books, Internet texts, pictures of buildings) and preparing the group members for the next level of the activity.

### Reconstruction

The most prominent observed skills practiced at this level were collaboration, cooperation, and design. The students divided the work based on their interests and found that building and creating a role-play scenario were equally important. In their responses to the questionnaire, the students reported that two to three individuals from each group built in Minecraft while two to three wrote the script. As one student noted, "We collaborated well, and split the [task] into different parts. We all contributed toward both building and the role-play [script] but focused on different subtasks" (Student 9). The two subtasks (building and scriptwriting) were interdependent because Minecraft structures provided the scenery for the role-play script and the script defined actions and interactions in the buildings. The students liked this way of working, as it allowed them to be creative and focus on an area they had experience with or interest in while simultaneously contributing to the common goal. As one student put it, they had "the possibility to create something, be creative, and learn to collaborate in groups. The pupils who have problems at school can shine in such an arena" (Student 7). Both tasks made use of knowledge acquired from phase 1, but during role-play planning, the students had to find more detailed information about the historical event so that their re-enactment would be recognizable to viewers outside the group.

Our observation notes and questionnaire data show that two strands of shared knowledge were established at this level: one associated with building in Minecraft and the other scenario development. However, they were not yet integrated and we characterize intersubjectivity as a fragmented knowledge object.

### Transformation

In this phase, the students used what they had created in the previous phase through a roleplay (historical re-enactment) set in the given time epoch (mid-19th century work and life in Norway). The generic skills of
creativity, communication, and collaboration were the most prominent ones observed at this level. For example, creativity had the effect of generating enthusiasm not only within the group but also among the audience, who later watched the video in a plenum session, or as one student put it, "A more enjoyable variation in role-play where pupils can practice other skills such as filmmaking, etc. ... In addition [the pupils] can practice collaboration in film creation" (Student 7). The use of shared prolepsis as a film technique for expanding the space of intersubjectivity among a group of interlocutors and viewers was observed in a few of the videos and is illustrated by the following example:

**Turn**
1

**Actor**
Owner

**Utterance**
Yes, indeed, that is a very nice bridge. I will of course maintain it in a good way. Hmm, I think it would be a good idea to introduce a toll collection system, so I can finance maintenance work.

2 Attorney Toll?

3 Owner
Yes, it will be the first toll road in Norway, and with regards to this bridge, which you say is the main crossing between here and Akershus Fortress, I think it's a good idea.

<figure><img src="image_128.png" /><figcaption>Figure 3. Role-play scene/Minecraft video image.</figcaption></figure>

The extract is part of a dialog between two interlocutors (Owner & Attorney), the owner of the property on which a bridge is located and a legal representative of the government, regarding who should pay for maintenance of the bridge, as the Owner has newly purchased the property. During the exchange, the Owner suggests imposing a toll for crossing the bridge to finance maintenance costs, which comes as a surprise to the Attorney since it is one of a few passage points between west and east in Oslo. The discussion is interesting in that it shows an important event in history, but, by using the word "toll," the exchange not only takes up and elaborates previous information, but spawns a new line of discourse, connecting the conversation to another event, a current debate in Norway regarding who should finance a new road system around Oslo and other cities (end users vs. central authorities). The students are doing more than recreating history; they are pointing toward a future event they can relate to, which is why we have labeled this level of intersubjectivity *transformation*. It provides a way for students to personalize their collaborative learning efforts in creative ways. Five out of 25 videos included proleptic instances in their role-plays, although there were no instructions to do so. We discuss the implications below.

The two strands of knowledge (Minecraft building and scenario development) are integrated at this level in the three- to five-minute role-play, which are recorded on video and called a focused knowledge object.

## General discussion

We discuss our findings in terms of the research questions we raised in the introduction.

## How are generic and domain-specific skills practiced in the different learning activities?

One of our aims has been to understand how student teachers organize and perceive the value of generic skills in specific subject areas. We addressed this through a model of collaborative knowledge adaptation (Table 1). This model engages the students at different levels of collaborative learning. For example, developing the role-play required in-depth studies of historical sources, which put domain-specific skills practice in the foreground (level 1). Throughout the planning of the role-play, the students had to take a detour and pose "as-if" questions to envision working conditions, class struggles, labor strikes, or rivalries between factory owners. Generic skills (Binkley et al., 2012; Danby et al., 2018) thus became important for the elaboration of domain-specific skills practice: reconstructing historical events through collaborative design in Minecraft and roleplay script writing together (level 2: generic skills practice in foreground and domain-specific in background) and enacting the historical knowledge in creative ways in the role-plays with generic skills in the background (level 3). The data from the open-ended questionnaire indicate that all students found it useful to start with domain-specific skills
and that both the creation of buildings in Minecraft and role-plays (developing and enacting) were equally
important.

## How are intersubjectivity and role-play relevant in learning social studies?

Historical knowledge and intersubjectivity have several common features. For example, they are temporary processes that must be periodically updated to sustain, and they are never complete, nor fully shared. Moreover, there are multiple didactical strengths in creating and playing out a historical role-play at different temporal levels. First, the students get a “current” perspective rather than a retrospective. While role-playing, the students get involved as actors and are thus enabled to see history through multiple perspectives and to develop an understanding of historical presentations as constructs (Seixas & Morton, 2013). Second, through role-play the students are creating micro-stories about humans and their environments in the past. These micro-stories can help to develop the students’ historical empathy. The interplay between the micro- and macro-stories will enhance the historical overview and provide a deeper understanding of the past (Kvande & Naastad, 2013).

Matusov (2001) suggested three aspects of intersubjectivity when developing a guide for teaching design: 1) intersubjectivity as having something in common, 2) intersubjectivity as coordination of participants’ contributions, and 3) intersubjectivity as human agency. Our model is inspired in part by these aspects, but it differs in that we make use of role-play and building in a virtual world. Our model can also be compared with the five cycles of enactive role-play sessions proposed by Jamaludin, Chee, and Ho (2009). However, their domain is different from ours, debating moral issues as argumentative dialog rather than techniques for expansion of intersubjectivity. Medina and Suthers (2013) suggested “uptake” as an analytic approach for doing CCSL research. We have expanded the repertoire with “prolepsis,” as a way to project a conversation forward. Our approach builds on the framework of Fugelli, Lahn, and Mørch (2013), who conceptualize intersubjectivity as a knowledge object trajectory from incomplete to more complete. We propose three key events along this trajectory (vague, fragmented, focused), aligned with temporal orientations: past, present, and future (emergent).

## Conclusions, shortcomings, and directions for further work

Our prior experience shows that student teachers tend to value domain-specific skills over generic skills. Therefore, models are needed that emphasize and integrate both types of skills. Towards that end, we have developed the collaborative knowledge adaptation model for integrating virtual worlds in social studies classrooms using a design-based research approach with two iterations, where the results of the first iteration informed the second. The model is based on a combination of empirical findings and theories of role-play and intersubjectivity. The model puts focus on enactment, as in historical re-enactment to address the CSCL 2019 conference theme of 4E learning. We proposed a new conversational act to expand the space of intersubjectivity for small group conversation (shared prolepsis) and gave an example of a new way to use role-play in CSCL.

We plan to continue to develop and test our model in several iterations. We have identified the following shortcomings and directions for further work: (1) The whole activity is time consuming compared to conventional teaching practice; (2) The model is presented as linear, but levels 2 and 3 depend on the previous level, and information may have to be updated (e.g., improving the building based on the role-play); (3) Video recording of collaborative design in Minecraft and script development; (4) Different criteria for evaluating the video products; and (5) The model will undergo four new iterations in higher education and be tested in two experiments in 7th grade classrooms during the 2019-2020 school year, when our student teachers will use Minecraft with 12-year-old pupils in preservice training.

## References

Arnseth, H.C., & Solheim, I. (2002). Making sense of shared knowledge. Proc. CSCL 2002 (pp. 102-110). Boulder, CO: International Society of the Learning Sciences.

Baker, M., Hansen, T., Joiner, R., & Traum, D. (1999). The role of grounding in collaborative learning tasks. In P. Dillenbourg (Ed.), Collaborative learning: Cognitive and computational approaches (pp. 31-63). New York: Pergamon.

Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., Rumble, M. (2012). Defining twenty-first century skills. In P. Griffin, B. McGaw & E. Care (Eds.), Assessment and teaching of 21st century skills (pp. 17-66). Dordrecht, the Netherlands: Springer.

Brown, A. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. *The Journal of Learning Sciences*, 2(2), 141-178.

Caldera, Y.M., Culp, A., O'Brien, M., Truglio, R.T., Alvarez, M., Huston, A.C. (1999). Children's play preferences, construction-play with blocks, and visual-spatial skills: Are they related? *Int. J Behavioral Development* 23(4), 855-872.
Cesareni, D., Cacciamani, S. & Fujita, N. (2016). Effects of role taking in online writing and reading activities for knowledge building in a blended university course. *Int. J. of Computer-Supported Collaborative Learning*, 11(1), 9-39.

Danby, S., Evaldsson, A. C., Melander, H., & Aarsand, P. (2018). Situated collaboration and problem solving in young children's digital gameplay. *British Journal of Educational Technology, 49*(5), 959-972.

Dede, C. (2009). Immersive interfaces for engagement and learning. *Science*, 323(5910), 66-69.

Fugelli, P., Lahn, L., & Mørch, A.I. (2013). Shared prolepsis and intersubjectivity in open source development: Expansive grounding in distributed work. *Proc. CSCW 2013* (pp. 129-144). New York, NY: ACM.

Gros, B. (2007). Digital games in education. *Journal of Research on Technology in Education, 40*(1), 23-38.

Guest, G. (2012). *Applied thematic analysis*. Thousand Oaks, California: Sage.

Hoadley, C. (2002). Creating context: Design-based research in creating and understanding CSCL. *Proc. CSCL 2002* (pp. 453-462). Mahwah, NJ: Lawrence Erlbaum Associates.

Jamaludin, A., Chee, Y.S., & Ho, C.M.L. (2009). Fostering argumentative knowledge construction through enactive role-play in Second Life. *Computers & Education*, 53, 2, 317-329.

Krange, I., & Ludvigsen, S.R. (2009). The historical and situated nature of design experiments: Implications for data analysis. *Journal of Computer Assisted Learning*, 25(3), pp. 268-279.

Kvande, L., & Naastad, N. (2013). *Hva skal vi med historie? Historiedidaktikk i teori og praksis*. (Why do we need history? Teaching history in theory and practice). Oslo: Universitetsforlaget. (In Norwegian)

Matusov, E. (2001). Intersubjectivity as a way of informing teaching design for a community of learners' classroom. *Teaching and Teacher Education* 17, 4, 383-402.

Mead, G.H. (1932). *The philosophy of the present*. Edited by A.E. Murphy. Chicago and London: The Open Court Publishing Company.

Medina, R., & Suthers, D. D. (2013). Juxtaposing practice: Uptake as modal transposition. *Proc. CSCL 2013* (pp. 328-335). LuLu and Amazon: International Society of the Learning Sciences.

Mørch, A.I., Eie, S., & Mifsud, L. (2018). Tradeoffs in combining domain-specific and generic skills' practice in Minecraft in social studies in teacher education. *Proc. CoPDA 2018* (pp. 44-52), Castiglione della Pescaia, Italy, May 29, 2018 (published at http://ceur-ws.org/Vol-2101/).

Mørch, A.I., Hartley, M.D. & Caruso, V. (2015). Teaching interpersonal problem solving skills using roleplay in a 3D virtual world for special education: A Case study in Second Life. *Proc. ICALT 2015* (pp. 464-468). Washington, DC: IEEE Computer Society.

Pasfield-Neofitou, S., Huang, H., & Grant, S. (2015). Lost in Second Life: Virtual embodiment and language learning via multimodal communication. *Educational Technology Research and Development, 63*(5), 709-726.

Paavola, S., & Hakkarainen, K. (2005). The knowledge creation metaphor: An emergent epistemological approach to learning. *Science & Education, 14*(6), 535-557.

Pivec, M. (2007). Play and learn: Potentials of game-based learning. *British Journal of Educational Technology, 38*(3), 387-393.

Prasolova-Førland, E., Fominykh, M., Darisiro, R., & Mørch, A.I. (2013). Training cultural awareness in military operations in a virtual Afghan village: A methodology for scenario development. *Proceedings 46th Hawaii International Conference on System Sciences* (pp. 903-912). Washington, DC: IEEE Press.

Rommetveit, R. (1976). On the architecture of intersubjectivity. In R. Rommetveit & R. M. Blakar (Eds.), *Studies of language, thought and verbal communication* (pp. 147-161). New York: Academic Press.

Roschelle J., & Teasley S.D. (1995). The construction of shared knowledge in collaborative problem solving. In C. O'Malley (Ed.), CSCL NATO ASI Series, vol. 128. Berlin, Heidelberg: Springer.

Seixas, P. & Morton, T. (2012). *The big six historical thinking concepts*. Toronto, CA: Nelson Publishing.

Sfard, A. (1998). On two metaphors for learning and the dangers of choosing just one. *Educational Researcher*, 27(2), 4-13.

Stahl, G. (2006). *Group cognition: Computer support for building collaborative knowledge*. Cambridge, MA: MIT Press.

Suthers, D.D. (2006). Technology affordances for intersubjective meaning making: A research agenda for CSCL. *Int. J. of Computer-Supported Collaborative Learning*, 1(3), 315-337.

Vygotsky, L. S. (1978). *Mind in society: The development of higher psychological processes*. Edited by M. Cole. Cambridge, MA: Harvard University Press.

Westera, W., Nadolski, R., Hummel, H.G., & Wopereis, I.G. (2008). Serious games for higher education: A framework for reducing design complexity. *J. Comp. Assisted Learning, 24*, 420-432.

Yardley-Matwiejczuk, K.M. (1997). *Role play: Theory and practice*. London, UK: Sage Publications.
# "Can't Nobody Floss Like This!": Exploring Embodied Science
Learning in the Third Space

Chris Georgen, Indiana University, cgeorgen@indiana.edu

**Abstract:** I present an out-of-school science study situated in a performing arts center. The design brings together a mixed-reality science learning environment with ensemble dance in a series of embodied learning activities exploring the states of matter. Building on the theoretical framework of the third space, I analyze hybrid embodied movements that highlight the diverse role the body can play in learning science. Additionally, I report on pre- and post-measures indicating participants learned targeted science concepts and hybrid embodiments supported learning one particularly unintuitive concept. Finally, I make a case for the design of embodied learning activities that extend beyond considerations of analogous movements to include local and diverse movements that, if recognized as legitimate, expand possibilities for learning and doing science.

**Keywords:** science, performing arts, technology, elementary education

## Introduction

Educational research has increasingly recognized ways in which the body can play an important role in learning and engaging with disciplined practices (Alibali & Nathan, 2012; Azevedo & Mann, 2018; Goodwin, 2000). More recently, our natural tendency to use the body as a learning resource has been paired with motion-tracking technologies (e.g., mixed-reality environments) in the design of individual and collective instructional contexts that leverage embodiment (Danish et al., 2015; Lindgren & Johnson-Glenberg, 2013). Often these designs focus on some degree of congruency between body movements and the content to be learned, for example pushing arms forward to represent applying force to an object in a Newtonian simulation (Enyedy, Danish, & DeLiema, 2015). However, a focus on cleanly mapping bodily movement to content potentially ignores a diversity and potential hybridity of embodiment and how more informal ways of moving can support learning.

Notions of diversity and hybridity in learning build on the theoretical construct of the third space (Gutiérrez, 2008). The third space highlights the transformative power of hybridity—how informal and formal discourses (e.g., home and school) can become interconnected to productively challenge what counts as knowledge and how knowledge is produced. Hybridity in language practices has been well explored through the lens of the third space (Gutiérrez, Baquedano-López, & Tejeda, 1999; Gutiérrez, Rymes, & Larson, 1995). I aim to extend the exploration of hybridity to consider how embodiment can bring together informal and formal ways of moving and knowing (Ma, 2016). In this case, I view hybrid embodiments as those that merge dance-based ways of moving with science. Specifically, I explore how one fun and familiar embodied action (known as “flossing”) can represent a scientific phenomenon to act as a shared resource for participation and sense-making in collective science learning activities. To unpack the role of hybridity and embodiment in learning, I describe the design of a 10-lesson science activity sequence that integrated mixed-reality science learning with ensemble dance. I report on the results of the activity sequence and discuss how hybrid forms of embodiment supported learning by connecting dance movements throughout the intervention to performances on pre- and post-measures. I highlight how the unintuitive scientific concept that solid particles vibrate (i.e., they have high energy but are stationary) is explored that through a variety of ways of moving. I aim to answer the following research questions:

1. How do participants use familiar dance movements to represent and build upon scientific understandings of states of matter?

2. How do these movements support performance on pre- and post-measures?

## Background, design, and community

To illustrate the role of hybridity in embodied learning, I describe the design of a science learning research study that took place in a performing arts center. In contrast to science classrooms, all activity surrounding science learning focuses on local music and dance practices in which the adults and children participate. Hybridity, therefore, is made possible in the ways in which participants make sense of the conceptual practices of science in relation to the local, everyday ways of knowing and doing in dance. The intervention design explicitly connects two thematically and conceptually related ensemble learning activities: mixed-reality science models and dance (Figure 1). Within ensemble settings, individuals learn to take on a role within a system (e.g., a collective dance),
but much of their understanding and performing requires moving and thinking relatively to one another (Ma & Hall, 2018). In science, I explore ensemble learning through what I call collective embodied models. I build on the successful Science Through Technology Enhanced Play (STEP) mixed-reality learning environment that combines motion-tracking technology with a computational simulation (Danish et al., 2015; Danish et al., 2017; Enyedy et al., 2017). STEP allows participants to move across the room pretending to be water particles. The technology tracks their respective movements in real-time to generate a virtual representation of what state of matter those particles would collectively create. Through engaging with and analyzing the simulation, participants learn to coordinate their movement in certain ways to build, test, and refine models that predict and explain the rules behind the particulate nature of matter (Schwarz et al., 2009).

In dance, ensemble learning takes the form of learning and performing choreography. As part of a performing arts program, participants learned, rehearsed, and refined an ensemble dance based on the music and themes of the Disney production *Frozen*. The choreography and materials (e.g., props and costumes) integrate science concepts related to why and how particles move in solid, liquid, and gas. The dance was iteratively rehearsed and refined as participants learned more about the particulate nature of matter and as they practiced an increasingly more complex dance routine. Like the STEP environment, ensemble dance requires working collectively toward representing an idea, feeling, or experience. Since the dance was tied to communicating scientific understanding, it made visible important concepts and supported additional discussion and reflection about what was learned. Yet, the dance was not “scientific.” Though it conveyed aspects of science, the *Frozen* dance was held accountable to dance criteria (e.g., timing, rhythm, presentation, etc.).

<figure><img src="image_129.png" /><figcaption>Figure 1. Representing ice in STEP (left) and dance (right).</figcaption></figure>

# Methods

## Participants and data sources

Over the course of a summer performing arts program, two groups of participants (n=35, ages 6-8, 34 black or African-American and 1 white) participated in the activity sequence summarized in Table 1. Of these, 18 assented and received parent consent to participate in the study. Participants were divided into two groups. Each group participated in a 45-minute STEP activity, which was immediately followed by *Frozen* dance practice and rehearsal. Each activity had a science learning goal related to rules governing the states of matter, which progressed from macroscopic to microscopic. The STEP activities were facilitated by the research team with support from local staff. The “Frozen Dance” was choreographed a camp instructor. The dance was design to support the ideas from STEP, but maintained the ultimate goal of performing in front of loved ones at the end of camp during a large community event.

Throughout the planning process, the research team met regularly with the dance instructor to discuss the science content, locate overlaps and synergies, and build bridges between the two activity strands. During collaborative planning sessions, the research team and local instructor aimed to connect the science and dance in content, but also thematically through the narrative, characters, and settings from *Frozen*. Connections and similarities between the activities were explicitly communicated to students during activity introductions and debriefs, as well as emergently during key instructional moments. All activities from STEP and dance were video recorded provided consent.
<table><caption>Table 1: Integrated Activity Sequence</caption><thead><tr><td>Activity Sequence</td><td>STEP Activity</td><td>Dance Activity</td><td>Learning Goal</td></tr></thead><tbody><tr><td>Activity 1-2</td><td>Participants embodied energy sources and explored how energy and temperature drive observable changes in state and particle movement.</td><td>Participants learned the first phase of their dance focused on representing particle speed and distance during ice to liquid state changes.</td><td>Macro-level: Water exists in different states of matter: solid, liquid, and gas. Water can change states.</td></tr><tr><td>Activity 3-4</td><td>Participants took on roles as “energy wands” to explore how energy causes state changes in a snowman, Olaf from Frozen.</td><td>The dance introduced “magic wands” for participants to hold, accenting the dance and representing energy sources in the Frozen dance.</td><td>Marco-level: Energy and temperature produce state changes, i.e., giving energy causes melting and taking away energy causes freezing.</td></tr><tr><td>Activity 5-6</td><td>Participants build and test models to explore the relationship between particle distance and speed during state changes.</td><td>The dance introduced “Elsa gloves” for participants to wear, including discussion of how Elsa [the main character in Frozen] took away energy to make ice in things she touched. Participants learned the second phase of their dance representing liquid to gas state changes.</td><td>Micro-level: Develop rules for particle speed and distance in different states (e.g., identify different distances and speeds and associate them with states of matter).</td></tr><tr><td>Activity 7-8</td><td>Participants build and test models to explore the relationship between particle attraction during state changes.</td><td>The dance introduced sticks with streamers on the end for students to hold and represent the chaotic movement of particles in gas.</td><td>Micro-level: Particles in different states of matter are arranged differently. Particles in a solid are patterned, “stuck,” and vibrating. Particles in liquid have no regular arrangement and move closely together. Particles in gas move freely at high speeds.</td></tr><tr><td>Activity 9-10</td><td>Participants took on roles as water particles inside of containers in the mixed-reality simulation to explore particle behavior.</td><td>Rehearsal for end of camp showcase performance.</td><td>Micro-level to macro-level: Particles in solids always keep their patterned shape, Particles in liquids keep their shape but can move around, and particles in gas move freely and can escape if a container is opened.</td></tr><tr><td>Wrap-up</td><td>Post-measures conducted.</td><td>Rehearsal for end of camp showcase performance.</td><td>N/A</td></tr><tr><td>Wrap-up</td><td>Post-measures conducted.</td><td>Rehearsal for end of camp showcase performance.</td><td>N/A</td></tr></tbody></table>

## Methods: Interaction analysis to investigate hybridity

To understand the role hybrid embodied practices played in learning science content, I used interaction analysis to better understand ways in which participants used dance movements to navigate science concepts and vice versa (Jordan & Henderson, 1995). Due to the focus on the role of hybridity in embodied learning science, I focused video analysis on interactions during STEP activities. Nonetheless, I viewed and content logged the entire dance video corpus to better understand how embodied actions were taken up and used across these activity spaces. Within STEP, I identified constructions of the third space by observing embodiment as it related to the science (e.g., moving like water particles in the simulation) or dance (e.g., moving gracefully like water particles
in dance), focusing on tensions and synergies between these two types of movements. Within these sequences of embodied activity, I transcribed the video verbatim with key attention to formal, informal, and hybrid embodiments. I then categorized and compared these sequences to others to see if any patterns emerged. From there, I flagged moments where the science content was made visible in talk or embodied action to reference with the ways participants performed on pre- and post-measures.

## Methods: Science content knowledge measures

To measure content knowledge, all participants completed a multiple choice pre- and post-test. Additionally, individual pre- and post-interviews were conducted with 18 participants. Interview questions focused on supporting participants in articulating how particles (on the microscopic level) resulted in how matter behaved (on the macroscopic level). I conducted paired t-tests to determine whether students learned about the particulate nature of matter over the course of the activity sequence.

Based on the work of Paik, Kim, Cho, and Park (2004), I coded interviews for reasoning levels about state change mechanisms from circular reasoning (Level 0) or observable macroscopic features (e.g., melting) (Level 1) to unobservable microscopic relationships between kinetic energy and structures of particle movement (Level 4) (Table 2). Coding was not exclusive; any utterance may have more than one code or no codes at all, with the highest level of marking the final code. I transcribed each interview and segments of talk were labeled when participants articulated their reasoning about state changes. For each interview question, codes were applied to each participant utterance, including gestures and embodied actions. I conducted a paired t-test to examine if students' average reasoning level increased from their pre- to post-interview responses. Given my focus on hybridity in movement based on one specific concept, I report on the analysis of one interview question ("How do you think the particles in the ice are behaving?") that specifically addressed participants' experience of hybrid embodiment as it relates to science.

<table><caption>Table 2: State Change Coding Scheme</caption><thead><tr><th>Code</th><th>Example Response</th></tr></thead><tbody><tr><td>Level 0 - Circular Reasoning: any explanation for why behaviors happen without providing additional information, i.e., things happen because they are just "the way they are."</td><td>"Ice doesn't change shape because it is ice."</td></tr><tr><td>Level 1 - Macroscopic explanation (superficial properties): Any explanation that uses observable properties of matter.</td><td>"Ice doesn't change shape because it is hard."</td></tr><tr><td>Level 2 - Macroscopic explanation (energy/temperature): Any explanation that uses energy or temperature.</td><td>"Ice doesn't change shape because it is frozen."</td></tr><tr><td>Level 3 - Microscopic explanation (single factor): Any explanation that uses the behavior of particles in articulated reasoning, but only gives a single factor in their explanation (e.g., distance, speed, or energy).</td><td>"Ice doesn't change shape because the particles have high attraction."</td></tr><tr><td>Level 4 - Microscopic explanation (complex interactions): Any explanation that uses the behavior of particles in articulated reasoning with multiple connected factors in the explanation (e.g., distance and energy).</td><td>"Ice doesn't change shape because the particles have low energy so the bonds can be strong."</td></tr></tbody></table>
## Findings

Overall, the integration of STEP and dance afforded participants the opportunity to explore hybrid movements that bridged fun and familiar dance embodiments with scientific ones. This hybridity also appeared to support reasoning across contexts, from dance and science activities to content knowledge post-measures. Throughout the activity sequence, participants moved their bodies in a variety of ways to support reasoning about the science. A key pattern that emerged was the role of the body in reasoning about vibration. To represent ice particles, participants used a ranged of embodied intuitions from more normative scientific embodiments (e.g., standing still and shaking or jumping) to dance-based representations (e.g., doing the “wiggle worm” or “slow-mo” dance). In this paper, I focus on “flossing” as an illustrative example of how participants used dance to reason around movement and to accent their scientific understanding in new ways.

## Hybridity in embodied science learning

Because children’s everyday observations of matter tend to lead their assumptions about particle behavior, it can be challenging to support learning about the observable properties of matter as they relate to particulate behavior (Talanquer, 2009). For example, children assume that particles in ice are frozen and therefore do not have energy and cannot move. In fact, solid particles do have energy and vibrate in a rigid lattice structure. While the STEP simulation is useful for supporting understandings about particle distance and movement over space, it is a challenge for young learners to understand that solid particles move but remain stationary. A number of dance-based movements provided a useful resource in exploring ideas related to vibration. The findings of this paper are focused on a specific hybrid embodiment that is taken up by participants that supported robust performances on post-measures related the particles in ice. The following excerpts illustrate one form of hybrid movement that appears to have supported learning about how particles in a solid behave. To illustrate the role of hybrid embodiment in learning about the behavior of particles in ice, Excerpt 1 details a moment when a fun and familiar dance move “the floss” is taken up to represent vibration (Table 3).

<table><caption>Table 3: Excerpt 1</caption><tr><td>1</td><td>Facilitator:</td><td>Let's look on the screen. How are these particles moving?</td></tr><tr><td>2</td><td>Peyton:</td><td>Glitchin' out.</td></tr><tr><td>3</td><td>Facilitator:</td><td>Glitchin' out. Right. Another way we can say that is vibrating. So, solid particles stay in place and they vibrate ((shaking arms and shoulders)). Everybody stand up and vibrate like a solid particle would. ((Students all stand up and start to shake their bodies))</td></tr><tr><td>4</td><td>PJ:</td><td>Can't nobody floss like this ((moving arms in front and behind of their body)).</td></tr><tr><td>5</td><td>All:</td><td>((Other students around PJ begin to laugh and floss))</td></tr><tr><td>6</td><td>Facilitator:</td><td>Yeah, PJ! That is a good way to vibrate.</td></tr></table>

During the second week of the activity sequence, students took on roles as energy sources and observe how energy effects particles in different states. Excerpt 1 took place while participants were making observations about how solid particles are moving in the STEP simulation. One participant, Peyton described the particles as “glitchin’ out” (Line 2). This bid was taken up by the facilitator as an opportunity to introduce the concept of vibration and a more normative embodied representation of standing still and shaking arms and shoulders. This embodied movement is transformed by another participant, PJ, who responded by performing the well known dance “the floss” (Line 4). Flossing is a dance move performed by standing in place still while swinging arms, with hands in fists, from the back of the body to the front and moving hips in the opposite direction (Figure 2).

Flossing was made popular on social media and exploded due to its performance in music videos and video games. While flossing was not an explicit aspect of the final choreography in the *Frozen* dance, it was an “insider” movement in the context of this performing arts center, and children could be seen flossing throughout the day (e.g., during free time and lunch). Flossing also often occurred informally during STEP activities that promoted the exploratory use of the body. However, in this instance PJ spontaneously introduced flossing to move like a solid particle in science (i.e., standing in place and vibrating). While not scientifically normative, the move to represent particles with locally recognized dance blended the scientific notion of vibration with a well-known
dance. Moreover, it anchored students' embodiment in a way that legitimately maps to the verbal descriptions of vibrating particles as staying stationary but moving.

<figure><img src="image_130.png" /><figcaption>Figure 2. Choreographic movements of “the floss.”</figcaption></figure>

After this moment, flossing was taken up as a legitimate way to represent the trajectory of particles in ice during later activities. Beyond the overlaps in embodiment, the “unofficial” movement of “the floss” became central to how students understand particulate behavior. Hybridity both leveraged the prior knowledge and experiences of the participants as well as expanded the possible ways of knowing and moving related to science. For example, Excerpt 2 occurs toward the end the activity sequence. This activity focused on how particles in different states behave given the macroscopic conditions of an environment (e.g., a container with a lid) (Table 4). After engaging with the mixed-reality simulation as particles, the students re-represented their understanding from STEP with an embodied model. During the activity, the participants pretended the classroom was a container and the door was the lid. Beginning as ice particles, the facilitator gave the particles energy until they ran around the room as gas. Once the door was opened, all the participants ran out of the room like gas particles escaping a container.

<table><caption>Table 4: Excerpt 2</caption><tr><td>1</td><td>Facilitator:</td><td>If we were particles, what state of matter would we be in right now?</td></tr><tr><td>2</td><td>All:</td><td>Solid!</td></tr><tr><td>3</td><td>Facilitator:</td><td>Solid, right. Do solid particles stay totally still?</td></tr><tr><td>4</td><td>All:</td><td>No.</td></tr><tr><td>5</td><td>Lauren:</td><td>No, they vibrate! ((Participants start to shake arms and floss))</td></tr></table>

Excerpt 2 begins with a typical instructional move of opening instruction with a content question (Line 1). At once, the participants respond with the correct answer that particles in this formation would create a solid (Line 2; Figure 3). Further probing the participants, the instructor calls for more detailed information about particles, which is returned with another collective response. Immediately after, another participant, Lauren, calls out the answer, which is next embodied by Lauren and her peers with flossing. This moment demonstrates that the original embodied connection persisted over the course of multiple activities, as many of the participants took up flossing to represent vibration in solid particles.

<figure><img src="image_131.png" /><figcaption>Figure 3. Embodied re-representation of solid particles.</figcaption></figure>
## Learning gains

Paired t-test results of the 10-question multiple choice measure indicate that individual participants significantly improved their general understanding about particle behavior (t(17)=-7.071, p < 0.001) from pre-test (mean=3.11, SD=1.9) to post-test (mean=6.44, SD=0.85).

Given the specific focus on how hybrid embodied movements supported learning, the analysis of pre- and post-interviews focused on the first question of the interview protocol: How do you think the particles in ice are moving? Paired t-test results of the highest level code applied to student responses to this question indicate that individual participants significantly advanced their reasoning (t(17)=-8.416, p < 0.001) from pre-test responses (predominately circular reasoning or superficial observations) (mean=0.944, SD=0.53) to post-test responses (single and multifactor micro-level particle behaviors) (mean=2.61, SD=0.77). Generally, in pre-interviews participants focused on macro-level properties of ice to reason around how particles in ice behaved. For example, most students gave the expected response that particles in a solid were frozen together and could not move. In post-interviews, most participants (14/18) said that solid particles vibrate (see Table 5). This result is in marked contrast to previous classroom implementations of the STEP activity sequence focused on modeling and embodied play where only half the students in post-interviews could accurately describe or demonstrate how particles in ice behave (Danish et al., 2015). While these previous implementations supported robust learning gains, participants' beliefs about ice particles persisted despite experiences in the mixed-reality science simulation. Given that pre-interviews in this and similar STEP studies showed that the groups were comparable on their prior knowledge of science, it seems that the dance context contributed to participants' ability to engage with the conceptual knowledge and practices of science. That is, the dance context that supported hybrid embodied movements appears to have supported students in moving beyond descriptions of solid particles based on a surface-level characteristic.

<table><caption>Table 5. Interview Responses</caption><thead><tr><td>How do you think the particles in the ice are behaving?</td><td>Particles are frozen or don't move</td><td>Particles are vibrating</td><td>Circular reasoning</td></tr></thead><tbody><tr><td>Pre-interview</td><td>13</td><td>0</td><td>5</td></tr><tr><td>Post-interview</td><td>4</td><td>14</td><td>0</td></tr></tbody></table>

## Summary and significance

The integrated sequence of STEP and dance activities resulted in both learning gains and opportunities for participants to explore diverse forms of embodiment. By focusing on moments of hybridity, analysis traced specific movements from their spontaneous enactment to their use as a shared resource for sense-making in collective activities and individual assessments. In particular, flossing appears to have supported understanding an unintuitive concept that historically challenged young learners (i.e., that solid particles do in fact move). These outcomes suggest that scientific embodiments can incorporate local ways of knowing and moving and become a powerful resource in meaning-making. These results have implications for researchers and practitioners. First, work should further recognize embodiment as a diverse and hybrid practice. Second, scholars should continue to encourage the co-construction of hybrid learning spaces in a variety of modalities. Third, this study helps to articulate how future designs for embodied activity can benefit from moving beyond considerations of analogous movement (although that is still important) to include a focus on a diversity of movements that, if properly supported, can blend personally-meaningful movements with scientifically normative ones.

Overall, the goal of this paper is to highlight the need to understand the diversity of potential body movements in designing and implementing classroom-based and out-of-school science learning contexts. I argue that these results further support the central role the body plays in learning and exploring concepts. At the same time, it expands notions of embodied cognition from the ways a deliberately cued and technologically-enhanced body might directly interface with science content to include spontaneous new forms of movement that support learning (Lindgren, 2014). Moreover, this work offers new perspectives on hybridity and the third space—that is, how an embodied third space can produce and invite new forms of body-based participation for both individual participants and collectives. In sum, these outcomes can continue to advance computer-supported collaborative learning contexts, and in doing so expand possibilities for young learners' future scientific experiences and identification (Bell, Tzou, Bricker, & Baines, 2012).
## References

Alibali, M. W., & Nathan, M. J. (2012). Embodiment in mathematics teaching and learning: Evidence from learners' and teachers' gestures. *Journal of the learning sciences, 21*(2), 247-286.

Azevedo, F. S., & Mann, M. J. (2018). Seeing in the dark: Embodied cognition in amateur astronomy practice. *Journal of the Learning Sciences, 27*(1), 89-136.

Bell, P., Tzou, C., Bricker, L., & Baines, A. D. (2012). Learning in diversities of structures of social practice: Accounting for how, why and where people learn science. *Human Development, 55*(5-6), 269-284.

Danish, J. A., Enyedy, N., Saleh, A., Lee, C., & Andrade, A. (2015). Science Through Technology Enhanced Play: Designing to Support Reflection Through Play and Embodiment In Lindwall, O., Häkkinen, P., Koschman, T. Tchounikine, P. Ludvigsen, S. (Eds.) (2015). Exploring the Material Conditions of Learning: The Computer Supported Collaborative Learning (CSCL) Conference 2015, Volume 1. Gothenburg, Sweden: The International Society of the Learning Sciences.

Danish, J., Humburg, M., Saleh, A., Lee, C., Dahn, M., Kiefert, D., & Enyedy, N. (2017). A Socio-Cultural Framework for Embodied Cognition. Paper presented at the Jean Piaget Society, San Francisco, CA.

Enyedy, N., Danish, J., DeLiema, D., Saleh, A., Lee, C., Morris, N., & Illum, R. (2017). Social affordances of Mixed Reality Learning Environments: A case from the Science through Technology Enhanced Play project (STEP).

Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of pragmatics, 32(10), 1489-1522.

Gutiérrez, K. D. (2008). Developing a sociocritical literacy in the third space. *Reading research quarterly, 43*(2), 148-164.

Gutiérrez, K. D., Baquedano-López, P., & Tejeda, C. (1999). Rethinking diversity: Hybridity and hybrid language practices in the third space. *Mind, culture, and activity, 6*(4), 286-303.

Gutierrez, K., Rymes, B., & Larson, J. (1995). Script, counterscript, and underlife in the classroom: James Brown versus Brown v. Board of Education. *Harvard educational review, 65*(3), 445-472.

Lindgren, R., & Johnson-Glenberg, M. (2013). Emboldened by embodiment: Six precepts for research on embodied learning and mixed reality. *Educational Researcher, 42*(8), 445-452.

Lindgren, R. (2014). Getting into the cue: Embracing technology-facilitated body movements as a starting point for learning. In *Learning Technologies and the Body* (pp. 51-66). Routledge.

Jordan, B., & Henderson, A. (1995). Interaction analysis: Foundations and practice. *The journal of the learning sciences, 4*(1), 39-103.

Ma, J. Y. (2016). Designing disruptions for productive hybridity: The case of walking scale geometry. Journal of the Learning Sciences, 25(3), 335-371.

Ma, J. Y., & Hall, R. (2018). Learning a part together: ensemble learning and infrastructure in a competitive high school marching band. *Instructional Science, 1*-26.

Paik, S. H., Kim, H. N., Cho, B. K., & Park, J. W. (2004). K-8th grade Korean students' conceptions of ‘changes of state’ and ‘conditions for changes of state’. *International Journal of Science Education, 26*(2), 207-224

Schwarz, C. V., Reiser, B. J., Davis, E. A., Kenyon, L., Achér, A., Fortus, D., ... & Krajcik, J. (2009). Developing a learning progression for scientific modeling: Making scientific modeling accessible and meaningful for learners. *Journal of research in science teaching, 46*(6), 632-654.

Talanquer, V. (2009). On cognitive constraints and learning progressions: The case of “structure of matter”. *International Journal of Science Education, 31*(15), 2123-2136.
# Fostering Collective Knowledge Advancement Through Idea-Friend Maps in a Large Knowledge Building Community

Xueqi Feng, The University of Hong Kong, fengxueqi@hotmail.com
Jan van Aalst, The University of Hong Kong, vanaalst@hku.hk
Carol K.K. Chan, The University of Hong Kong, ckkchan@hku.hk
Yuqin Yang, Central China Normal University, yuqinyang0904@gmail.com

**Abstract:** This study designed and examined a knowledge-building environment that combined a learning analytics tool for creating "Idea-Friend Maps" with Knowledge Forum for Primary Five students. Students in the experimental class (n=53) used the Idea-Friend Map tool, whereas those in the comparison class (n=54) conducted the same knowledge-building activities without the use of analytics tools. Results showed that students in the experimental class showed better conceptual understanding, higher levels of social participation, and more advanced collective knowledge. Analyses were conducted on how students used analytics to support their knowledge building inquiry. These findings suggested that learning analytics can be used to promote knowledge building in large communities.

## Introduction

China is committed to education reform and is open to Western theories that have been developed in the learning sciences (Ryan, 2013). Knowledge building is a major pedagogical approach in the learning sciences community that focuses on theory building and knowledge creation (Scardamalia, 2002). The online platform Knowledge Forum® (KF) was designed to promote collective knowledge advancement by creating an online environment where students can pose problems, propose explanations, test ideas, and conduct sustained pursuit of inquiry to facilitate collective knowledge advancement. Generally, most studies on knowledge-building practice involve smaller-sized classes (15-30 students) (Chen, Scardamalia, & Bereiter, 2015; Oshima et al., 2004; van Aalst & Truong, 2011; Zhang, Scardamalia, Reeve, & Messina, 2009); however, classes in China generally contain 50-60 students, and very few studies involve classes with more than 50 students. Thus, little is known about how to implement knowledge building in such settings, nor about issues of scale in online discourse. How to facilitate large knowledge-building communities engaging in productive discourse is a challenging issue.

Generally, when classes larger than about 20 students, teachers have adopted group-based instruction (van Aalst, 2009) in knowledge building environments. Zhang et al. (2009) examined different social configurations—i.e., fixed groups (students work in fixed groups), interactive groups (interacting groups with cross-group activities), and opportunistic groups (new groups forming and disbanding based on emergent goals)—and found opportunistic groups yielded the best learning outcomes. However, whether opportunistic interaction adapts to large class environments still needs to be verified. Furthermore, when such a community promotes flexible participation and emergent inquiries on KF, it is difficult for students to understand the changing status of collective knowledge at the community level (Kimmerle, Cress, & Hesse, 2007). Therefore, the most urgent challenge is how to design an environment to help large class students identify, from the large volume of online discussion notes, the cutting edge of community knowledge, conduct sustainable problem solving, and structure what the community should investigate. However, the large number of online notes generated by large classes may provide opportunities for educational data mining and learning analytics, as it involves large datasets (Baker & Inventado, 2014).

There is increasing interest in learning analytics that emphasizes informing and empowering learners and instructors to improve their learning processes (Nistor & Hernández-Garcíac, 2018). However, much attention in this field involves evaluating what learners have done or predicting what they will do in the future (Siemens & Long, 2011); the area of how analytics are used as part of teaching and learning processes is relatively unexplored (Wise, Vytasek, Hausknecht, & Zhao, 2016). Knowledge Building Discourse Explorer (KBDex) is a novel learning analytic tool designed to investigate student-, keyword-, and discourse-based social networks on KF (Oshima, Oshima, & Matsuzawa, 2012). However, many current studies use KBDex merely as a tool for evaluating learning processes (Ma, Matsuzawa, Chen, & Scardamalia, 2016; Matsuzawa, Oshima, Oshima, & Sakai, 2012; Resendes, Scardamalia, Bereiter, Chen, & Halewood, 2015). Our program aims to design representations of the learning analytics tool KBDex, integrated with relevant social configurations for Grade Five students in large knowledge-building communities, and to examine the designed environment's impact on students' learning outcomes and collective knowledge advancement. Our research questions are: (1) Did students in the experimental class have better learning outcomes, greater KF participation, and deeper KB discourse? (2)
How did students in the experimental class engage in collective knowledge advancement, with the support of learning analytics?

# Methods

## Participants

This study was carried out in two Grade Five classes in a primary school in China, with 53 students in the experimental class and 54 in the comparison class. The first author was also the science teacher of the class. These students have had one-year experience in learning using knowledge building with this teacher before this study.

## Pedagogical design

The two classes investigated the topic of *Electricity* on KF (Figure 1) for 9 weeks. In Phase 1 (Weeks 1-3), students in both classes first created KF notes for inquiry, categorized their online notes into seven subtopics, formed learning groups, and chose subtopics according to interests. In Phase 2 (Weeks 4-6), students in both classes interacted with each other in different subtopics, did experiments, and conducted theory-building; however, students in the experimental class used the group-level Idea-friend Maps (IFM) (Figure 2, (a) and (b)), while those in the comparison group did not. In Phase 3 (Weeks 7-9), students in both classes worked in opportunistic groups (students formed/disbanded new groups based on emergent goals/common interests) to contribute to community views (included all the subtopics); however, students in the experimental class used the community-level IFM (Figure 2, (c) and (d)), while those in the comparison group did not.

<figure><img src="image_132.png" /><figcaption>Figure 1. The KF view of *Conductor and Insulator* (a); the experimental pedagogical design (b).</figcaption></figure>

<figure><img src="image_133.png" /><figcaption>Figure 2. The intervention of IFM: (a) Group 9's group-level IFM in Week 4; (b) Group 5's group-level IFM in Week 5; (c) community-level IFM in Week 8; (d) community-level IFM in Week 9.</figcaption></figure>
## The learning analytic tool (Idea-Friend Map)

The IFM intervention was the word network exported from KBDex. We designed the group-level IFM to help interactive activities in Phase 2, and the community-level IFM to promote opportunistic knowledge advancement in Phase 3. Figure 2 (a) and (b) show group-level IFM of different groups in different weeks; the red and yellow balls indicate words that had or had not already been discussed by the group, respectively; yellow balls near red balls indicate idea friends. For instance, students in Group 9 had already discussed those red balls words as “wire”, “circuit” and “current” until Week 4 (see Figure 2 (a)), while those yellow balls might be discussed by other groups. Words “dry” and “wood” from Figure 2(b) were “plastic”'s idea friends because they were very close to “plastic”. Students in different groups first identified idea friends, then went to other subtopics that included those ideas friends, enabling them to identify connections among different science concepts, make learning plans, interact with different groups, which would result in the advancement of collective knowledge. Correspondingly, Figure 2 (c) and (d) show community-level IFM during the final weeks of the intervention; balls with different colors (except yellow) indicate key topics identified in the community view, allowing students in opportunistic groups to synthesize different topics, summarize a topic, and create new knowledge with the support of community-level IFM. For example, the green ball with “Franklin” from Figure 2 (c) represented the problem of “Why did Benjamin Franklin not die from his lightning strike?” (see Table 2).

## Data analysis and findings

### RQ1: Did students in the experimental class have better learning outcomes, greater KF participation, and deeper KB discourse?

#### Changes in science learning outcomes across classes

We analyzed students' science domain tests to examine the effects of IFM on students' learning outcomes. There was no significant difference in the pre-test scores of the two classes (F (1, 101) = 0.81, p= .372). A two-way (Intervention × Time) ANOVA showed a significant main effect of Time (F (1, 199) = 145.53, p < .001, Partial eta² = .42). A significant Time × Intervention effect was obtained (F (1, 199) = 5.84, p = .017, Partial eta² = .03), indicating that students in the experimental class improved more on science learning outcomes, compared to students in the comparison class.

#### Class differences in KF contribution and interaction

We assessed quantitative data about the number of students' notes written, notes linked, notes read, scaffolds used, reference used, and views worked on KF, using the analytic toolkit (ATK) (Burtis, 1998). One-way ANOVA was performed to assess whether students' contributions to the online discourse were different among the classes. There were significantly higher values of notes read (F (1, 105) = 10.10, p< .01), views worked (F (1, 105) = 10.54, p< .01), and references used (F (1, 105) = 13.11, p< .001) in the experimental class. In contrast, there was no significant difference in the notes written, notes linked, and scaffolds used between the two classes (p > .1). These results suggest that the IFM intervention helped students to engage in different views, read more notes, and reference useful ideas.

#### Class differences in student online knowledge building discourse

We analyzed 588 KF notes from the experimental class and 528 from the comparison class to characterize students' contributions to knowledge-building discourse. We adopted a framework to code the notes in each interactive view and the community view. The coding schemes included three main categories—*questioning*, *theorizing* and *community*—and corresponding subcategories, and drew upon theoretical frameworks for progressive inquiry (Hakkarainen, 2003) and knowledge creation (Chuy, Zhang, Resendes, Scardamalia, & Bereiter, 2011; Fu, van Aalst, & Chan, 2016). Two raters independently coded about 50% of all the notes. The inter-rater reliability was .94 for *questioning*, .94 for *theorizing*, and .93 for *community* (Cohen's kappas).

As Table 1 shows, the experimental class created more *theorizing* and *community* notes, while the comparison group tended to raise questions. One-way ANOVA was adopted to further examine the differences between the two classes and found significantly higher values of *improving an explanation* (F (1, 14) = 5.88, p<.05), *bridging knowledge* (F (1, 14) = 8.30, p<.05), and *synthesis* (F (1, 14) = 5.15, p<.05) in the experimental class. We also observed significant higher values for *simple claim* (F (1, 14) = 5.88, p<.05) created in the comparison class. The differences in other codes were not significant between the two classes (p > .05). These results indicate that, when confronted with new problems, students in the experimental class tended to bridge knowledge through the website, engage in deeper theory-building processes, and synthesize useful ideas, while the comparison group tended to ask more fact-oriented questions and respond with simple claims, rather than seek relative notes or authoritative materials.
<table><caption>Table 1: Number of different categories of questioning, theorizing, and community in interactive views (experimental class/ comparison class)</caption><thead><tr><th rowspan="2">View</th><th colspan="3">Questioning</th><th colspan="4">Theorizing</th><th colspan="4">Community</th></tr><tr><th>Fact seeking</th><th>Explanatory seeking</th><th>Sustained inquiry</th><th>Simple claim</th><th>Proposing an explanation</th><th>Supporting an explanation</th><th>Improving an explanation</th><th>Shared regulation</th><th>Lending support</th><th>Bridging knowledge</th><th>Synthesis</th></tr></thead><tbody><tr><td>#1</td><td>7/7</td><td>15/10</td><td>8/0</td><td>7/14</td><td>29/40</td><td>16/9</td><td>10/1</td><td>8/0</td><td>0/1</td><td>15/1</td><td>2/0</td></tr><tr><td>#2</td><td>8/9</td><td>9/12</td><td>6/3</td><td>19/24</td><td>23/16</td><td>15/4</td><td>8/1</td><td>2/2</td><td>1/1</td><td>18/9</td><td>2/1</td></tr><tr><td>#3</td><td>0/5</td><td>3/7</td><td>1/2</td><td>0/16</td><td>6/11</td><td>4/1</td><td>2/1</td><td>2/1</td><td>0/3</td><td>10/10</td><td>0/0</td></tr><tr><td>#4</td><td>3/16</td><td>8/9</td><td>6/6</td><td>12/15</td><td>15/26</td><td>7/9</td><td>10/0</td><td>4/1</td><td>2/4</td><td>6/13</td><td>1/0</td></tr><tr><td>#5</td><td>8/13</td><td>7/7</td><td>3/3</td><td>16/34</td><td>7/7</td><td>3/2</td><td>1/0</td><td>2/2</td><td>1/1</td><td>12/5</td><td>2/1</td></tr><tr><td>#6</td><td>15/19</td><td>11/12</td><td>6/7</td><td>13/26</td><td>39/16</td><td>11/4</td><td>10/3</td><td>5/3</td><td>1/2</td><td>18/7</td><td>9/1</td></tr><tr><td>#7</td><td>2/10</td><td>3/14</td><td>4/1</td><td>10/18</td><td>8/16</td><td>0/4</td><td>1/0</td><td>0/2</td><td>4/3</td><td>12/10</td><td>2/1</td></tr><tr><td>#8</td><td>2/7</td><td>9/11</td><td>19/13</td><td>1/6</td><td>38/26</td><td>22/10</td><td>30/1</td><td>1/0</td><td>0/0</td><td>15/6</td><td>9/0</td></tr><tr><td>Total</td><td>45/86</td><td>65/82</td><td>53/35</td><td>78/153</td><td>165/158</td><td>78/43</td><td>72/7</td><td>24/11</td><td>9/15</td><td>106/61</td><td>27/4</td></tr><tr><td>Mean</td><td>5.63/10.75</td><td>8.13/10.25</td><td>6.63/4.38</td><td>9.75/19.13</td><td>20.63/19.75</td><td>9.75/5.38</td><td>9.00/0.88</td><td>3.00/1.38</td><td>1.13/1.88</td><td>13.25/7.63</td><td>3.38/0.5</td></tr><tr><td>SD</td><td>4.87/4.86</td><td>3.98/2.49</td><td>5.45/4.21</td><td>6.76/8.61</td><td>13.66/10.48</td><td>7.55/3.46</td><td>9.43/0.99</td><td>2.56/1.06</td><td>1.36/1.36</td><td>4.10/3.70</td><td>3.54/0.53</td></tr></tbody></table>

Notes: Views #1 to #7 mean subtopics that interactive groups worked on, while view #8 means the community view. #1=Electricity generation; #2=Franklin and electricity; #3=Lightning; #4=Static electricity; #5=Circuit; #6=Conductor and insulator; #7=Electricity and magnet

<table><caption>Table 2: Categories of problems in the community views for both classes</caption><thead><tr><th></th><th>Static electricity</th><th>Circuit</th><th>Conductivity</th><th>Franklin and electricity</th><th>Electricity and magnet</th></tr></thead><tbody><tr><th>Experimental class</th><td>#1 How does lightning form?<br>#2 What is the object can create static electricity?<br>#3 Why is static electricity formed from friction?<br>#4 Can I see current when static electricity is transmitted?</td><td>#1 Why does circuit need a closed loop?<br>#2 How does the body make electricity?<br>#3 Why does people might die from an electric shock?</td><td>#1 Why is mineral water/beverage/wet wood/metal conductive?</td><td>#1 Why did Benjamin Franklin not die from his lightning strike?<br>#2 What is the relationship between a lightning rod and insulator?</td><td>#1 Why is iron used for the core of an electromagnet?<br>#2 What is the relationship between electricity and magnet?</td></tr><tr><th>Control class</th><td>#1 How does lightning form?<br>#2 Why is static electricity formed from friction?<br>#3 How to remove static electricity?</td><td>#1 Why does circuit need a closed loop?<br>#2 How is electricity produced?<br>#3 Will a short circuit cause an explosion?</td><td>#1 Why can't the insulator be conductive?<br>#2 Why is metal/mineral conductive?</td><td>#1 Why did Benjamin Franklin not die from his lightning strike?</td><td>#1 What is the relationship between electricity and magnet?</td></tr></tbody></table>
## RQ2: How did students in the experimental class engage in collective knowledge advancement with the support of the learning analytics?

### Class differences in interactive views

We compared key interactive views reflecting how student advanced collective knowledge with the support of the group-level IFM. We first quantitative compared the Total Degree of Centrality (TDC) of the KBDex word network to assess the collective knowledge advancement (Oshima et al., 2012) of the interactive views *Conductor and Insulator* (Figure 3(a)) and *Circuit* (Figure 3(b)). We focused on these two views because they are the most fundamental parts when students discussed *Electricity*. A higher degree centrality means a denser social network. Figure 3 shows the word network’s increasing TDC over time, indicating how students collectively worked on key science ideas. In Phase 1, there were no significant differences in the collective knowledge advancement of the two classes in either view (p> .5), when they shared the same instruction. However, there were significantly higher TDC values in *Conductor and Insulator* (F (1, 4) = 101.86, p= .001) and *Circuit* (F (1, 4) = 112.16, p<.001) for the experimental class compared to the comparison class when students used IFM, in Phase 2. We also qualitatively analyzed students’ online notes, prompt sheets and experiment reports when they engaged in the interactive views, *Conductor and Insulator* and *Circuit*.

<figure><img src="image_134.png" /><figcaption>Figure 3. TDC of word network for the *Conductor and Insulator* view (a) and the *Circuit* view (b).</figcaption></figure>

### View: *Conductor and insulator*

We focused on the students from Group 9 in the experimental class who took responsibility for the *Conductor and Insulator* view to clarify how they engaged in collective knowledge advancement with the support of group-level IFM. After posting questions, proposing explanations, and searching authoritative information in the previous three weeks of Phase 1, Group 9 students had a general understanding of conductors and insulators, then they looked at their group IFM in Week 4, they found the keywords related to their topics, such as “conductivity”, occupied the right half of the entire IFM (Figure 2 (a)), while some keywords, such as “battery” and “voltage”, belonged to the *Circuit* group. Therefore, they speculated that, if they wanted to further explore their own topic, they must first understand the basic knowledge of “circuit”. Then they went to the *Circuit* view and managed to design an experiment to verify the elements of a circuit by lighting up a light bulb (Figure 4 (a)) by themselves. Later, they successfully revised their circuit experiment to explore whether water/wood was conductive (Figure 4 (b)). In addition, the students could also spontaneously review their thinking processes and try to explain their experimental phenomena in light of another’s perspective. Below is a summary note tracing how they explored the relationship between *Circuit* and their own topic, *Conductor and Insulator*.

[My idea] What is the relationship between circuit and conduction? In fact, the most basic thing about conduction is the circuit... To make an object conductive, we must first do a circuit experiment.

(■ “Group summary of experiment” by Group 9) We conducted two experiments: the first was to make the bulb light up. This experiment was a pilot for conductor and insulator... The second experiment, of water’s conductivity, was the experiment we originally wanted to explore... Because pure water has no impurities, most of it is not electrically conductive. In mineral water there are impurities, so the light bulb is illuminated when the wire is placed in it, so the two are electrically conductive.

(■ “Reply” by student c726) c726 also said that water can conduct electricity because there are cells inside it. You can understand in this way.
But the *Conductor and Insulator* group in the comparison class tended to ask fact-oriented questions, simply look up information, and copy/paste difficult conceptions from websites. The following is a cluster from the comparison group discussing the same topic (water's conductivity).

[I want to know] What object can be conductive?

[My idea] Water!

[I want to know] Why is water conductive?

[My idea] If all the impurities in the water are removed, the water resistivity will increase up to 18.2 megaohms x cm...

[My idea] Please provide information that we can understand.

<figure><img src="image_135.png" /><figcaption>**Figure 4.** Experiment reports of Group 9 ((a) and (b)) and Group 5 ((c) and (d))</figcaption></figure>

**View: Circuit**

Then we focused on Group 5, who worked on *Circuit* in the experimental class. After they acquired the basics of circuit and successfully connected circuit to light up a light bulb, they began to explore deeper knowledge of electricity from other views. Therefore, they first identified the keywords “wood” and “dry” as idea friends from their group-level IFM in Week 5 (Figure 2 (b)), went to the *Conductor and Insulator* view to get the information they wanted, made a group plan for exploration, and did experiments to explore why wet wood can conduct electricity while dry wood cannot (Figure 4, (c) and (d)). Finally, Group 5 and Group 9 students collaborated to explain their common experimental phenomena and created a theory-building note (below) to explain the conductivity of different waters and wood.

[Putting our theories together] 1. Mineral water/beverage is conductive because there are ions that can move freely; pure water can't conduct electricity because there are no ions that can move freely. 2. Why wet wood is conductive: wet wood contains water, water is conductive.

The *Circuit* topic is the foundation of *Electricity*, so Group 5 students began to explore other important topics from Week 5, at which point the TDC value began to show a slight downward trend, as shown in Figure 3 (b); however, the TDC value in the experiment class remained higher than that of the comparison class. Students in the comparison class were concerned with fact-seeking questions—e.g., “What is voltage?”; “What is the unit of voltage?”; “What contribution does Ampere make to electricity?”; and “Where can I see the circuit in my life?”

**Class differences in the opportunistic community view**

To characterize how community-level IFM helped students advance collective knowledge, we analyzed students' activities in the community view. First, we compared five categories of big problems identified in each class (Table 2). Results show that the experimental class identified more and deeper problems, while the comparison class tended to raise fact-seeking questions. This phenomenon is consistent with the notes coding results in RQ1.

In addition, students in both classes were invited to explore relationships, summarize, and create new knowledge according to the identified problems in Table 2. Students in the experimental class could conduct those activities smoothly with the support of community-level IFM (Figure 2, (c) and (d)), while students in the comparison class could only build on their original notes. We selected some community notes from the experimental class to show how those students synthesized, summarized, and created new knowledge through IFM.
**Theme 1: Synthesizing different topics**

Student c705 from Group 5 found the keywords of the two topics “Franklin” and “metal” on IFM (Figure 2 (c)) were particularly close, then they tried to explain “why Franklin did not die” from the perspective of “why metals can conduct electricity”.

[I want to discuss this topic] Why did Benjamin Franklin not die from his lightning strike?

[I want to explore relationships among those keywords] metal, Franklin and copper key

[My theory] Thunderbolt voltages range from hundreds of millions to billions of volts, but Franklin was not electrocuted. We can conclude that he did not directly touch the copper key. Even if using Leiden bottles and ribbons, people will die if they touch a few hundred volts (unless the voltage is very, very small).

**Theme 2: Summarizing a key theme**

Students c718, c726 and c711 from Group 6 found there were many keywords around “Franklin” in Figure 2 (c) and speculated those might be factors in protecting Franklin from the lightning strike. Then, they wrote down those keywords and selected relative information from the community.

[I want to discuss this topic] Why did Benjamin Franklin not die from his lightning strike?

[I want to summarize those keywords] Voltage, Leiden bottle, capacitor, wire, copper key, lightning rod, wind, ribbon, current, and metal.

[Putting our theories together] 1. Voltage (Low voltage by student c751); 2. Leiden bottle (Low voltage by student c722); 3. Iron wire (Iron wire by student c723); 4. Copper key (Synthesizing: Franklin and Metal by student c705); 5. Lightning rod (Lightening rod by student c713); 6. Ribbon (Ribbon by student c722); 7. Current (Current by student c723).

**Theme 3: Creating new knowledge**

Students c733 and c734 from Group 4 took responsibility for Static Electricity and found that the keywords “cloud” and “insulator” were very close in Figure 2 (c). Then they searched for information, identified related notes, and finally came up with new “knowledge”: “the cloud is an insulator.”

[I want to discuss this topic] What is the object can create static electricity?

[I want to create new knowledge on these keywords] cloud and friction.

[My theory] Our new knowledge is “the cloud is an insulator.” Evidence: 1. Electrostatics are all insulators (from Baidu); 2. Lightning is caused by frictions among clouds (from notes)

Students c708 from Group 5 and c752 from Group 2 provided more evidence to support the new knowledge—i.e., water evaporates into pure water vapour, which in turn forms clouds. Experiments proved pure water is an insulator, so the resulting cloud is also an insulator. Finally, we can see the new knowledge of “evaporation” in the next-week community-level IFM (Figure 2(d)). These data suggest that these young children, even in their simple writing, had strong knowledge-creation abilities, with the aid of IFM.

**Discussion**

This study investigated how learning analytics can be used to promote knowledge building in a Grade 5 science classroom. First, we compared differences in learning outcomes, KF participation, and depth of knowledge building discourse in both experimental and comparison classes. Students in the experimental class showed more improvement in science concept understanding, higher social dynamics of reading, referencing and interacting across different subtopics, and a deeper discourse on theory building and synthesizing. Analysis of students’ group plans, experiment reports, and summary notes revealed how students used both group- and community-level IFM to identify learning gaps, make plans, co-construct, collect evidence, synthesize key ideas, summarize, and create new knowledge. Even though the IFMs are not concurrent, representations of learning analytics could be helpful suggesting some practical use in classrooms. Furthermore, we integrated learning analytics with social configurations in large classes—i.e., interactive groups with group-level IFM, and opportunistic interactions with community-level IFM—provide a pedagogical design for the less explored field of knowledge building in large communities (Scardamalia & Bereiter, 1996).
## References

Baker, R. S., & Inventado, P. S. (2014). Educational Data Mining and Learning Analytics. In J. A. Larusson & B. White (Eds.), *Learning Analytics: From Research to Practice* (pp. 61-75). New York, NY: Springer New York.

Burtis, J. (1998). The analytic toolkit. *The Ontario Institute for Studies in Education, The University of Toronto: Knowledge Building Research Team*.

Chen, B., Scardamalia, M., & Bereiter, C. (2015). Advancing knowledge-building discourse through judgments of promising ideas. *International Journal of Computer-Supported Collaborative Learning, 10*(4), 345-366.

Chuy, M., Zhang, J., Resendes, M., Scardamalia, M., & Bereiter, C. (2011). *Does contributing to a knowledge building dialogue lead to individual advancement of knowledge*. Paper presented at the Connecting computer-supported collaborative learning to policy and practice: CSCL2011 conference proceedings.

Fu, E. L. F., van Aalst, J., & Chan, C. K. K. (2016). Toward a classification of discourse patterns in asynchronous online discussions. *International Journal of Computer-Supported Collaborative Learning, 11*(4), 441-478.

Hakkarainen, K. (2003). Progressive inquiry in a computer-supported biology class. *Journal of Research in Science Teaching, 40*(10), 1072-1088.

Kimmerle, J., Cress, U., & Hesse, F. W. (2007). An interactional perspective on group awareness: Alleviating the information-exchange dilemma (for everybody?). *International Journal of Human-Computer Studies, 65*(11), 899-910.

Ma, L., Matsuzawa, Y., Chen, B., & Scardamalia, M. (2016). *Community knowledge, collective responsibility: The emergence of rotating leadership in three knowledge building communities*. Paper presented at the Proceedings of the 12th International Conference of the Learning Sciences, International Society of the Learning Sciences, Singapore.

Matsuzawa, Y., Oshima, J., Oshima, R., & Sakai, S. (2012). Learners’ use of SNA-based discourse analysis as a self-assessment tool for collaboration. *International Journal of Organisational Design and Engineering, 2*(4), 362-379.

Nistor, N., & Hernández-García, Á. (2018). What types of data are used in learning analytics? An overview of six cases. *Computers in Human Behavior, 89*, 335-338.

Oshima, J., Oshima, R., & Matsuzawa, Y. (2012). Knowledge Building Discourse Explorer: a social network analysis application for knowledge building discourse. *Educational technology research and development, 60*(5), 903-921.

Oshima, J., Oshima, R., Murayama, I., Inagaki, S., Takenaka, M., Nakayama, H., & Yamaguchi, E. (2004). Design experiments in Japanese elementary science education with computer support for collaborative learning: hypothesis testing and collaborative construction. *International Journal of Science Education, 26*(10), 1199-1221.

Resendes, M., Scardamalia, M., Bereiter, C., Chen, B., & Halewood, C. (2015). Group-level formative feedback and metadiscourse. *International Journal of Computer-Supported Collaborative Learning, 10*(3), 309-336.

Ryan, J. (2013). *Education reform in China: Changing concepts, contexts and practices* (Vol. 69): Routledge.

Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. *Liberal education in a knowledge society, 97*, 67-98.

Scardamalia, M., & Bereiter, C. (1996). Engaging Students in a Knowledge Society. *Educational Leadership, 54*(3), 6-10.

Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. *EDUCAUSE review, 46*(5), 30.

van Aalst, J. (2009). Distinguishing knowledge-sharing, knowledge-construction, and knowledge-creation discourses. *International Journal of Computer-Supported Collaborative Learning, 4*(3), 259-287.

van Aalst, J., & Truong, M. S. (2011). Promoting knowledge creation discourse in an Asian primary five classroom: Results from an inquiry into life cycles. *International Journal of Science Education, 33*(4), 487-515.

Wise, A. F., Vytasek, J. M., Hausknecht, S., & Zhao, Y. (2016). Developing Learning Analytics Design Knowledge in the "Middle Space": The Student Tuning Model and Align Design Framework for Learning Analytics Use. *Online Learning, 20*(2), 155-182.

Zhang, J. W., Scardamalia, M., Reeve, R., & Messina, R. (2009). Designs for Collective Cognitive Responsibility in Knowledge-Building Communities. *Journal of the Learning Sciences, 18*(1), 7-44.
# Using and Perceiving Emoji in Design Peer Feedback

Na Sun, Pennsylvania State University, nzsl62@psu.edu

Elise Lavoué, University of Lyon, University Jean Moulin Lyon 3, iaelyon school of Management, CNRS,

LIRIS, elise.lavoue@univ-lyon3.fr

Chulakorn Aritajati, Pennsylvania State University, cya5092@psu.edu

Aurélien Tabard, Université Lyon 1 & CNRS, aurelien.tabard@univ-lyon1.fr

Mary Beth Rosson, Pennsylvania State University, mrosson@psu.edu

**Abstract:** Emoji are often used to express emotions or visually enrich text communications, but little is known about the impact of emoji in peer feedback. We conducted a qualitative study on emoji's roles in peer feedback in a small group collaborative design activity; the activity was conducted entirely online as part of a distance education class. Using content analysis on the feedback peers provided, and thematic analysis on transcribed interviews, we analyzed how emoji are used and perceived in four types of design feedback. Our analysis suggests that emoji increase peers' awareness of affective status, and are also used to mark minor cognitive feedback with a more personal flavor as part of document-based conversations. We found that emoji help to establish a general informality for peer feedback activities. We discuss the implications for the design of collaborative learning software that supports richer peer feedback.

## Introduction

As Värlander argues, “emotions should not be considered as hindering learning. Rather, it underlines the focal role of emotions in learning as being a natural part of it” (Värlander, 2008). Delivering and receiving feedback among peer learners is more than simply transferring information (Falchikov, 2013; Jacobs, 1974); each emotion-containing communication is influenced by students’ emotions at the time of expression (Race, 1996). With the prevalence of emoji and their prospects of expressing emotions, this paper examines the use and perception of emoji in peer feedback.

## The role of emoji in peer feedback work

Emoji are a very popular manner of expressing emotions in real-time communication, and have also become increasingly prevalent in a range of social computing technologies (e.g. Twitter, Instagram, SNSs). In an educational context, Zhang et al. showed that students annotated discussion posts with emoji or tags to signal their confusions or curiosity; these signals directed instructors' attention to places where students needed help or further clarification (Zhang, Igo, Facciotti, & Karger, 2017). In this case, the use of emoji rather than text-based hashtags offered a relatively easy mechanism for conveying emotions. Such findings suggest that the use of emoji among students may allow for transparent emotional exchanges and further direct instructors' cognitive efforts. This motivated us to investigate the role of emoji in peer feedback within small groups in online classes as a way to facilitate social interaction among distance learners.

One recent work leveraged crowdsourcing intelligence to express emotions in feedback with pictures versus text (Robb, Padilla, Kalkreuter, & Chantler, 2015). The designers who created the work being critiqued felt that both text and emotional images were more “threatening” than an abstract image; they also reported that the abstract images were inspiring. Nonetheless, text-based feedback is more likely to be used for giving conventional critiques: In a follow up study that examined how crowd workers felt when giving feedback using abstract images, emotion images and text, Robb et al. revealed that the type of feedback preferred was correlated with users’ cognitive styles, and that over half of the participants valued engagement over clarity (Robb, Padilla, Methven, Kalkreuter, & Chantler, 2017). Furthermore, some participants who provided feedback said that they would have liked to use emoji in addition to image-based emotion feedback. Although images have been found more expressive for emotions, some users find ambiguity to be a desirable advantage of giving emotion feedback because it is harder to put emotions in words than to use an image, and emoji are prone to ambiguity due to varying interpretations (Miller et al., 2016).

In addition to expressing emotions for the user, the use of ambiguous and playful expressions of emotions may invite and leave more space for communicative moves from the recipients, which could encourage more social interaction. In this regard, ambiguous and abstract indicators (e.g. emoji) seem consistent with Jakobson's model of communication (Jakobson, n.d.). In that model, the simple act of continuing a
conversation is valuable. Indeed, emoji have been used to manage conversation (Pohl, Domin, & Rohs, 2017), and even have been perceived as a more efficient and playful way of expression than text.

As a complement to text-based comments, emoji may provide additional emotional and situational information to the recipients. For example, a study that examined senders' intentions in US-based messaging reported that emoji are used as a social tool to engage the recipients in a conversation, to convey intended meaning, and to solicit certain reaction from the recipients, even when the senders intended nothing specific in the emoji themselves (Cramer, de Juan, & Tetreault, 2016). More generally, in a quantitative study of Twitter data (Pohl et al., 2017), emoji have been used as message decoration, to replace words, and to enrich text messages with respect to tones or meaning, in addition to serving the function of emotional annotation.

Despite the fact that emoji have become prevalent as part of a growing and universal language in text-based communication, researchers have not yet understood the role of emoji in remote collaborative learning tasks. Emoji seems to be a promising but under explored option for engaging online learners in conversations with additional situational information. We are especially interested in its potential in expressing emotional information in online learning environments where students often feel isolated and yearn for social connections with others (Sun, Rosson, & Carroll, 2018). However, emoji are subject to misinterpretations according to individual differences and varied rendering mechanisms across devices (Miller et al., 2016). Although it is possible that reacting to others' learning artefacts may stem from different interpretation and lead to differing emotional states, our study focuses on the impact of such emotions in interactive moves and interpersonal relationships situated in social learning activities. We now turn to our study of emoji as a mechanism for adding emotional content to peer-based critiques as part of a collaborative learning activity.

## Research setting

Situated in collaborative design activities among group members of an online bachelor degree program, we designed an online feedback tool called Emoviz to probe how peer feedback with emoji-entries affects the social interactions and relationships among online students.

## Selecting appropriate emoji for emotion feedback

Previous studies found that emoji can be perceived very differently within and across platforms, and thus lead to senders' and receivers' misunderstandings (Tigwell & Flatla, 2016). In addition, there are certain emoji with complex emotion interpretations that can cause high levels of sentiment misconstrual. There seems to be a tradeoff between communicative expressiveness and the consistency of emoji interpretation. To wit, the simpler an emoji is, the more uniform are people's interpretations (Miller et al., 2016). To make our studies more generalizable and comparable with existing studies, we adopted the Unicode standardized emoji, displaying them consistently as a Portable Network Graphics (png) file to minimize the cross-device differences.

We used existing emotion frameworks and emotion-related learning models (D'Mello, Lehman, Pekrun, & Graesser, 2014; Pekrun, Goetz, Titz, & Perry, 2002; Plutchik, 2003) to select eleven emotions that are relevant to academic peer feedback and design activity. These emotions include joy, surprise, praise, pride, love, anger, confusion, anxiety, disapproval and boredom. Considering the functions of emoji in social media research, we further included playfulness to extend the corpora of emotions in a broader sense (Pohl et al., 2017). As Table 1 shows, we chose three emoji as representatives for each emotion and proceeded to investigate the effectiveness of each one with a survey distributed to crowdworkers.

<table><caption>Table 1: Selecting representative emoji for relevant emotions for feedback</caption><tr><td>joy</td><td> 😊 😊 😊 n.s.</td><td>confusion</td><td> 😞 😞 😞 ? n.s.</td></tr><tr><td>surprise</td><td> 😃 😃 😃 n.s.</td><td>anxiety</td><td> 😞 😞 😞 n.s.</td></tr><tr><td>praise</td><td> 😊 😊 😊 n.s.</td><td>disapproval</td><td> 😞 😞 😞 n.s.</td></tr><tr><td>pride</td><td> 😊 😊 😊 n.s.</td><td>boredom</td><td> 😞 😞 😞 n.s.</td></tr><tr><td>love</td><td> 😊 😊 😊 n.s.</td><td>playfulness</td><td> 😊 😊 😊 n.s.</td></tr><tr><td>anger</td><td> 😞 😞 😞 n.s.</td><td></td><td></td></tr></table>

We obtained 72 responses from Amazon Turk where we asked MTurkers to rate “To what extent do you agree that the following emoji represent <selected emotion>,” on a Likert scale from 0 (Strongly disagree) to 100 (Strongly agree), excluding three that failed the quality test questions (we inserted two questions with
clear answers to select the qualified respondents for emoji interpretations for our survey). Our participants are all Americans (n = 69, 36 females), aged between 19 to 65, at an average age of 35.48. A majority of them (88.4%) used emoji in their online communications at least sometimes. Based on the General Linear Model to compare repeated measures, we found significantly different emoji for all emotions except for joy, surprise, pride, anxiety at the significance level of .001 with a Greenhouse-Geisser correction (see Table 1, highest emoji for each emotion is underscored). Regardless of the significance, the underlined emoji in Table 1 indicates which emoji candidates of each group were rated highest in the triplet. After this process, we selected the highest-crowd-rated emoji to represent each emotion, except for the confusion emotion; in that case we chose the higher-rated emoji that was also graphically consistent with the other set (i.e., 😂 instead of ?).

## Emoviz: An emoji-enabled online annotation tool

Emoviz is the online feedback tool depicted in Figure 1, offering customized regions for comments in text and/or emoji. Users can drag a rectangle over anywhere in a PDF file to leave comments (see Figure 1); the emoji included in the software were carefully selected through the aforesaid crowdsourcing study. They are displayed against a light-yellow rectangular background; any emoji can be appended to the end of a text comment. We adapted Emoviz for within-group feedback activities: students can upload documents in PDF format, give and receive peer feedback directly on the document. In addition, when comments are generated in the PDF document, a notification is sent to the PDF owner every 24 hours so that the submitters are aware of the feedback made to their document.

<figure><img src="image_136.png" /><figcaption>Figure 1. Emoviz User Interface for Giving and Receiving Feedback on PDF documents.</figcaption></figure>

## Method

### Procedure

Our study took place in an User-centered Design course offered by an online bachelor degree program of a Northeastern American university. Across the semester, the 50 students are tasked with a mix of individual assignments and group projects. In particular, our study built on two individual assignments that were embedded within a larger group project: one is the persona assignment, which is individual work but supposed to have other group members' feedback; another is the low fidelity prototype turned in PDF by individuals, aiming to yield a final prototype that the whole group agrees to proceed with (either building on top of one individual's submission or collaboratively generating a new prototype with all individuals' input). At the 8th week of the Fall 2017, we recruited students for the aforesaid two design activities via an opt-in invitation: all students of the
class were invited via Email link sent from Emoviz system, preceded by a call for participation sent by the course instructor on behalf of the research team and followed by a post-survey.

## Data collection and analysis

Throughout this deployment of Emoviz, we collected both quantitative and qualitative data and analyzed the emoji use and feedback activities among our participants to triangulate our findings (see Table 2). As for the quantitative data, we collected survey responses after the first design activity with regards to their demographic information, group cohesiveness, frequency of using emoji and emotional awareness of other members in general. To measure group cohesiveness, we employed six items from a well-established scale (Podsakoff, Niehoff, MacKenzie, & Williams, 1993). Only 7 students out of the entire class filled in this survey, and group cohesiveness range from 5.29 to 6.29 on a 7-point Likert scale.

<table><caption>Table 2: Triangulated data for Emoviz use and feedback activity</caption><thead><tr><td>Data source</td><td>#</td></tr></thead><tbody><tr><td>Comment</td><td>51</td></tr><tr><td>Survey</td><td>7</td></tr><tr><td>Interview</td><td>6</td></tr></tbody></table>

To gather qualitative data, we recorded Emoviz use with respect to the uploaded design documents and comments left during the collaborative process. The first author applied open coding and axial coding to analyze the feedback content at the unit of a comment (n = 51) by first categorizing feedback content learners have left in the document, and then scrutinizing the emoji's role in different types of feedback. Specifically, we began with an existing feedback model that distinguish cognitive feedback (e.g. summarization, specificity of problems, explanations) and affective feedback (e.g. praise, mitigation) (Nelson & Schunn, 2009); in parallel we adopted elements of grounded theory to label feedback with emergent codes (Strauss & Corbin, 1998).

In addition to classifying feedback content and form, we interviewed six Emoviz users remotely over Google Hangout after the design activities had been completed. We chose to use Google Hangout because this particular real-time communication tool is quite familiar to online students (Sun & Rosson, 2017). The interviews lasted 53 minutes on average, and were transcribed by the first author. Using MAXQDA12 software, the first author generated initial codes and reviewed and iterated the codes on using thematic analysis (Braun & Clarke, 2006). In particular, we found these themes: general emotional connections felt in the remote group; task reflection; other tools they have used for group activities in the class; usability issues of Emoviz; barriers they faced in using emoji or expressing emotions with other peers; and the effect of using and interpreting emoji. For the purposes of this paper, we only present the codes from interview related to the use of emoji and the perceived functions of using emoji in peer feedback activities.

## Participants

For our study, 13 students uploaded at least one design assignment for peer feedback activities from six groups in Emoviz. However, only 10 students representing three different groups received comments from other members, while the other three students' uploaded design work was unmarked. Among the participants, two were female, and half were Caucasian, followed by African American and Mixed Race. All but one student were working full-time while pursuing online degrees. Most students were in their fourth year of study (mean age = 33.14) and thus can be viewed as relatively experienced online learners.

## Results

Among the 13 documents that received comments from peers, we found 51 comments from eight different learners; 22 of these contained a total of 30 emoji. Based on an existing cognitive and affective feedback model (Nelson & Schunn, 2009) and open-coding of feedback that was not covered by that model, we identified four types of feedback towards the design work or described scenarios in Emoviz (note that these are not mutually exclusive, see table 3): 1) cognitive feedback (e.g. specific suggestions); 2) affective feedback (i.e. positive or negative); 3) conversational feedback (e.g. raising attention, showing idiosyncrasy, and responses to others' comments); 4) comparative feedback based on feedback givers' self-reflections. Accordingly, by a close examination on each comment and the context in which the comment is made, we identified different roles emoji play in these different types of feedback situations. In the following sections, we use the notation that