combines the data collection name and the participant ID in the following sections. For example, IP28 refers to participant 28 in the interview data, whereas DP28 refers to the same person's data in the content analysis.

<table><caption>Table 3: 4 Types of feedback and the use of emoji</caption><thead><tr><th>Type</th><th>Cognitive</th><th>Affective</th><th>Conversational</th><th>Comparative</th></tr></thead><tbody><tr><td># of comments</td><td>19</td><td>43</td><td>20</td><td>10</td></tr><tr><td># of emoji</td><td>0</td><td>28</td><td>18</td><td>7</td></tr><tr><td>Examples</td><td>DP13: "You probably want "the" instead of "to" here"</td><td>P13: "Clean looking recap screen option! 👏"<br>DP13: "Great size option! 😊❤️"</td><td>DP38: "She is rice crispy lol...nice detail though 😊❤️"<br>DP27: 😊❤️</td><td>DP33: "Well written short story. So much more interesting than the bio I wrote :)"</td></tr></tbody></table>

## Emoji were used only with non-negative feedback

emoji were more pervasively used in the comments that expressed favorable review of design elements; in contrast criticisms or suggestions for improvement were shared as text only. More specifically, we observed that all the cognitive feedback (e.g. suggestions for improvement or critiques) was articulated in a text-only manner. However, participants wrote 28 positive comments to recognize, compliment or make jokes about others' work, and the remaining 23 comments were either neutral or negative, for example being used to ask for clarification, point out errors and to provide specific suggestions for improvement.

26 out of the 30 emoji and emoticons (e.g. :-)) or :)), such as 👄, 😊, were used to praise or acknowledge others' good work, for example commenting on a clean layout or a specific design element. Although the numbers are too small to make a statistical comparison, there seemed to be a tendency to prefer emoji when commenting on graphical elements, such as persona photos or design sketches. For example, DP21 commented on a graphic photo at corresponds to DP28's persona description, "What a serious looking man. Doesn't look like he's actually smiling 😂," whereas DP28 responded with a 😂.

Only two comments leveraged emoji for negative emotions (i.e. 😞, which, instead of being a critique to the writing or design work, was a reaction to the story characters' unhealthy habits or behaviors. For example, DP40 commented "Whoa! 😂" in response to DP33's persona description "Bob ordered nine cheese burger". In general, 😞 were used most often (10 times), followed by 😊 (8 times), and 😊 (4 times); meanwhile, most of negative emoji were never used (e.g. 😞, 😊, 😂).

Our follow-up interviews revealed that students chose not to use emoji in negative comments because the emoji available for negative emotion (e.g., 😞), were perceived as too aggressive for criticism that they wished to express in a mild way. For example, IP27, a stay-at-home Mom with 5 kids, would never use a sleepy face "with people you don't truly know that well- you only know them through group experience." IP25, a senior male student, also felt it more appropriate to use positive emoji, thinking that "the thumbs down could be a bit aggressive, especially if you are trying to approach from a 'yes, and' perspective and not the 'yes, but' one."

Indeed, honest expressions of critiques were also associated with perceived closeness among collaborators: IP38 suggested that he feels comfortable joking with familiar teammates, those who he regarded as "friends" even without emotional tones, such as "lol," as opposed to "when you are saying a statement you want to make sure the receivers understand you are not being serious what you are saying". The unfamiliarity and lack of intimacy prevent students of a virtual group from giving negative comments more frankly.

## Signaling insignificant cognitive feedback with emoji markers

A closer scrutiny on the 22 comments that contain emoji revealed that, aside from the association between emoji and positivity, these comments were often informal, casual, and light-hearted – anything but serious 😂, and DP13's reaction to the drawings of four types of bread in a User Interface sketch: "Oprah 'I Love Bread' and so do I 😊. Such comments seemed to contain a hint by the comment giver that a comment is meant to be expressing emotions at the moment (e.g. cheerful, surprised, etc.) IP33 appreciated the expressiveness of emoji, commenting "without emoji it is just hard to convey what you are feeling about something - it is a very simple (way) of doing that." IP40 liked how emoji can articulate her real reaction in response to the described scenario
at the moment: Looking at the document she commented on, she said, “*That emoji face I put there was exactly the big sad face I had after I read it (giggling). It is like a 'Wow'. You know a jaw drop reaction.*” She liked how the facial expressions of emoji in the feedback represented her real reaction.

IP34 further noted that emoji of different facial expressions enabled more nuanced expression and capture of feedback givers' emotional statuses: (“*if you have somebody laugh with tears flowing from it- I tell you there is a person who is cracking up or the person is really amused compared with a mild amusement. Yeah, so emoji do make differences in those occasions.*” He further explained that the explicit emotional annotations with emoji denoted one's real social indication better than audio, “*if a person uses emoji, you precisely get that indication, but other than that, (like) audio chat without visuals, it is almost as limited as just text without any emoji.*” In this regard, emoji provide subtle social cues that enlighten the social interaction in an online environment where emotional statuses are hard to convey or perceive.

In addition to articulating the emotions right at the moment, emoji also helped to provoke an interesting conversation, by communicating an open posture of being engaged or sociable. For instance, 12 out of 20 conversational comments were decorated with emoji. IP34 attributed emoji to cues that remote collaborators “*can easily pick and follow,*” since use of emoji indicates “*when somebody is relaxed and engaged when we are having those conversation compared with someone (who) is not really engaged.*” Therefore, use of emoji gives students like IP34 a sense of “*if somebody is open to more communication.*” IP40 suggested that emotional elements evoked through humor also help online student get through course projects easier as they come and go from one course to another. For instance, after reading DP 27's description about an old grandma user “*her bones snap, crackle and pop with each movement!*” (Figure 1), DP38 joked about how the character is extremely vulnerable, pointing out “*she is rice crispy lol...nice detail though 😂*,” and another team member DP13 responded to the joking comment with a 😂. DP38 explained in the follow-up interview that he liked to leave the impression of himself being humorous, and that emoji or its alternative, such as “*lol*”, allowed him to express his personality more boldly. IP32 also found emoji help to create feelings of closeness to their peers who used them. As another example, DP31 joked about the looking of the persona picture along with the described scenario in text format when the persona was hesitating of quitting his job: “*Judging by the look of his picture, he plans to quit his job 😂*”

Whether emoji are used to express emotions or initiate an interesting conversation, comments that contain emoji should not be taken seriously (as opposed to how to improve the design work or to praise). Such non-serious signals are also perceived and accepted by feedback receivers - students interpreted the traditionally “negative emoji” in a positive way, appreciating the reaction from the readers. For example, IP33 regarded 👉, 👋 and 😂 altogether as positive feedback, articulating that he likes them. As he elaborated, “*it shows that she read it and she was not really disagreeing but empathizing with what I have written.*” In this way, emoji help to assure that designers' creative composition, such as illustrative depiction of a character or situation, in the design work has been understood in the intended way. IP27 also found it a relief to see others' funny comment on her carefully composed design persona, “*the fact they talked about how they enjoyed, the funny comments in return saying that they enjoyed it makes me think that 'okay, good'. It gave me a good feel as to whether or not what I was trying to say came across right, or what I said is alright to say.*” This use of emoji extended how people usually use emoji to complement a text and express their emotions (Zhou, Hentschel, & Kumar, 2017) with enriched semantics of feedback content.

## Self-Reflection and Conversational Feedback

Surprisingly, we found a number of feedback comments that referred to a feedback-giver's own work or situation. For example, DP33 wrote “*Well written short story. So much more interesting than the bio I wrote :)*” in DP40's submission as a compliment while comparing it with his own submission. Description in a persona also sometimes evokes a sense of self-reflection, for example, DP27 felt a desirable state in reflection on her own tight schedule when reading DP13's work, commenting that “*Nice job! I kinda envy Noah's availability. :-)*” In another example, after reading the isolated living atmosphere of a grandma described in a scenario, DP38 expressed his empathy with the written character by DP27 in a sense of humor with “*Thanks for the guilt trip 😂*.” It seems the reviewing experience is not only a critique session about a peer's work quality, but also sometimes a venue to share personal reflections with reference to personal work or experiences.

Meanwhile, we coded 20 comments as conversational feedback because of their dialogic nature, as if the feedback givers are talking with the feedback receivers in a social chit-chat: For instance, DP27 commented on the scenarios with an obvious conversational tone: “*This is so very true, I empathize with this user! For sure!*”. The emotional sentiments aroused by the design were depicted via emoji; Vice versa, as a feedback
receiver, IP27 found comments with emoji more impressive and fun to be remembered. In particular, she took
content related comment as indicators that her original meaning has been clarified enough to be understood by
the reader to the effect that is intended by the designer.

Discussion

In contrast to previous studies that looked at emoji in message-based communication or social media (Miller et al., 2016; Zhou et al., 2017), our study is the first to examine the role of emoji in a group learning activity. Our study confirms the speculation from previous studies on emoji' advantage of representing ambiguous emotion feedback content (Robb et al., 2017). We describe how learners utilized the toning-up effects of emoji to signal non-verbal cues and personality, such as sense of humor and facial expressions. Embedding emoji as part of feedback options allows for more casual and in-situ marks of one's own emotion, and thus enhances perceived sociability in CSCL environments (Kreijns, Kirschner, Jochems, & van Buuren, 2007).

We also found that emoji served to initiate and manage conversation as part of a learning activity, as documented in social media contexts (Cramer et al., 2016). In this sense, emoji seemed also to carry over the norm of being socializing with other users as in social media platforms. Emoji in the peer feedback scenarios are used by feedback givers for initiating informal and light-weighted conversations. They are interpreted by feedback receivers as a social posture of the other interlocutor being friendly and responsive to their creativity and personality. In addition, we also observed the role of emoji to in eliciting empathized self-assessment and generating feedback based on social comparison with one's own design work. Therefore, our study points to the design opportunities of leveraging emoji to add more nuanced peer feedback content, such as depiction of in-situ emotional responses, and to elicit personal interaction among learners.

Further, we propose emoji can enrich the markers to specify the anchor points during the learning
process beyond using emotional vs non-emotional markers (Lavoué, Molinari, Prié, & Khezami, 2015). With
the lightened tones of emoji alongside the text comments, feedback givers can mark the “insignificance” of the
feedback, enabling more nuanced layers of feedback types in addition to affective and cognitive feedback. In
particular, feedback givers’ use of emoji hints at the informal nature of the particular comment, and thus helps
feedback receivers’ distinguish casual content for entertainment, fun or other engaging gestures for social
purpose from serious comments, which aligned with the conventional critique formats (Robb et al., 2015).

Limitation and future work

Although we have found evidence of using emoji to support peer feedback interaction, we admit the potential
limitation introduced by our small sample size and the particular online class environment where group
members are relatively cohesive. Although we do not have enough data to investigate this point, the use of
emoji may also encourage more casual discussions which may hinder knowledge building dynamics. Besides,
our interviews asked participants to recall their emoji use and response after finishing the peer feedback task,
which may lead to memory lapse and inaccurate descriptions.

As discussed in the literature, the interpretation of emoji depends on platforms and situational factors.
For the purpose of the study, we controlled the number and design of emoji to learning related emotions. Future
research could expand to open-ended use of emoji and study their relationship to learners' perceived emotions.
More quantitative research could also examine the association between emoji use and feedback perceptions.

Conclusion

Our study presents the first attempt of involving emoji in the context of design feedback activity, and also
demonstrates ways of engaging distance students with emoji as feedback options in an asynchronous peer
learning activity. Our findings revealed how emoji are associated with different types of feedback, and indicated
the potential of emoji to transform a formal peer critique task into a more emotion-aware collaborative process.
We also provide implications of utilizing informal language cues, such as emoji, in designing learning systems
that consider and support socioemotional dimensions of online interaction.

Acknowledgement

Our research was conducted as part of the EmoViz project funded by the Région Auvergne-Rhône-Alpes. We
thank our student participants and Emily Wenk, the instructor who helped us recruit students.

References

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. *Qualitative Research in Psychology, 3*(2), 77–101.
Cramer, H., de Juan, P., & Tetreault, J. (2016). Sender-intended Functions of Emojis in US Messaging. *Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services*, 504–509. http://doi.org/10.1145/2935334.2935370

D'Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. *Learning and Instruction*, 29, 153–170. http://doi.org/10.1016/j. learninstruc.2012.05.003

Falchikov, N. (2013). *Improving assessment through student involvement: Practical solutions for aiding learning in higher and further education*. Routledge.

Jacobs, A. (1974). The use of feedback in groups. *The Group as Agent of Change*, 408–448.

Jakobson, R. (n.d.). I960.

Kreijns, K., Kirschner, P. a., Jochems, W., & van Buuren, H. (2007). Measuring perceived sociability of computer-supported collaborative learning environments. *Computers & Education*, 49(2), 176–192. http://doi.org/10.1016/j.compedu.2005.05.004

Lavoué, É., Molinari, G., Prié, Y., & Khezami, S. (2015). Reflection-in-action markers for reflection-on-action in computer-supported collaborative learning settings. *Computers and Education*, 88, 129–142. http://doi.org/10.1016/j.compedu.2015.05.001

Miller, H., Thebault-Spieker, J., Chang, S., Johnson, I., Terveen, L., & Hecht, B. (2016). “Blissfully happy” or “ready to fight”: Varying Interpretations of Emoji. *Association for the Advancement of Artificial Intelligence*, 259–268. http://doi.org/10.1089/cyber.2011.0179

Nelson, M. M., & Schunn, C. D. (2009). The nature of feedback: How different types of peer feedback affect writing performance. *Instructional Science*, 37(4), 375–401. http://doi.org/10.1007/s11251-008-9053-x

Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic Emotions in Students’ Self-Regulated Learning and Achievement: A Program of Qualitative and Quantitative Research. *Educational Psychologist*, 37(2), 91–106. http://doi.org/10.1207/S15326985EP3702

Plutchik, R. (2003). *Emotions and life: Perspectives from psychology, biology, and evolution*. American Psychological Association.

Podsakoff, P. M., Niehoff, B. P., MacKenzie, S. B., & Williams, M. L. (1993). Do substitutes for leadership really substitute for leadership? An empirical examination of Kerr and Jermier’ s situational leadership model. *Organizational Behavior and Human Decision Processes*, 54(1), 1–44.

Pohl, H., Domin, C., & Rohs, M. (2017). Beyond Just Text: Semantic Emoji Similarity Modeling to Support Expressive Communication ??? *ACM Transactions on Computer-Human Interaction*, 24(1), 1–42. http://doi.org/10.1145/3039685

Race, P. (1996). The art of assessing 2. *NEW ACADEMIC-BIRMINGHAM*-, 5, 3–6.

Robb, D. A., Padilla, S., Kalkreuter, B., & Chantler, M. J. (2015). Moodsource: Enabling Perceptual and Emotional Feedback from Crowds. *Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work &#38; Social Computing*, 21–24.

Robb, D. A., Padilla, S., Methven, T. S., Kalkreuter, B., & Chantler, M. J. (2017). Image-based Emotion Feedback. In *Proceedings of the 2017 Conference on Designing Interactive Systems - DIS '17* (pp. 451–463). http://doi.org/10.1145/3064663.3064665

Strauss, A., & Corbin, J. (1998). *Basics of qualitative research: Techniques and procedures for developing grounded theory*. Sage Publications, Inc.

Sun, N., & Rosson, M. B. (2017). Finding the Community in Online Education: It’s in the Instructors’ Eyes. Philadelphia, PA: International Society of the Learning Sciences.

Sun, N., Rosson, M. B., & Carroll, J. M. (2018). Where is Community Among Online Learners?: Identity, Efficacy and Personal Ties. In *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems* (p. 292). ACM.

Tigwell, G. W., & Flatla, D. R. (2016). “Oh that ’ s what you meant !”: Reducing Emoji Misunderstanding. *Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI '16)*, (Cmc), 859–866. http://doi.org/2957265.2961844

Värlander, S. (2008). The role of students’ emotions in formal feedback situations. *Teaching in Higher Education*, 13(2), 145–156. http://doi.org/10.1080/13562510801923195

Zhang, A. X., Igo, M., Facciotti, M., & Karger, D. (2017). Using Student Annotated Hashtags and Emojis to Collect Nuanced Affective States. *Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale - L@S '17*, 319–322. http://doi.org/10.1145/3051457.3054014

Zhou, R., Hentschel, J., & Kumar, N. (2017). Goodbye text, hello emoji: mobile communication on wechat in China. In *Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems* (pp. 748–759). ACM.
# On the Understanding of Students' Learning and Perceptions of Technology Integration in Low- and High-Embodied Group Learning

Marianna Ioannou, Cyprus University of Technology, marianna@cyprusinteractionlab.com

Yiannis Georgiou, Cyprus University of Technology, Research Center on Interactive Media, Smart Systems and Emerging Technologies (RISE), yiannis@cyprusinteractionlab.com

Andri Ioannou, Cyprus University of Technology Research Center on Interactive Media, Smart Systems and Emerging Technologies (RISE), andri@cyprusinteractionlab.com

Mina Johnson-Glenberg, Embodied Games, Arizona State University, minaj@embodied-games.com

**Abstract:** Embodied learning activities supported by motion-based technologies are becoming popular in various contexts and settings. However, little is yet known about technology integration for embodied learning in groups in authentic classroom settings, as existing studies have been mostly conducted in laboratory settings. In this work, we examine students' learning and perceptions of technology integration for group learning in a high-embodied, Kinect-based educational game (Condition1, n=24 students), in comparison with a low-embodied, desktop-based version of the same game (Condition2, n=18 students), in an authentic classroom setting. Data collection included questionnaires evaluating students' baseline, knowledge gains, perceptions of technology integration, and post-activity interviews. Findings showed higher learning gains and more positive perceptions of technology integration, in terms of opportunities for student negotiation, for the students in the low-embodied condition. Implications are discussed for supporting high-embodied learning activities for group learning in authentic classroom settings.

## Introduction and theoretical background

Embodied learning, as an application of the embodied cognition theory, constitutes a contemporary pedagogy of learning, which emphasizes the use of the body in the educational practice (Antle, 2013, 2009; Barsalou, 2010; Georgiou & Ioannou, in press). Embodied learning environments based on motion-based technologies appear to have gained ground during the last decade, providing new ways in which the embodied cognition theory can be enacted in the field of education (Kosmas, Ioannou, Retalis, 2018, 2017). In particular, the widespread population of affordable motion-based technologies and natural user interfaces (e.g., Wii, Xbox Kinect, Leap Motion) have nowadays opened the doors for the design of technology-enhanced embodied learning environments, responding to the need to highlight the aspects of motion and physicality as a crucial part of the learning process (Melcer & Isbister, 2016). As argued by Maliverni and Pares (2014), technology-enhanced embodied learning environments open up new possibilities due to their potential affordances for promoting psychomotor, cognitive and affective learning gains.

From a technology integration perspective, incorporating innovative motion-based technologies and natural user interfaces in the school classroom introduces new challenges to teaching and learning; therefore, their incorporation in mainstream education is at a very slow pace (Abrahamson & Sánchez-García, 2016). According to Karakostas, Palaigeorgiou, and Kompatsiaris (2017) existing research on embodied learning technologies has been fragmented and is driven largely by technological innovations. Also, it has taken place mostly in laboratory settings focusing on the participants' interactions with the embodied environments, therefore lacking a clear focus on investigating their efficacy in authentic educational contexts (Karakostas et al., 2017). Moreover, while a number of technology-enhanced embodied learning environments have received positive evaluation in high-controlled laboratory settings (e.g., Homer et al., 2014; Lindgren, Tscholl, Wang, & Johnson, 2016), the limited number of studies conducted in authentic school classrooms had not be as successful as initially expected in promoting students' learning gains, compared to low-embodied, desktop-based environments (e.g., Anderson & Wall, 2016; Hung, Lin, Fang & Chen, 2014). The later warrants further investigation towards shedding light on the efficacy of high-embodied vs. low-embodied learning environments in authentic classroom settings.

The study adopts Moos (1987)'s conceptual framework of technology integration grounding the experience in three dimensions: "Relationship", "Personal development" and "System maintenance and change". Moos (1987)'s framework was used by Wu, Chang, and Guo (2007) and Maor and Fraser (2005) to derive subscales for the measurement and evaluation of the experience of technology integration. In particular, in this work, we examine students' perceptions of technology integration aimed at a high-embodied, Kinect-based learning experience (Condition1, n=24 students), in comparison with a low-embodied, desktop-based learning experience (Condition2, n=18 students), in addition to the low-technology conditions (e.g., the use of cell phones instead of Kinect).
experience (Condition2, n=18 students) for leaning in groups in an authentic classroom setting. The study sought to answer the following research questions: (i) *Are there differences in student's learning gains and attitudes between the conditions?* (ii) *What are the main factors affecting student's perceptions of technology integration in the two conditions?*

# Methodology

## Participants

Participants were 42 students in 4th grade (aged 8-9 years old), who were enrolled in a public primary school in the Eastern Mediterranean. The students were randomly assigned to the two conditions. Group1 (Kinect-based gaming condition) had 24 children (12 boys, 50%) and Group2 (Desktop-based gaming condition) had 18 children (11 boys, 61%).

## Research design

This study followed an explanatory sequential design, composed of two sequential phases (Creswell, Clark & Vicci, 2011). In phase 1, we adopted a two-group quasi-experimental design for investigating students' learning gains and perceptions of technology integration in both conditions. In phase 2, we proceeded with qualitative data collection to deepen our understanding of the factors relating to students' perceptions of technology integration.

## The digital game

We employed the “Alien Health” digital game, which was designed to teach 4th-12th grades about nutrition and healthy food choices. The game is well-related to the school curriculum whilst findings from previous studies of “Alien Health” indicated its acceptability by the children and its affordances to improve content knowledge (Johnson-Glenberg & Hekler, 2013). Children’s mission in the game is to make the right nutritional choices for the alien to make him feel better as he in charge of stopping the collision of an asteroid with the Earth. During the gameplay, children are presented with combinations of food and are requested to make choices within predefined timeframes, considering a constellation of five nutrients per food (protein, fats, carbohydrates, fiber, and vitamins/minerals). The digital game became available in both a low-embodied (desktop-based) and in a high-embodied (Kinect-based) version.

## The interventions

Considering the research goals of this study, an 80-minute intervention was developed for each condition. Children in the low-embodied (desktop) condition were divided in dyads and used the desktop-based version of the digital game. In this version children used the mouse and the keyboard for making a choice and feeding the alien (see Figure 1). Children in the high-embodied (Kinect-based) condition were divided in groups of four (the limited classroom space allowed only six Kinect work-stations of the game) and used the Kinect-based version of the game. In this case, the game was projected on a big screen and there was touchless interaction via the Kinect camera which can identify children’s arm/hand movement hovering over a single food item and moving it into the Alien’s mouth (see Figure 2). In both conditions the game was contextualized in a collaborative educational activity. In particular, the children took turns in playing (game affords only single player mode); the other child(ren) of the group was/were asked to provide feedback to the player, discuss the selections made, and record their food choices on a structured worksheet.

<figure><img src="image_137.png" /><figcaption>Figure 1. A group of students in the low-embodied condition.</figcaption></figure>

<figure><img src="image_138.png" /><figcaption>Figure 2. Two group of students in the high-embodied condition.</figcaption></figure>

## Data collection and analysis

Data collection included questionnaires evaluating students' baseline, knowledge gains, perceptions of technology integration, and post-activity interviews with eight students from Condition1 (33.3%) and eight students from Condition2 (44.4%).
## Baseline data

We aimed at establishing the equivalency of the two conditions in terms of computer and gaming attitudes. We used the Computer Attitude Measure for Young Students questionnaire (CAMYS, Teo & Noyes, 2008), which is composed of 12 items on a five-point Likert scale and has a documented reliability alpha coefficient of .85. Gaming attitudes were measured using an 11-item Likert scale (Cronbach's alpha=0.73) validated in the study of Bressler and Bodzin (2013). Differences between the two conditions were examined using the Mann-Whitney U test, given the small sample size of participants in each condition and the lack of normal distribution in the data.

## Knowledge test

A knowledge test was administered in pre-post format. The test was developed by Johnson-Glenberg and Hekler (2013) for evaluating students' learning gains in the Alien Health game.

<table><caption>Table 1: Questionnaire Dimensions, Subscale Details and Individual Items</caption><tbody><tr><td rowspan="10">DIMENSION 1:<br/>Relationships</td><td colspan="2">Student Negotiation (SN): a 5-item subscale assessing the extent to which students have opportunities to discuss their questions and their solutions to questions (adapted from Maor & Fraser, 2005)</td></tr><tr><td>SN-1</td><td>I get the chance to talk to other students</td></tr><tr><td>SN-2</td><td>I discuss with other students how to conduct investigations</td></tr><tr><td>SN-3</td><td>I ask other students to explain their ideas</td></tr><tr><td>SN-4</td><td>Other students ask me to explain my ideas</td></tr><tr><td>SN-5</td><td>Other students discuss their ideas with me</td></tr><tr><td colspan="2">Student Cohesiveness (SC): a 6-item subscale assessing the extent to which students are supportive to each other (adapted from Wu, Chang & Guo, 2007)</td></tr><tr><td>SC-1</td><td>Students are friendly to each other</td></tr><tr><td>SC-2</td><td>Students are willing to help each other</td></tr><tr><td>SC-3</td><td>It is easy to find members for grouping</td></tr><tr><td>SC-4</td><td>Students share information with each other</td></tr><tr><td>SC-5</td><td>Students have opportunities to discuss questions with classmates</td></tr><tr><td>SC-6</td><td>Group members complete assignments together in class</td></tr><tr><td rowspan="10">DIMENSION 2:<br/>Personal Development</td><td colspan="2">Competition & Efficacy (CE): a 6-item subscale assessing the extent to which students are motivated and confident to compete each other (adapted from Wu, Chang & Guo, 2007)</td></tr><tr><td>CE-1</td><td>Students care about their own performance</td></tr><tr><td>CE-2</td><td>Students work hard to outperform others</td></tr><tr><td>CE-3</td><td>Classmates' performances push students to be more diligent</td></tr><tr><td>CE-4</td><td>Students set up study goals on their own</td></tr><tr><td>CE-5</td><td>Comparisons among groups occur</td></tr><tr><td>CE-6</td><td>Students are confident of learning this subject well</td></tr><tr><td colspan="2">Reflective Thinking: a 5-item subscale assessing the extent to which students have opportunities to discuss their questions and their solutions to questions (adapted from Maor & Fraser, 2005)</td></tr><tr><td>RT-1</td><td>I get to think deeply about how I learn</td></tr><tr><td>RT-2</td><td>I get to think deeply about my own ideas</td></tr><tr><td>RT-3</td><td>I get to think deeply about new ideas</td></tr><tr><td>RT-4</td><td>I get to think deeply how to become a better learner</td></tr><tr><td>RT-5</td><td>I get to think deeply about my own understandings</td></tr><tr><td rowspan="5">DIMENSION 3:<br/>System<br/>Maintenance and<br/>Change</td><td colspan="2">Complexity: a 5-item subscale assessing the extent to which the program is complex and represents data in a variety of ways (adapted from Maor & Fraser, 2005)</td></tr><tr><td>C1</td><td>It has an interesting screen design</td></tr><tr><td>C2</td><td>It is easy to navigate</td></tr><tr><td>C3</td><td>It is fun to use</td></tr><tr><td>C4</td><td>It is easy to use</td></tr><tr><td>C5</td><td>It takes only a short time to learn how to use</td></tr></tbody></table>

## Technology integration survey

Students' perceptions of the technology integration were evaluated at the end of the experience. The questionnaire was composed of five subscales guided by Moos (1987)'s conceptual framework of technology integration, later used by Wu, Chang and Guo (2007) and Maor and Fraser (2005) to derive subscales for its three dimensions: "Relationship", "Personal development" and "System maintenance and change" as in Table 1.
## Post-activity interviews

Eight students from each condition participated in an approximately 15-minute semi-structured individual interview, which took place right after the intervention. The students were asked to talk about their learning experience with Alien Health, as well as their use and perceptions of the utilized technology. Driven by Moos (1987)'s conceptual framework of technology integration and its three dimensions, the students were particularly probed to discuss the factors affecting their experience in terms of: (a) Personal development (e.g., What were the main factors that help you learn during your participation in this digital game?), (b) Relationships with others (e.g., How was the collaboration among team members structured around the digital game employed?), and (c) Technology use (e.g., Did you encounter any problems while using the digital game? How those problems affected you?). All interviews were transcribed and coded within the three dimensions of our conceptual framework.

## Findings

### Setting the baseline

A Mann-Whitney U test was used to identify any potential differences between groups in student' attitudes towards computers and digital games (Table 2). Results showed that there were no statistical differences in the student's gaming attitudes ($U(40)=198.5, z=-.45, p>.05$) and attitudes towards computers ($U(40)=183, z=-.84, p>.05$) between the groups.

<table><caption>Table 2: Baseline assessment of students' gaming attitudes and attitudes towards computers</caption><thead><tr><th rowspan="2"></th><th colspan="2">Condition 1<br>Kinect-based game</th><th colspan="2">Condition 2<br>Desktop-based game</th><th rowspan="2">Z</th></tr><tr><th>Mean</th><th>SD</th><th>Mean</th><th>SD</th></tr></thead><tbody><tr><td>Gaming attitudes</td><td>3.38</td><td>0.65</td><td>3.28</td><td>0.52</td><td>-0.45</td></tr><tr><td>Computers attitudes</td><td>3.93</td><td>0.70</td><td>3.69</td><td>0.84</td><td>-0.84</td></tr></tbody></table>

Note. *p<.05, **p<.01. ***p<.001

### Knowledge gains

A Mann-Whitney U test was used to identify any potential differences between groups in student' pre- and post-test scores (Table 3). Results showed that there were no statistical differences in the student's pre-test scores ($U(40)=212, z=-.11, p>.05$). However, focusing on the post-test scores, students in Condition 2, who employed the desktop-based game, outperformed their counterparts in Condition 1, who employed the kinect-based game, and this difference was statistically significant ($U(40)=139.5, z=-1.96, p<.05$) between the groups.

<table><caption>Table 3: Pre-test and Post-test scores</caption><thead><tr><th rowspan="2"></th><th colspan="2">Condition 1<br>Kinect-based game</th><th colspan="2">Condition 2<br>Desktop-based game</th><th rowspan="2">Z</th></tr><tr><th>Mean</th><th>SD</th><th>Mean</th><th>SD</th></tr></thead><tbody><tr><td>Pre-test scores</td><td>1.33</td><td>1.49</td><td>1.39</td><td>1.69</td><td>-0.11</td></tr><tr><td>Post-test scores</td><td>5.52</td><td>2.99</td><td>7.14</td><td>2.75</td><td>-1.96*</td></tr></tbody></table>

Note. *p<.05, **p<.01. ***p<.001

### Perceptions of technology integration

A Mann-Whitney U test was used to identify any potential differences in student's perceptions of technology integration across conditions. The results showed that students in Condition 2 had better perceptions of technology integration regarding "Student Negotiation" subscale. Yet, there were no statistical differences between the groups on all other subscales (see Table 4).

<table><caption>Table 4: Technology integration perceptions</caption><thead><tr><th rowspan="2"></th><th colspan="2">Condition 1<br>Kinect-based game</th><th colspan="2">Condition 2<br>Desktop-based game</th><th rowspan="2">Z</th></tr><tr><th>Mean</th><th>SD</th><th>Mean</th><th>SD</th></tr></thead><tbody><tr><td>Student Negotiation</td><td>3.36</td><td>1.01</td><td>3.97</td><td>0.60</td><td>-2.20*</td></tr><tr><td>Student Cohesiveness</td><td>3.85</td><td>0.97</td><td>4.08</td><td>0.70</td><td>-0.61</td></tr><tr><td>Reflective Thinking</td><td>3.41</td><td>0.95</td><td>3.81</td><td>0.79</td><td>-1.24</td></tr><tr><td>Competition & Efficacy</td><td>3.53</td><td>0.81</td><td>3.59</td><td>0.93</td><td>-0.06</td></tr></tbody></table>
Factors contributing to students' perceptions of technology integration

The thematic analysis led to the identification of numerous factors (codes) influencing students' attitudes, namely perceptions of technology integration, within the three dimensions of our conceptual framework: "Relationship", "Personal development" and "System maintenance and change". A conceptual map was the result of further organizing the emerging factors into basic themes: (a) Content-related factors, referring to the features of the gaming content, (b) Interface-related factors, referring to the affordances of the gaming platform, (c) Activity-related factors related to the pedagogical setting in which the game was contextualized and (d) Context related factors, referring to the characteristics of the physical environment in which the activity was enacted. All factors were evaluated as positive or negative in relation to their impact on students' perceptions (see Table 5).

<table><caption>Table 5: Categorization of factors reported as affecting students' perceptions on technology integration</caption><thead><tr><th rowspan="2">Framework Dimensions</th><th rowspan="2">Basic themes</th><th colspan="2">High-embodied condition [Kinect-based game]</th><th colspan="2">Lowly-embodied condition [Desktop-based game]</th></tr><tr><th>Positive factors (+)</th><th>Negative factors (-)</th><th>Positive factors (+)</th><th>Negative factors (-)</th></tr></thead><tbody><tr><td rowspan="5">Personal development</td><td rowspan="4">Content related factors</td><td>Learning content</td><td rowspan="4">Textual information</td><td>Learning content</td><td>Textual information</td></tr><tr><td>Gaming features</td><td>Gaming features</td><td></td></tr><tr><td>Narrative plot</td><td>Narrative plot</td><td>Repetition of gaming stages</td></tr><tr><td>Integrated scaffolding</td><td>Integrated scaffolding</td><td></td></tr><tr><td>Activity related factors</td><td>Worksheets</td><td>Gaming nature</td><td>Worksheets</td><td>Gaming nature</td></tr><tr><td>Interface related factors</td><td>Embodied interactions</td><td>Locomotion</td><td></td><td></td></tr><tr><td rowspan="4">Relationship</td><td rowspan="3">Activity related factors</td><td>Context related factors</td><td>Classroom noise</td><td></td><td>Classroom noise</td></tr><tr><td></td><td>Other group interventions</td><td></td><td>Other group interventions</td></tr><tr><td>Team-based mode</td><td>Large groups</td><td>Team-based mode</td><td></td></tr><tr><td></td><td>Collaborative writing task</td><td>Unstructured collaboration</td><td>Collaborative writing task</td><td></td></tr><tr><td></td><td>Peer feedback strategies</td><td>Waiting time</td><td>Peer feedback strategies</td><td></td></tr><tr><td rowspan="4">System Maintenance and Change</td><td rowspan="3">Interface related factors</td><td>Interface related factors</td><td>Single-player mode</td><td></td><td>Single-player mode</td></tr><tr><td>Novel interface</td><td>Gaming controls</td><td>Gaming controls</td><td>Small projection</td></tr><tr><td>Large projection</td><td>Synchronization issues</td><td></td><td>Low graphics interface</td></tr><tr><td></td><td>Bodily movement</td><td></td><td></td><td></td></tr><tr><td></td><td></td><td>Touchless interaction</td><td></td><td></td><td></td></tr></tbody></table>

Personal development

Students in both conditions reported how a set of content-related factors such as the *learning nature* of the game, the *gaming features* (e.g. stages, points, rewards), the *narrative plot* on which the game was structured, as well as the *integrated scaffolding* (e.g., hints and prompts) had a positive contribution to their personal learning development. E.g.

> "I liked the game's narrative plot as there was an alien trying to go back to his planet. We had to feed the alien with healthy foods. I liked the fact that every new planet was a new stage in the game with a new activity to do. It was an educational game because you could learn about nutrients in food." [#Girl-L-, Desktop-based version]

However, students in the high-embodied cognition highlighted that there was too much *textual information*, while students who worked in the lowly-embodied condition also added that there was a *repetition of the gaming stages*, which in turn had a negative impact of their interest. E.g.
"A negative factor in the game that I can think of was the large text. A box popped up in every new stage writing a lot of text. Text could be limited." [#Girl -E-, Kinect-based version]

Focusing on the activity-related factors, students in both conditions, added that while the *worksheets* that they were required to complete contributed positively to their personal development, the *gaming nature* sometimes inhibited the learning process, as in many cases the students would deal with the activity as a playful rather than as an educational experience. E.g.

"We were carried away when playing the game and were oftentimes forgetting to complete our paper assignment. It was difficult to remember later on what to write." [#Girl -L-, Desktop-based version]

Focusing on the interface-related factors, students in the high-embodied condition reported that while the activity allowed for *embodied interactions* which were valuable for their learning and personal development, *locomotion* was in some cases a negative aspect. In particular, as some of students admitted, in some cases they would be more focused on coordinating their body movements, rather than on the learning content. E.g.

"Moving my body did not help me being concentrated but I had a lot of fun. If I was stable in front of a computer I would have been more concentrate because I would focus on the screen and click the correct answer, rather than trying to coordinate my body." [#Girl -E-, Kinect-based version]

Finally, students in both conditions reported how the *classroom noise* and *other groups' interventions* while working, were two main context-related factors negatively affecting their personal development.

## Relationship

Students' in both conditions reported how a set of activity-related factors such as the *team-based mode* in which the activity was enacted, the *collaborative writing task* that were assigned (one worksheet to be completed by each group) and the *peer feedback strategies* that were followed, had a positive impact on their collaboration. More specifically, as the students mentioned all these factors promoted productive social interactions, such as exchange of views and ideas, peer scaffolding and assistance. E.g.

"I liked working in my group to complete the paper assignment. We worked collaboratively and we were helping each other. We were helping our co-players to choose the correct answer, we were giving instructions and we were encouraging each other to try harder." [#Girl -E-, Kinect-based version]

However, students in the high-embodied condition also negatively elaborated on how a set of activity-related factors, such as working in *large groups* (of 4 students) in combination to the *unstructured collaboration*, affected their relationships negatively. In particular, as the students admitted, both of these factors prohibited their effective collaboration, as it was more difficult to agree on a common strategy and plan their next steps, while there were also many disagreements with children often fighting over turn-taking and roles in the group. E.g.

"I wanted to play more but the other members in my team urged me to finish so they could play. There was also a boy taking my turn in the game. He wanted to play instead of me. I couldn't concentrate because my team members were telling me the correct answers, or they were trying to show me how to move. I got confused!" [#Boy -M-, Kinect-based version]

Finally, students in both conditions, highlighted that in terms of the interface-related factors, the *single-player mode* of the game, transformed the non-player(s) as spectators, and this had also a negative effect on students' relationships. Importantly in the high-embodied condition, this factor had an increased negative effect given the increased *waiting time* between turns, which resulted in off-task discussions and behaviors amongst the members of the group. E.g.

"The game was for a single player. All the other members of the group stayed aside, they had conversations with each other about topics unrelated to the game's content and they were not concentrated in their team members' actions neither on contributing to the group's collaboration." [#Girl -A-, Kinect-based version]

## System maintenance and change

According to the students of the high-embodied condition, the *large projection* (bigger screen providing more heightened sensory stimuli), the *interface* (with the use of novel technologies), as well as the affordances of the
gaming platform for promoting *bodily movement* (via the gesture-based interactions), contributed to their experienced immersion and this had a positive impact on their perceptions of technology use. E.g.

> "There was a large screen which seemed nicer and easier. I could see everything in that big screen. I could have better control of the game and I could feel like being in the game!" [#Boy-I-, Kinect-based version]

However, the students of the high-embodied condition reported that the *controls* of the game which were rather different from traditional gaming controls, some *synchronization* issues often presented between students' movements and their belated projection on the screen, as well as some technical bugs (provoked by students' proximity to the Kinect), affected their perceptions of technology in a negative way. E.g.

> "Sometimes there were problems with the technology. The game blocked and our hand signal was not appearing on the screen or was presented in a wrong position. This cost us time as we had to wait for the problem to be resolved!" [#Boy-I-, Kinect-based version]

Finally, students in the low-embodied cognition reported that the small projection (limited desktop screen) and the low graphics *interface* had a negative effect on their perceptions about the technology use. However, students in the low-embodied condition explained that the game had familiar *gaming controls* (keyboard and mouse) and thus, was more easily integrated in the lesson.

## Discussion and implications

The present investigation examined students' learning and perceptions of technology integration for group learning in a high-embodied, Kinect-based educational game, in comparison with a low-embodied, desktop-based version of the same game, in an authentic classroom setting. Findings suggest higher learning gains and more positive perceptions of technology integration, in terms of opportunities for student negotiation, for the students in the low-embodied condition.

*Are there differences in student's learning outcomes and attitudes between the conditions?* In the present work, there was no difference in most dimensions of students' attitudes, namely, perception of technology integration, across conditions. Yet, the opportunity for “student negotiation” was deemed higher in the low-embodied condition. Also, students in the low-embodied condition presented increased knowledge gains in comparison to their counterparts in the high-embodied condition. In general, the results contradict findings of prior research conducted in laboratory settings in which the prevalence of high-embodied, over low-embodied games, in students learning is presented (Homer et al., 2014; Lindgren et al., 2016). Indeed, the present study supports previous evidence (from a limited number of studies conducted in authentic school classrooms) that the high-embodied experience has not been as successful as initially expected in promoting students' learning compared to low-embodied, desktop-based environments (e.g., Anderson & Wall, 2016; Hung, Lin, Fang & Chen, 2014). That is, while being enjoyable and engaging, the experience with embodied learning technologies used in a typical classroom environment to run learning tasks may not always produce significant learning gains.

*What are the main factors affecting students' perceptions of technology integration in the two conditions?* The analysis of students' post-activity interviews shed light to our findings; a series of contextual factors were mentioned by students to have affected their perceived experience in a negative way. For example, common technical issues or a noisy classroom environment, may detract from, rather than enhance student learning, which is not a surprising result (e.g., Darling-Aduana & Heinrich, 2018). The study presented a conceptual map to summarize these factors into content-related factors, interface-related factors, activity-related factors and context related factors affecting the experience, positively or negatively, in both conditions. The map can be informative in future research and practice in the area allowing to control for some of these factors in the authentic learning environment.

Evaluating the outcomes of the present case study, several limitations should be noted. Conducting scientific research in a functioning classroom environment was challenging which naturally introduced flaws in the implementation of the study. For example, the overall time for the activity was fixed by the school time-table which did not allow much time for familiarization with the game mechanisms, especially in the high-embodied (Kinect-based) condition. The classroom's setting imposed several additional constraints; having a number of students in groups interacting with Kinect cameras created undesirable noise and interference. There is clearly much more work that could be done to explore the best ways of integrating embodied learning technologies within a classroom setting. Given the popularity of embodied learning technologies in the recent days, this work helps to identify issues which warrant future investigation in the field of emerging technology integration. A central question remains to be answered: under what circumstances can embodied technologies be educationally beneficial in authentic classroom settings? Future research may wish to focus on specific strategies for embodied
learning technology integration, designed to be immediately adopted by in-service teachers. Such work could be
beneficial in increasing integration of emerging technology promising to enable students' learning gains and
positive perceptions of embodied technology use for learning in groups.

## References

Abrahamson, D., & Sánchez-García, R. (2016). Learning is moving in new ways: The ecological dynamics of mathematics education. *Journal of the Learning Sciences*, 25(2), 203-239.

Anderson, J. L., & Wall, S. D. (2016). Kinecting physics: Conceptualization of motion through visualization and embodiment. *Journal of Science Education and Technology*, 25(2), 161-173.

Bonanno, P., & Kommers, P. A. (2008). Exploring the influence of gender and gaming competence on attitudes towards using instructional games. *British Journal of Educational Technology*, 39(1), 97-109.

Bressler, D. M., & Bodzin, A. M. (2013). A mixed methods assessment of students' flow experiences during a mobile augmented reality science game. *Journal of Computer Assisted Learning*, 29(6), 505-517.

Creswell, J. W., & Clark, V. L. P. (2017). *Designing and conducting mixed methods research*. Sage publications.

Darling-Aduana, J., & Heinrich, C. J. (2018). The role of teacher capacity and instructional practice in the integration of educational technology for emergent bilingual students. *Computers & Ed*, 126, 417-432.

Georgiou, Y., Ioannou, A.: Embodied learning in a digital world: A systematic review of empirical research in K-12 education. In: Díaz, P., Ioannou, A., Spector, M., & Bhagat K.-K.(eds.) Learning in a digital world: A multidisciplinary perspective on interactive technologies for formal and informal education. Springer series: Smart Computing and Intelligence. Springer (In press).

Homer, B. D., Kinzer, C. K., Plass, J. L., Letourneau, S. M., Hoffman, D., Bromley, M., ... & Kornak, Y. (2014). Moved to learn: The effects of interactivity in a Kinect-based literacy game for beginning readers. *Computers & Education*, 74, 37-49.

Hung, I. C., Lin, L. I., Fang, W. C., & Chen, N. S. (2014). Learning with the body: An embodiment-based learning strategy enhances performance of comprehending fundamental optics. *Interacting with Computers*, 26(4), 360-371.

Johnson-Glenberg, M. C., & Hekler, E. B. (2013). "Alien Health Game": An embodied exergame to instruct in nutrition and MyPlate. *Games for Health: Research, Dev, and Clinical Applications*, 2, 354-361.

Karakostas, A., Palaigeorgiou, G., & Kompatsiaris, Y. (2017, November). WeMake: A framework for letting students create tangible, embedded and embodied environments for their own STEAM learning. In *International Conference on Internet Science* (pp. 3-18). Springer, Cham.

Kosmas, P., Ioannou, A., & Retalis, S. (2018). Moving Bodies to Moving Minds: A Study of the Use of Motion-Based Games in Special Education. TechTrends, 1-8.

Kosmas P., Ioannou A., Retalis S. (2017). Using Embodied Learning Technology to Advance Motor Performance of Children with Special Educational Needs and Motor Impairments. In: Lavoué É., Drachsler H., Verbert K., Broisin J., Pérez-Sanagustín M. (eds) Data Driven Approaches. EC-TEL 2017. Lecture Notes in CS, vol 10474. Springer.

Lindgren, R., Tscholl, M., Wang, S., & Johnson, E. (2016). Enhancing learning and engagement through embodied interaction within a mixed reality simulation. *Computers & Education*, 95, 174-187.

Malinverni, L., & Pares, N. (2014). Learning of abstract concepts through full-body interaction: A systematic review. *Educational Technology & Society*, 17 (4), 100–116.

Maor, D., & Fraser, B. J. (2005). An online questionnaire for evaluating students' and teachers' perceptions of constructivist multimedia learning environments. *Research in Science Education*, 35(2-3), 221-244.

Melcer, E. F., & Isbister, K. (2016, May). Bridging the physical divide: A design framework for embodied learning games and simulations. In *Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems* (pp. 2225-2233). ACM.

Moos, R.H. (1987). *The social climate scales: A user's guide*. Palo Alto, California: Psychologists Press.

Teo, T., & Noyes, J. (2008). Development and validation of a computer attitude measure for young students (CAMYS). *Computers in Human Behavior*, 24(6), 2659-2667.

Wilson, M. (2002). Six views of embodied cognition. *Psychonomic bulletin & review*, 9(4), 625-636.

Wu, W., Chang, H. P., & Guo, C. J. (2009). The development of an instrument for a technology-integrated science learning environment. *International Journal of Science and Mathematics Education*, 7(1), 207.

## Acknowledgments

This work is part of a project that has received funding from the European's Horizon 2020 research and innovation programme under grant agreement N°739578 and the government of the Republic of Cyprus through the Directorate General for European Programmes, Coordination and Development.
# Effective Regulation in Collaborative Learning:
An Attempt to Determine the Fit of Regulation Challenges and
Strategies

Nadine Melzner, Martin Greisel, Markus Dresel, and Ingo Kollar

nadine.melzner@phil.uni-augsburg.de, martin.greisel@phil.uni-augsburg.de, markus.dresel@phil.uni-augsburg.de, ingo.kollar@phil.uni-augsburg.de

University of Augsburg

**Abstract:** University students often self-organize in groups for collaborative exam preparation. To learn effectively, they need to regulate emerging learning challenges through the choice of fitting strategies. So far, little is known about what strategies fit which challenges. We present a theoretical account for the definition of fit between regulation challenges and strategies in collaborative learning, thereby differentiating between direct and indirect strategies, and test its validity in an empirical study. We asked 163 university students in 90 groups to rate the challenges they encountered in their self-organized group meetings, and to report strategies by aid of which they regulated their biggest challenge. Answers were coded into 26 strategy types. We found that students mostly used strategy types that directly address their biggest challenges. Multilevel-modelling indicated that direct strategies are associated with higher satisfaction for coordination- and comprehension-related challenges only, but not for motivational challenges.

**Keywords:** regulation challenges, collaborative learning, regulation fit, exam preparation, self-regulated learning

## Problem statement

Many university students choose to prepare for exams in groups. Potential benefits of this are to maintain one's own discipline or to improve one's own understanding by discussing learning material (Järvenoja, Volet, & Järvelä, 2013). Yet, groups often face coordination-related, motivational, or comprehension-related challenges. When such challenges are not regulated, group members may be dissatisfied and reach poor group outcomes (Freeman, 1996). However, little is known about what strategies actually fit what kinds of challenges that may occur during collaborative learning and how their application relates to learners' satisfaction with their learning process. We (a) provide a theory-based analysis of the fit between learning challenges and regulation strategies and (b) investigate whether this fit is predictive for learners' satisfaction with their group learning experience.

## Theoretical Background

### Types of challenges

We propose that most challenges that may occur during collaborative learning fall into one of the following three categories: (a) coordination between group members, (b) motivational aspects, or (c) comprehension of learning topics and materials. Coordination-related challenges may arise when group members set different priorities in learning the exam contents, or when they set different learning goals (Järvelä & Järvenoja, 2011). Motivational challenges may emerge when group members, for example, cannot find interest in the exam topics. Finally, comprehension-related challenges may arise when groups have too little subject-specific prior knowledge, or when the learning material is structured in a confusing way. For successful exam preparation, learners need to effectively regulate these challenges. So far, studies (e.g., Malmberg, Järvelä, Järvenoja, & Panadero, 2015) have tried to predict the regulatory success by the quantity or quality of strategy use. Yet, in line with theoretical models of self-regulated learning (e.g., Zimmermann & Moylan, 2009), we additionally suppose that in order to be effective, there is the need for a chosen strategy to match the specific challenge at hand.

### Strategy fit

The fit of strategies to challenges has been described as an important factor that influences the success of learning regulation (Engelschalk, et al., 2015). Nevertheless, in collaborative learning, research about which strategy types are effective in what situation, i.e., what strategy types “fit” a given challenge, is scarce. Therefore, in this paper, we make an attempt to determine the appropriateness of specific strategy types to regulate specific coordination-related, motivational, and comprehension-related challenges.
We firstly assume that in a given situation some strategies may fit a current regulation challenge better than others. In one situation, group learners might for example differ in their ideas on how to proceed with the task (= a specific coordination-related challenge): One group member might want to recapitulate a topic that was already covered by the group, while other group members might prefer to improve their comprehension of a new topic. Now the group needs to react to the challenge by choosing a regulation strategy. It could, for example, apply a motivation strategy like planning to reward themselves after the learning session, or it could discuss these divergent plans and find an agreement on how to proceed. For the given situation, the latter strategy seems to be more effective to overcome the given challenge than the aforementioned strategy because it aims at directly eliminating the challenge. At least, communicating own plans and intentions is necessary to negotiate the further and joint proceeding (= to resolve the challenge). This does not mean at all that the reward strategy necessarily is inappropriate in this learning situation as a whole. Actually, it might support learning at a more general level because it might address different (motivational) problems that might be present in the same learning situation simultaneously. Nevertheless, it would rather not or not effectively help to dissolve the specific challenge of divergent plans. Hence, strategies that can help to *directly* eliminate a given challenge might be more likely to result in effective regulation than strategies that might support the learning process in a different, but non-direct way that leaves the central challenge itself unaffected. Thus, we regard the choice of strategies that directly aim at eliminating the challenge as “fitting”.

Other challenges, e.g., “distraction”, can be overcome by multiple strategy types. In this case, specific resource-oriented strategies (e.g., controlling the environment to reduce external stimuli), metacognitive strategies (discussing how to organize learning to hinder distraction), or a selection of motivational strategies (e.g., making the topic more fun to learn, i.e. increasing situational interest; see Table 1) seem promising, as they all address the problem in a direct manner (e.g., when any of these strategies is applied successfully, the problem should disappear). We assume that the amount of strategy types considered as being effective varies with the specificity or globality of the perceived challenge: Coordination-related and comprehension-related challenges seem more specific, thus less strategy types directly address them. Motivational challenges are more global and thus can be addressed by more different strategy types. Table 1 provides an overview about which strategy types theoretically can be considered to directly regulate different kinds of regulation challenges based on the considerations above.

## Relation of adaptive strategy use and satisfaction with the learning experience

One (among several) way(s) to determine the effectiveness of specific regulation strategies is through measuring learners' satisfaction with the overall learning experience. This assumption can be derived from the cyclic phase model of self-regulated learning (Zimmerman & Moylan, 2009): In terms of this model, satisfaction is a self-reaction which depends on the goals that were set in the beginning of a learning activity (or the evaluation standards in the model of Winne, Jamieson-Noel, & Muis, 2001). Concerning self-initiated study groups in preparation for an exam, one may assume that the main goal these students set for their meetings is to learn effectively (and eventually to receive good grades). Thus, the (subjectively) acquired knowledge that is accumulated during collaborative learning may be understood as a measure for learning success in the self-reflection phase. If the goals/standards are met, satisfaction should result.

Even though the assumption that regulation strategies need to fit the regulation challenges to yield effective learning is immanent in models of self-regulated learning, it has rarely been empirically tested. In one study, Järvelä, Volet, and Järvenoja (2010) filmed the regulatory activity of 63 teacher education students during collaborative learning tasks, and had participants rate the perceived intensity of various emotional challenges they experienced during group work as well as their satisfaction with their group learning process. Using qualitative analyses, they showed that one group demonstrated effective social and self-regulation of different challenges, and, at the same time, rated its group learning as “relatively satisfactory” up to “very satisfactory”. Even though this study investigated regulation fit only for emotional challenges, it suggests a link between the problem-specific fit of regulatory strategies and learners’ satisfaction with the learning experience. If we find that learners who apply strategies that based on our theoretical analysis directly address their challenges are more satisfied with their learning than those learners who apply other strategies, this result could be regarded as evidence that is in line with our theoretical assignment of strategies to challenges.

## Research questions and hypothesis

We ask the following research questions: To what extent do group learners use strategies that directly address a reported challenge compared to other strategies (RQ 1)? Can this extent be considered to be sensitive to the challenge type, i.e., is it more likely that direct compared to other strategies are chosen (RQ 2)? Are learners who apply direct strategies more satisfied with their learning than those who used other strategies and do differential effects depend on the problem type being regulated (RQ 3)? RQ 1 and RQ 2 will be answered in an exploratory
<table><caption>Table 1: Specific strategy types for direct regulation of specific coordination-related, motivational, and comprehension-related challenges</caption><thead><tr><th rowspan="2">Specific challenge</th><th colspan="4">Strategies directly addressing the challenge</th><th rowspan="2">Strategy example</th></tr><tr><th>Cognitive</th><th>Metacognitive</th><th>Motivational</th><th>Resource-oriented</th></tr></thead><tbody><tr><td><strong>Coordination-related Challenges</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>A. The group members have different goals for the meeting.</td><td></td><td>PRL</td><td></td><td>EM</td><td>PRL: "I advised her to take a closer look at it at home, as I do not want it that much in detail."</td></tr><tr><td>B. The group members seem to have incompatible working styles.</td><td></td><td>PRL</td><td></td><td></td><td>PRL: "I gave them tips for learning."</td></tr><tr><td>C. The group members seem to have different communication styles.</td><td></td><td>PRL</td><td></td><td></td><td>(No examples available)</td></tr><tr><td>D. The group members understand concepts / tasks differently.</td><td>RDU</td><td>PRL</td><td></td><td></td><td>RDU: "I have justified my conception of the task."</td></tr><tr><td>E. The group members have different ideas on how to proceed with the task.</td><td></td><td>PRL</td><td></td><td></td><td>PRL: "We discussed how we want to proceed."</td></tr><tr><td>F. The contributions of individual group members are differently strongly considered.</td><td></td><td></td><td></td><td>EM</td><td>(No examples available)</td></tr><tr><td>G. Not every group member always dares to participate.</td><td></td><td></td><td>ASI</td><td>EM</td><td>EM: "I participated in the study group."</td></tr><tr><td>I. Not all group members have enough time for the meeting.</td><td></td><td></td><td></td><td>TMC, KIM, AM</td><td>TMC: "I tried to use the time as effectively as possible through targeted questions."</td></tr><tr><td><strong>Motivational Challenges</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>H. The group has distraction problems.</td><td></td><td>PRL</td><td>SIT, DSG, ACC, FC</td><td>EC, AM</td><td>EC: "I have asked my classmates to turn off all phones."</td></tr><tr><td>K. The group members have motivational problems.</td><td></td><td></td><td>RS, SIT, IPS, LPS, ASI, DSG, AFC, HUG, ECG, UMS</td><td></td><td>RS: "I told her that there is only little content left and that we have already done most of the work"</td></tr><tr><td>M. The group members consider the study material to be boring.</td><td></td><td></td><td>RS, SIT, IPS, LPS, DSG, AFC, HUG, ECG</td><td></td><td>ESG: "Tried to keep them at it."</td></tr><tr><td><strong>Comprehension Challenges</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>J. The group members have only low prior knowledge of the learning contents.</td><td>SIC, CGP, ERM</td><td></td><td></td><td></td><td>SIC: "The mutual explanations were helpful for the whole group"</td></tr><tr><td>L. The group perceives the learning material as difficult.</td><td>ORR, SIC</td><td></td><td></td><td>TMC</td><td>TMC: "Tried to together clarify unclear contents"</td></tr><tr><td>N. The group members perceive the study material as confusing.</td><td>ORR, SIC</td><td></td><td></td><td></td><td>ORR: "We have created a mind map to structure the topic."</td></tr></tbody></table>

Note. Cognitive: Organizational / Knowledge Reduction Strategies / Knowledge Recall Strategies (ORR), Strategies for Improving Comprehension (SIC), Strategies for Closing Gaps in Prior Knowledge (CGP), Strategies to Resolve Differences in Understanding (RDU), Surface-oriented strategies / Knowledge Retention Strategies / Consolidation Strategies (SRC)

Metacognitive: Planning and Regulation of the Learning Process (PRL), Reflection and Evaluation of the current state of Knowledge (REK)

Motivational: Reward Strategies (RS), Increasing Situational Interest (SIT), Increasing Personal Significance (IPS), Learning and Performance-related Self-instruction (approach and avoidance) (LPS), Ability-related Self-Instruction (ASI), Declaring Successful Self-control as Goal (DSG), Accentuating Frame Conditions or Constraints (AFC), Highlighting Group Utility as a Goal (HUG), Emotional Contagion (ECG), Unspecific Motivational Strategies (UMS)

Resource-oriented: Time Management and Coordination (TMC), Environmental Control (EC), Knowledge and Information Management (KIM), Attention Management (AM), Effort Management (EM), External Resource Management (ERM), Care of the Social Atmosphere (CSA).
fashion. With regard to RQ 3, we hypothesize that learners who apply direct strategies to remedy their biggest regulation challenge are more satisfied with their group learning than students who use other strategies. Yet, we additionally explore whether there are differential effects of the problem type, i.e. that direct strategies are only related to satisfaction when addressing certain, but not other kinds of challenges.

# Method

## Sample

Participants were N = 174 university students (74.29% female, 25.14% male, 0.57% other) belonging to 90 self-organized study groups. They had an average age of 22 years (M = 22.26, SD = 2.68) and were, on average, in their 5th semester (M = 4.64, SD = 3.03) of their current studies and in their 6th (M = 6.21, SD = 3.85) semester of studies in general. Participants were enrolled in various study programs (e.g. teacher education, law, mathematics, computer science) at different German universities. The number of students per group participating in the study ranged from 1 to 5, with two participating group members being the most frequent fraction (23.27%). After each group meeting, participants were asked to (individually) answer an online questionnaire. Participants were reimbursed with 14 € if they filled in the questionnaire 3 times. For each additionally completed questionnaire, they received an additional 2.50 €. The maximum was 10 questionnaires (35 €).

## Instruments

### Questionnaire

The used questionnaire was a modified version of the Adaptive Instrument for Regulation of Emotions (“AIRE”; Järvenoja & Järvelä, 2009). It asked for the experience and regulation of coordination-related, motivational, and comprehension-related challenges during the current group meeting. Participants first were asked to rate on a five-point scale (1= “does not apply at all”, 5= “does squarely apply”) how intensively they experienced each of the 14 listed challenges during their group meeting (see Table 1). Then, they should mark their biggest of those 14 challenges and describe it briefly. This was followed by an item that asked students to rate their satisfaction with the respective group meeting on a seven-point rating scale (1= “totally unsatisfied”, 7= “totally satisfied”). In the end, participants were asked to enter strategies they used to regulate their biggest self-categorized challenge at the self-, co-, and shared level in an open answer format (e.g. self: “What did you personally think, do or say to ensure high quality of your own learning in this situation?”, co: “...of the learning of individual others”, shared: “...of the learning of the group as a whole”; Järvelä & Hadwin, 2013).

### Coding procedure

After data collection, a coding scheme to classify learning strategies was developed, based on established schemes of Mandl and Friedrich (2006), and of Engelschalk et al. (2015). It contained 26 categories: Twenty-four specific strategy types (see “Note” in Table 1 for a complete list), one residual category for unspecific strategies, and one category when no strategy was provided. Trained coders’ categorization yielded a sufficient interrater agreement (Cohen's κ = .85). After coding, a binary “fit” variable was generated by assigning the values 1 or 0 based on the aforementioned theoretical scheme (1 if a person mentioned at least one strategy directly addressing the perceived biggest challenge, 0 if only other strategies were mentioned; see Table 1 for details about which strategies theoretically match which challenges).

## Analysis

To answer RQ 1, we calculated absolute and relative frequencies of the use of the different strategies (together with 95% confidence intervals) and visualized them with a decision tree (see Figure 1). Concerning RQ 2, we added ratios “R” for each challenge type: To give an impression of how to interpret the magnitude of the relative frequencies of strategies directly addressing a challenge to other strategies (= whether regulation is adapted to a specific challenge), we calculated ratios of the empirical to the theoretical relations of direct and total strategy use. The theoretical relation signifies the probability to choose a direct strategy, if a random distribution across all strategy categories was assumed, weighted with the average number of strategies reported for each challenge type. In summary, this ratio expresses the empirical probability that a person directly addresses a specific challenge qualified by what would be expected if there was no adaption in the choice of regulation strategy to a challenge at all. Values above 1 signify that more participants actually chose direct strategies than one would expect under the assumption of random distribution of strategies, whereas values below 1 signify that less participants chose direct strategies than one would expect if there was no adaption to challenge. The deviation from 1 was tested by asymptotical X²-tests.
To test if choosing a strategy type that fits a current challenge increases the satisfaction with learning (Hypothesis 1), hierarchical linear modelling using R (reml estimation) was conducted. Data was analysed using a three-level data structure (group meetings, persons, groups; Singer & Willett, 2003). Students' satisfaction with their learning and group differences were estimated by specifying the *unconditional means model* where satisfaction with learning is independent of predictors. Nevertheless, we allowed for random variations in satisfaction between groups and individuals in groups (random parameters r₀, u₀₀). We z-standardized the satisfaction variable to interpret differences in satisfaction in units of standard deviation. General development of students' satisfaction with their learning across group meetings and group differences in these developments were estimated by specifying the *unconditional growth model*. In this model, satisfaction with learning is modeled as dependent on the number of learning sessions (γ₀₀₁). This time effect is allowed to vary randomly between individuals and between groups (random parameters r₀, u₀₀). The group meeting was coded by number to approximately interpret the parameters of the time variable as changes per meeting (approximately because some groups already existed before our study). Then, we specified a *fit model* that included the effects mentioned above and additionally the effect of the fit between the challenge and strategy type on satisfaction. The fit variable was included as a fixed effect because there is no plausible argument for random variation over groups and individuals in groups. Last, we added the *interaction of fit and problem type* to the last model.

# Results

## Preliminary analysis

We first looked at whether the types of challenges we presented to participants actually fell into the three assumed categories "coordination-related", "motivational", and "comprehension-related" challenges. To judge this, we used the individual ratings to what extent the 14 challenges actually occurred in the group in exploratory and confirmatory factor analyses. The EFA supported the choice of 3 factors as the optimal factor structure by 3 (out of 9, 33.33%) methods (Optimal Coordinates, Parallel Analysis, VSS Complexity 2). A further CFA indicated a superior model fit for the three-factor structure compared to theoretically plausible one- or two-factor structures. However, the overall model fit was not optimal ($\chi^2=2965.899$, p< .001, $\chi^2/df=32.59$, CFI=0.89, RMSEA=0.07, SRMR=0.06). Since the following calculations are performed separately for each specific challenge, and grouping into larger problem dimensions was used for clarity of visualization only, the decision for a three-factor structure did not pose a problem for later analyses.

## Distribution of fit between learning challenges and regulation strategies

Concerning RQ 1, Figure 1 shows the transition probabilities with which participants who encountered a specific challenge type chose a direct strategy. It also provides information on whether a larger or smaller proportion of participants regulated a specific challenge with a direct versus with other strategies. It can be seen that direct strategies were more frequent than alternative strategies across all types of challenges except for "boring learning material" for which the confidence interval includes equal frequency (50%). Overall, these results suggest that students predominantly choose strategy types for directly controlling specific challenge types, indicating high fit between selected strategies and regulation challenges. The exception for "boring learning material" was surprising since our theoretical model assumed a whole set of different strategy types to directly address this challenge type. Although the baseline probability of choosing a direct strategy was rather high, students chose other types of strategies equally often here. In figure 1, the ratios "R" signifying the relation of the empirical percentage of direct strategies to what could be expected under the assumption of a random distribution of strategies across all coding categories demonstrate values significantly (if testable due to low cell frequencies) greater than 1 for eleven out of 14 challenge types (coordination problems were not listed individually in the tree for lack of space). This hints towards a rather high ability of students to adaptively select direct strategies to remedy their regulation challenges (RQ 2). Though, the ratios of the three motivational challenges were significantly below 1, indicating regulation to be sensitive to the kind of challenge as well, but in an unexpected direction. For these problem types, students chose direct strategies less frequently and other strategies more frequently than expected when strategy selection would be random. In order to test whether the observed regulation differences between coordination- and comprehension-related problems on the one hand and motivational problems on the other hand also show up in regard to satisfaction, this split of problems was maintained in the multi-level models. So, we suppose that direct strategies are positively associated with satisfaction only for coordination- and comprehension-related challenges, whereas for motivational challenges, they might be negatively associated.
<figure><img src="image_139.png" /><figcaption>**Figure 1.** Decision tree with (relative) frequencies of challenges and direct vs. other strategy types (with 95% confidence intervals). Ratios “R” show the empirical probability for directly addressing a specific challenge compared to the probability expected with no adaptation in the choice of regulation strategy to a challenge. * *p* ≤ .05, ** *p* ≤ .01, *** *p* ≤ .001.</figcaption></figure>

## Relation of fit with outcome

Regarding RQ 3, the unconditional means model (see Table 2) showed that 19% of the variance of satisfaction was explained by the group level. Thus, groups differed in the effectiveness of regulation. The unconditional growth model retained this effect but showed a main effect of time (γ₁₀₀) indicating that satisfaction decreased across learning sessions in average. Further, the fit model showed the effect of time again (γ₁₀₀) but yielded no significant effect for fit (γ₂₀₀). Thus, the fit between the biggest challenge and the selected strategy type to regulate this challenge did not significantly predict satisfaction. The interaction of fit and problem type model revealed a main effect for problem type (γ₃₀₀) indicating that group learners were more dissatisfied with their learning when facing motivational instead of coordination- or comprehension-related problems. This model further revealed a significant (one-sided according to hypothesis stated above) interaction effect between fit and problem type (γ₄₀₀). Thus, for coordination- and comprehension-related challenges, satisfaction was higher when direct instead of other strategies were chosen, but was lower for motivational challenges.

<table><caption>**Table 2: Outcomes of multi-level models**</caption><thead><tr><th rowspan="3"></th><th colspan="5">Satisfaction with the learning</th></tr><tr><th rowspan="2">Unconditional means model</th><th colspan="2">Unconditional Growth Model</th><th colspan="2">Fit Model</th></tr><tr><th></th><th colspan="4">Fixed Effects</th></tr></thead><tbody><tr><td>Intercept γ<sub>000</sub></td><td>-0.030</td><td>0.116</td><td>(0.078)</td><td>0.115</td><td>(0.078)</td><td>0.107</td><td>(0.078)</td></tr><tr><td>Time γ<sub>100</sub></td><td></td><td>-0.037**</td><td>(0.011)</td><td>-0.036**</td><td>(0.011)</td><td>-0.036**</td><td>(0.011)</td></tr><tr><td>Fit γ<sub>200</sub></td><td></td><td></td><td></td><td>0.044</td><td>(0.034)</td><td>0.042</td><td>(0.034)</td></tr><tr><td>Problem type γ<sub>300</sub></td><td></td><td></td><td></td><td></td><td></td><td>-0.083*</td><td>(0.035)</td></tr><tr><td>Problem type*Fit γ<sub>400</sub></td><td></td><td></td><td></td><td></td><td></td><td>-0.061*</td><td>(0.033)</td></tr><tr><th colspan="8">Random Parameters</th></tr><tr><td>Level 2 (Persons)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Intercept Var(Γ<sub>0</sub>)</td><td>0.053*</td><td>0.048*</td><td></td><td>0.050*</td><td></td><td>0.049*</td><td></td></tr><tr><td>Level 3 (Groups)</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Intercept Var(v<sub>000</sub>)</td><td>0.193***</td><td>0.205***</td><td></td><td>0.201***</td><td></td><td>0.197***</td><td></td></tr></tbody></table>

*Note.* Standard errors are in parentheses. Time = the number of study group meeting (Level 1), Fit = the fit between biggest challenge and the chosen strategy type to regulate this challenge (Level 1). *** *p* ≤ 0.001, ** *p* ≤ 0.01, * *p* ≤ .05.
**Discussion**

The aims of our study were twofold. First, we wanted to investigate to what extent group learners who encounter specific coordination-related, motivational, and comprehension-related challenges during their exam preparation select strategy types that directly address those challenges. In this context, we also reflected on whether this extent can be considered to be sensitive to the challenge type. Second, we tried to validate our postulated fit categorization by regressing differences in satisfaction with the learning experience on the fit of the strategy types students selected with the biggest learning challenge they experienced during learning.

As expected, learners in groups particularly seem to choose strategy types directly addressing their biggest challenge. For thirteen out of 14 challenge types, they chose significantly more direct than other strategies. When additionally controlling for baseline probability of the measurement instrument, they still regulated eleven out of 14 challenges sensitive to the specific challenge. Thus, group learners seem to be able to adaptively regulate their learning challenges dependent on the kind of challenge they are confronted with. In line with the SRL model of Zimmerman and Moylan (2009), it may be that learners choose strategies that match their experienced challenges because they have turned out to be most effective for their control. However, this result seems to contradict observations of Engelschalk et al. (2015) who found that students did not differentiate much in their strategy choices and rather activate proven strategies repeatedly across situations. The diverging findings might be related due to the different study designs or contexts: In the study by Engelschalk and colleagues (2015), individual learning students were presented with hypothetical motivational problems and asked to describe how they would regulate each of them. Contrary to that, we asked group learners to report their real experienced coordination-related, comprehension-related, and motivational learning challenges.

For motivational challenges, though, we found regulation to be sensitive to the reported challenge as well—but in the opposite direction than we had expected. For one out of three motivational challenges, direct strategies were not more frequently used than other strategies. Since we proposed that there are more direct strategy types for regulating motivational than coordination- or comprehension-related challenges, this is even more surprising. When this baseline rate was controlled for, direct strategies were less significantly chosen for all three motivational challenges than expected under random distribution. Hence, learners who encounter motivational challenges seem to specifically avoid using motivational strategies. There are at least the three different explanations for this result: First, our theoretical considerations might have been wrong so that we failed to identify the right strategy types to directly address motivational challenges. Second, the matching we derived based on our theoretical understanding might be applicable but our participants were not able to select direct strategies and chose other strategies due to their lacking strategy knowledge or due to an inability to activate direct strategies here. Third, the social nature of the regulation situation might show differential effects of social desirability: According to this interpretation, it might be plausible that group members appreciate receiving help from another group member in understanding the learning content (= a comprehension-related challenge), or in negotiating the subsequent learning process (= coordination-related challenge). In case of motivational problems, however, it might be much less desirable to intervene as a group member. Trying to motivate a group member might be perceived as unwanted intrusion and attempt to change the attitudes towards the learning task and thus produce reactance within the group. For a less obvious and less socially conflict-loaded control of their group members' motivation, learners could have tried to indirectly motivate others through providing them assistance in learning the material.

Following Zimmerman and Moylan (2009), we supposed that if strategies were applied that match the learning challenge, then successful regulation should be likely. Regulation success, in turn, should result in satisfaction with the learning process. Therefore, we tested the association of the directness of strategy choice with satisfaction with the learning experience as an indication of criterion validity of our theoretical model on the fit between strategies and challenges. Controlling for random variations of satisfaction between groups and between individuals in groups, and for fixed and random effects of time, we found a significant link between the fit of strategy to challenge type and satisfaction with the learning experience only for coordination- and comprehension-related, but not for motivational challenges. Thus, we assume that our conceptualization of fit is actually predictive of regulation success measured by satisfaction rating, but differentially for specific challenge types. Additionally, this result mirrors the finding of the single case analysis of Järvelä et al. (2010).

**Limitations and conclusions**

Of course, this study has limitations: First, we did not have a direct measure of regulation success. Using satisfaction as a more distal measure surely is influenced by more than just the directness of chosen strategies. Perhaps the relationship between strategy-challenge fit and learning outcomes would be stronger if learning outcomes would be assessed by aid of objective measures (e.g. standardized test measuring the acquired knowledge). Second, even though we investigated regulation in real groups, analyses were based on self-reported
strategy use. Video-based analysis of real group interactions would be valuable and could nicely complement our more subjective data. Third, we looked for the fit between challenge and strategy types on an aggregated level. If the individual open-answer descriptions of each challenge and the corresponding regulation strategies would be used to propose a new fit, the results might provide a more precise picture of an even larger amount of direct strategies. Thus, as before, we believe that the estimated relative frequencies were also underestimated in this study. As a consequence, we plan to determine the fit for each individual strategy to each individual challenge as a future step. Furthermore, we have proposed strategy fit only in reference to the biggest challenge that participants experienced. Students who appropriately coped with all challenges during their group meetings except for their biggest challenge were likely to have lowered the empirical relationship between fit and satisfaction.

An important educational implication of our study is the following: We saw that students do not regulate motivational challenges much with strategies that (theoretically) would fit. In case that this results proves reliable in further studies, students should be trained in effectively regulating motivational challenges in particular, to increase their use of direct strategies for such challenges. For individual learning situations, Eckerlein, Steuer, and Dresel (2018) already showed that effective motivational regulation can successfully be fostered through dedicated training programs. Also, scaffolds or scripts (Kollar, Fischer, & Slotta, 2007) may additionally prompt group learners to adjust their use of strategies to challenges for more effective and satisfying collaborative learning experiences.

## References

Boekaerts, M. (1997). Self-regulated learning: A new concept embraced by researchers, policy makers, educators, teachers, and students. *Learning and Instruction*, **7**, 161–186. https://doi.org/10.1016/S0959-4752(96)00015-1

Eckerlein, N., Steuer, G. & Dresel, M. (2018). Training university students in motivational regulation. Manuscript submitted for publication.

Engelschalk, T., Steuer, G., & Dresel, M. (2015). Wie spezifisch regulieren Studierende ihre Motivation bei unterschiedlichen Anlässen? *Zeitschrift Für Entwicklungspsychologie Und Pädagogische Psychologie*, **47**, 14–23. https://doi.org/10.1026/0049-8637/a000120

Freeman, K. A. (1996) Attitudes toward work in project groups as predictors of academic performance, *Small Group Research*, 27(2), 265-282.

Järvelä, S., & Hadwin, A. F. (2013). New Frontiers: Regulating Learning in CSCL. *Educational Psychologist*, **48**(1), 25–39. https://doi.org/10.1080/00461520.2012.748006

Järvelä, S., & Järvenoja, H. (2011). Socially constructed self-regulated learning and motivation regulation in collaborative learning Groups. *Teachers College Record*, **113**(2), 350–374.

Järvelä, S., Järvenoja, H., & Veermans, M. (2008). Understanding dynamics of motivation in socially shared learning. *International Journal of Educational Research*, **47**, 122–135.

Järvelä, S., Volet, S., & Järvenoja, H. (2010). Research on motivation in collaborative learning: Moving beyond the cognitive-situative divide and combining individual and social processes. *Educational Psychologist*, **45**(1), 15–27. http://dx.doi.org/10.1080/00461520903433539

Järvenoja, H., Volet, S., & Järvelä, S. (2013). Regulation of emotions in socially challenging learning situations: an instrument to measure the adaptive and social nature of the regulation process. *Educational Psychology*, **33**(1), 31–58. https://doi.org/10.1080/01443410.2012.742334

Kollar, I., Fischer, F., & Slotta, J. D. (2007). Internal and external scripts in computer-supported collaborative inquiry learning. *Learning and Instruction*, **17**, 708–721. https://doi.org/10.1016/j. learninstruc.2007.09.021

Malmberg, J., Järvelä, S., Järvenoja, H., & Panadero, E. (2015). Promoting socially shared regulation of learning in CSCL: Progress of socially shared regulation among high- and low-performing groups. *Computers in Human Behavior*, **52**, 562–572. https://doi.org/10.1016/j.chb.2015.03.082

Mandl, H., & Friedrich, H. F. (Eds.). (2006). *Handbuch Lernstrategien*. Göttingen: Hogrefe.

Singer, J., & Willett, J. (2003). *Applied longitudinal data analysis: Modeling change and event occurrence*. New York, NY: Oxford University Press.

Winne, P. H., Jamieson-Noel, D., & Muis, K. Methodological issues and advances in researching tactics, strategies, and self-regulated learning. In *New Directions in Measures and Methods* (pp. 121–155). Elsevier Science Ltd.

Zimmerman, B. J., & Moylan, A. R. (2009). Self-regulation: Where metacognition and motivation intersect. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), *The educational psychology series. Handbook of metacognition in education* (pp. 299-315). New York, NY, US: Routledge/Taylor & Francis Group.
# Unpacking Socio-Metacognitive Sense-Making Patterns to Support Collaborative Discourse

Marcela Borge, Tugce Aldemir, and Yu Xia

mborge@psu.edu, tfa5065@psu.edu, yzx64@psu.edu

The Pennsylvania State University

**Abstract:** This study explores the dynamics between socio-metacognitive communication patterns and collaborative processes, as students engage in collaborative discussions about course concepts. Building upon a series of studies that aimed to design and validate an intervention to help students develop collaborative competencies at the group level, the study aims to map how socio-metacognitive sense-making patterns are associated with the collaboration quality, by comparing the patterns for low, medium, and high performing teams. Discussion and after-discussion reflection transcripts of 12 teams over five sessions were analyzed and assessed, using previously developed collaborative discourse rubric and sense-making coding construct. The results showed a significant correlation between frequency of sense-making acts and the quality of the collaborative discourse.

## Introduction

Collaborative competencies are essential sets of skills for society and it is vital to identify strategies and approaches to nurture them in learners. Unfortunately, many learners do not have the opportunities to develop collaborative competencies, which can lead to poor performance outcomes and undesirable group conflicts (Borge & Carroll, 2014; Fischer et al., 2013; Kozlowski & Ilgen, 2006). In addition to individual problems associated with underdeveloped collaborative skills, in CSCL environments, group-level issues such as social loafing, sucker effect (Salomon & Globerson, 1989), lack of non-verbal expressions, and time lag between the interactions can contribute to lower quality collaboration (Kreijns, Kirschner, & Jochems, 2003). This is why helping learners engage in higher quality collaborative processes has been a central concern for CSCL. As such, many CSCL researchers have attempted to help students improve collaborative skills, but few have recognized the complex and nested nature of collaboration (Borge & White, 2016; Borge, Ong Shiou, & Rosé, 2018; Baker et al., 2007; Stahl, 2006).

Prior research has indicated that without any guidance, support, or training, students tend to demonstrate dysfunctional group processes (Borge at al., 2018; Barron, 2003; Hogan, 1999; Webb & Palincsar, 1996). To address this problem, existing studies have designed and evaluated interventions to support group function, including (1) scripting collaborative interactions, (2) helping students improve metacognitive sense-making through reflecting upon individual and group performances and comparing them to the models of competence, (3) supporting students to develop self- and group-regulatory behaviors, and so forth. However, there still are issues that need further investigation, such as what kind of scripting is needed, how to balance the level of scripting, when to fade the external support, and how to help groups internalize the external support (Borge & White, 2016; Hogan, 1999; Kozlowski, Watola, Jensen, Kim, & Botero, 2009).

Building upon the critical role that self-regulation plays in individual's learning (Zimmerman & Schunk, 2001), Borge et al.'s (2018) suggested that metacognitive guidance and regulation at the group level can serve to help students optimize their collaborative experiences. They referred to collective awareness of and the collective ability to monitor and regulate the collaborative process as socio-metacognition. Borge et al.'s (2018) findings were promising, but also highlighted the need for further research on the development of socio-metacognitive expertise. Similarly, the scarcity of research on the interplay between socio-metacognitive process and collaborative discussion necessitates further exploration of their dynamics in CSCL context (Kwon, Liu, & Johnson, 2014; Rogat & Adams-Wiggins, 2015). This paper builds upon this existing literature and aims to extend what is known about collective regulation by identifying critical socio-metacognitive sense-making patterns in process-related dialogue acts in real-world collaborative learning contexts. Thus, we explored the dynamics between socio-metacognitive sense-making patterns and collaborative processes, as students engaged in collaborative discussions about course concepts.

## Theoretical framework

Our work is influenced by the theory of group cognition and thus recognizes collaboration as a form of nested cognition that entails intersubjective knowledge construction and collective sense-making that are situated at different levels, including individual, group, and community levels (Stahl, 2006). Collaborative activities offer
sophisticated learning opportunities that might not be available in individual learning activities, which are likely
to help learners meet the need in our society for higher collaborative competencies and to succeed in collaborative
teams. However, problems often arise in collaborative activities, especially in terms of the collective sense-making
processes (Barron, 2003). The collaborative discourse that occur during group interactions has a large influence
on the group performance and outcome (Kozlowski & Ilgen, 2006; Stahl, 2006). Thus, the examination of
communication patterns can help evaluate the quality of collaborative discourse.

Challenges for group regulation

Literature on self-regulation in individual learning has identified problems that pose barriers for learners to successfully regulate their own learning, such as not being aware of learning problems, misdiagnosing them, etc. (Winne & Nesbit, 2009). In collaborative contexts, these individual regulation problems arise alongside group regulation problems, where may teams fail to identify or accurately assess problematic processes and devise remediation strategies. There are also social and emotional difficulties involved in communicating problems and regulating group behaviors (Järvelä, & Hadwin, 2013). In each stage of regulating collaborative process, there is a requirement on building shared understanding through synthesizing information and idea negotiation (Stahl, 2006). We argue that in order for teams to engage in high-quality collaborative activities, they need to develop socio-metacognitive expertise: the knowledge of and ability to monitor and regulate collective cognitive processes.

Socio-metacognitive sense-making

As collective literature argues, engaging in high quality collaborative discussion is a hard but crucial skill to develop. Despite multiple complexities and interrelated variables to consider, research also suggested that learners can improve the quality of their collaborative activities over time, if they learn how to regulate their collaborative process (Borge et al., 2018; Kozlowski et al., 2009).

Collaborative interactions are the externalized forms of collective thinking, and thus, how teams collectively make sense of, monitor, and regulate these interactions play a central role in collaborative process. Recognizing the role of group regulation in collaborative activities, recent research efforts have been directed toward developing technological tools to support group regulation. As one of the pioneers of these efforts, Järvelä and Hadwin (2013) developed a technological tool to help learners develop awareness and planning strategies for their collaborative activities. However, developing awareness and planning is not sufficient for the regulation process; learners need to develop an understanding for how a high-quality collaborative discussion should be like, compare their process to a model, identify problems in their process, collectively develop or choose appropriate remedial strategies to solve problems, and take action (Nesbit, 2012; Winnie, & Nesbit, 2009). In group contexts, these steps demand both individual and collective efforts, attention, and time, which might be the reason why teams tend to neglect their collaborative process while reserving all the attention to make sense of the content (Kerr & Tindale, 2004). Collective sense-making of the collaborative process requires teams to monitor and reflect on their collaborative activity (Nesbit, 2012). However, research suggested that individuals do not perform well at asking and addressing these questions, and situation gets even intensified when sense-making moves from individual to group cognition (Gabelica et al., 2014).

In our previous work, we addressed problems associated with collective regulation and developed a theoretically informed technological intervention to help students develop their socio-metacognitive expertise. We identified the communication patterns associated with high quality collaborative discussion, proposed two core capacities for collaborative sense-making, and listed concrete patterns of communication associated with more or less optimal collaborative sense-making processes. We then helped students monitor and regulate their collaborative processes by guiding and constraining how they get prepared for discussion and how they engaged in socio-metacognitive sense-making and regulation after the discussion. The intervention succeeded in getting teams to improve the quality of their collaborative activity over time. However, we did not fully examine the dynamics between socio-metacognitive competence, collective sense-making, and the quality of the collaborative discussion. The collection of research on socio-metacognitive sense-making suggests that it is critical to identify how teams engage in socio-metacognitive sense-making and regulation of both course content and team process discourse, and how these interconnected processes may impact each other. To address this need, we aim to identify socio-metacognitive sense-making patterns in collaborative discussions and thus examined how these patterns are associated with team collaborative performance. Our research questions were:

(RQ1) What patterns of socio-metacognitive sense-making (SMS) talk do teams engage in when unpacking course content and thinking about their own discussion processes?

(RQ2) What are the differences in these patterns between low, medium, and high performing teams?

Methods
## Course context and participants

The study was conducted in an online 16-week undergraduate course designed to introduce students to information science concepts. As part of the course, students were expected to engage in collaborative reasoning practices and discussion activities. Developing collaborative discussion skills was one main goal of the course. Participants were 34 online students who enrolled in the course (11 females, 33.3%; 22 males, 66.7%). Students' ages ranged from 25 to 44, and the majority of students were part-time students with full-time jobs.

## Procedure

Students were assigned to 12 teams of three based on when they were available to meet. Due to two students dropping the class, two teams ended up as dyads. These teams were required to meet synchronously for five sessions to collectively make sense of course concepts. They met every other week for ten weeks. As a pre-discussion activity, the students were required to read the weekly readings and write an individual reflection in response to four higher-order questions about the readings. Then, they were asked to set a meeting time with their teammates to synchronously discuss the questions and readings. Each discussion session was about 90 minutes: 60-minute main discussion, 15-minute individual assessment of team discussion, and 15-minute collective planning discussion. These activities counted towards 25% of students' grades. For the 15-minute individual assessment of the team discussion that followed the 60-minute main discussion, students were provided with a collaborative process rubric detailing how to assess discussion quality, guides containing goals for collaboration, problems that interfered with good collaboration, and strategies for improving collaborative processes. The discussions were held on a computer supported collaborative discussion environment, and saved automatically in the system. After each individual scored their team's collective processes, the entire team was responsible for completing a collective planning session, where they discussed their scores and process weaknesses they identified during individual reflection for the purpose of collectively diagnosing problems and planning out strategies the team could use to improve in future sessions.

## Research design and analysis

We implemented explanatory mixed methods (Creswell, 2015). The main discussions, individual reflections, and collective planning sessions were collected and analyzed for 12 teams across five sessions, following the verbal analyses guidelines offered by Chi (1997). We coded a total of 12,755 utterances, 10,764 from content-based discussions and 1,991 from team planning sessions. All ethical guidelines were followed in collecting, analyzing, and reporting the study.

## Evaluating teams' discourse quality

Building upon previous theoretical frameworks (Borge et al., 2018), teams' discourse quality, when teams work to collectively understand course content, is defined as the extent to which team members provide evidence of engaging in communication patterns associated with high-quality information synthesis and knowledge negotiation (see Table 1). The quality of teams' discussions was assessed by a research assistant with two years of communication analysis training, using a rubric developed by Borge et al., (2018), which measures two core capacities each with three categories of behavior (see Table 1). To score each item, the entire transcript is

<table><caption>Table 1: Summary of core capacities and categories of behavior associated with high quality collaborative communication behaviors from Borge et al. (2018).*</caption><thead><tr><th>Core Capacities</th><th>Categories of Behavior</th><th>What is Examined in the Transcript</th></tr></thead><tbody><tr><td rowspan="3">Information Synthesis</td><td>Verbal Equity</td><td>To what extent team members contributed equally.</td></tr><tr><td>Developing Joint Understanding</td><td>To what extent team members make an effort to ensure that they fully understand the ideas.</td></tr><tr><td>Joint Idea Building</td><td>To what extent team members elaborate on others' contributions.</td></tr><tr><td rowspan="3">Knowledge Negotiation</td><td>Exploration of Different Perspectives</td><td>To what extent teams present and discuss alternative perspectives.</td></tr><tr><td>Quality of Claims</td><td>To what extent teams provide logical and fact-based evidence to their claim.</td></tr><tr><td>Norms of Evaluation</td><td>To what extent teams adhere to social norms.</td></tr></tbody></table>

Note: * Each score ranging from 1 to 5 outlines a set of guidelines indicating what each score means for each category. For example, in Quality of claims, a score of five means "There are at least two examples where claims are supported by references to course readings or online content AND at least one example of weighing of options or examination of different perspectives." (Borge et al., 2018).
examined for specific discourse quality markers (see top of figure 1 for example) and these markers are used as a means to provide evidence for a score from 1 to 5 (see bottom of Figure 1). Scores for the six categories were summed to a single collaborative discussion quality score for each discussion. Once all the five main discussion scores were identified, they were averaged to produce the average collaboration performance of each team. The quality of the collaborative discussion was assessed using only the main discussion transcripts. 20% of the data were double coded by two trained students with extensive communication analysis experience. Significant agreement was reached, r = .86, p < .001; Kappa = .64, p < .001.

<table><thead><tr><td>Session</td><td>Turn</td><td>User ID</td><td>Entry</td><td>Codes</td></tr></thead><tbody><tr><td>1</td><td>29</td><td>86</td><td>He tries to balance his views with a pro for privacy violations, asserting that it may be used for national security. Do you agree with this implication?</td><td>DJU-1, JIB-1</td></tr><tr><td>1</td><td>31</td><td>85</td><td>I apologize for the delay, I've been rereading your post, I'm not sure I understand what you mean. The author is pro privacy violation?</td><td>DJU-1</td></tr></tbody></table>

<table><caption>Figure 2. Two screenshots depicting how lines of transcript are examined for specific markers for developing joint understanding (DJU-1 represents first instance of DJU) and joint idea building (JIB-1 represents first instance of JIB) (top) and how these markers are used as evidence for discourse quality scores (bottom).</caption><thead><tr><td>Criteria</td><td>Score</td><td>Evidence</td></tr></thead><tbody><tr><td>Joint Idea Building</td><td>4</td><td>I could not locate more than one instance [of JIB] where the participants extended or supported the original claim with additional information. The participants seemed to either skip to another subject very quickly or discuss an opinion over one or two turns.</td></tr><tr><td>Developing Joint Understanding</td><td>5</td><td>There are at least two instances where one participant made an effort to ensure s/he understood what the previous participant said. They used the following strategies: (DJU-1) Rewards another member's ideas to make sure s/he understands it; (DJU-2) Asks another member to explain an idea by elaborating further.</td></tr></tbody></table>

## Analysis of socio-metacognitive sense-making (SMS) talk

The transcripts of main discussions and after-discussion collective reflection and planning sessions were analyzed for 12 teams across five sessions, for a total of 60 analyzed discussions. Socio-metacognitive sense-making is a specific type of process talk where students think about their collaborative processes to try to understand or modify them. To identify SMS talk, main discussions and reflection discussions were segmented into chat turns; then each turn was coded as process (P), content (C), or other (O). Inter-rater reliability for 20% initial coding was Kappa = .79, p < .001. The total frequency of P acts varied for each team. To compare teams' SMS behaviors, we calculated percentages of SMS talk out of P acts.

We used two versions of a socio-metacognitive sense-making coding scheme originally developed by Borge et al. (2018): the original version was used for reflections about the processes that occurred during their content-based discussion and a second version for the content-based discussion itself. Both original and modified rubrics are presented in Table 2. The original version was designed to code talk that occurred during reflection sessions to identify the extent to which teams engaged in socio-metacognitive sense-making activities during the reflective activity. It included process reporting, monitoring, reflecting, planning, and revising. Two trained coders

<table><caption>Table 2: Examples of SMS patterns by the participants</caption><thead><tr><td>SMS Pattern</td><td>Rubric Version</td><td>Example</td></tr></thead><tbody><tr><td>Reporting (RP)</td><td>Both rubrics</td><td>"We sure covered a lot." (MD)<br>A student reports her/his opinion about the collaboration quality without referring to concrete events or patterns from discussion or justifying her/his judgment.)</td></tr><tr><td>Process Monitoring (MO)</td><td>Both rubrics</td><td>"and we also evaluated trade-offs for some while comparing implications." (MD)<br>(A student demonstrates evidence of paying attention to the ongoing collective process by pointing out a specific activity/requirement that the team has done or needs to do.)</td></tr><tr><td>Process Reflection (RF)</td><td>Both rubrics</td><td>"I think the problem is that we read 2 different things." (MD)<br>(A student demonstrates evidence of reflecting on their ongoing collaborative discussion by pointing out a reason for why s/he thinks that particular incident happened.)</td></tr><tr><td>Process Planning (PL)</td><td>Original rubric</td><td>"To begin, we definitely need to work on time management. Our communication skills are sufficient when it comes to the subject matter, but we definitely need to get tasks done with a sense of urgency." (AR)<br>(A student demonstrates evidence of planning by unpacking the problem and proposing new goals.)</td></tr><tr><td>Regulation (R)</td><td>Modified rubric</td><td>"Let me play devil's advocate, since we need to think about the other side of the coin." (MD)</td></tr></tbody></table>
(A student demonstrates evidence of regulating ongoing activity reminding teammates about previously set process goals.)

Note: MD=Main discussion, AR=After-discussion reflection.

coded 23% of the total reflective discussion data using the original construct, with Kappa = .806; p < .001. The coders discussed and resolved disagreements and then one coder re-coded all the reflection data. A second version of the coding construct was created to code SMS talk that occurred during the content-based discussion. This new version included all the previous forms of talk plus a new category: regulation talk. Regulation talk identified socio-metacognitive strategies, including moves such as proposing or using a discussion strategy from guides we provided. Inter-rater reliability was checked on this new version on 24% data with Kappa = .725; *p* < .001. Two coders discussed and resolved disagreements and then one coder re-coded all the main discussion data.

## Selection of high, medium, and low performing teams

Average collaborative discussion quality scores were used to assess performance for each team (see Table 3). High, medium, and low performing categories each have four teams.

<table><caption>Table 3: Mean discussion quality scores of the teams (Lowest to Highest)</caption><thead><tr><th rowspan="2"></th><th colspan="4">Low Quality Discourse Teams</th><th colspan="4">Medium Quality Discourse Teams</th><th colspan="4">High Quality Discourse Teams</th></tr><tr><th>T9</th><th>T7</th><th>T1</th><th>T4</th><th>T3</th><th>T6</th><th>T8</th><th>T12</th><th>T5</th><th>T10</th><th>T11</th><th>T2</th></tr></thead><tbody><tr><th>M</th><td>19.80</td><td>20.20</td><td>21.40</td><td>21.80</td><td>23.40</td><td>23.40</td><td>23.40</td><td>23.80</td><td>24.00</td><td>24.40</td><td>24.60</td><td>25.40</td></tr><tr><th>SD</th><td>3.56</td><td>4.15</td><td>3.36</td><td>3.42</td><td>2.88</td><td>2.51</td><td>3.97</td><td>2.59</td><td>3.32</td><td>1.52</td><td>3.65</td><td>3.21</td></tr></tbody></table>

## Comparing team process interactions

We conducted one-way ANOVA to determine whether there was a significant difference in the average means of the amount of socio-metacognitive sense-making talk that occurred in low, medium, and high performing teams. For that, Levene's test for homogeneity of variances was found to be protected for the total percentage of SMS acts (F(2,9)= 1.23, *p*=.34). Shapiro-Wilk test was performed to ensure normality, and no significant value was identified; therefore, the normality assumption was met to perform ANOVA.

## Findings

### (RQ1) Patterns of socio-metacognitive sense-making (SMS) talk

On average, each team created 263.58 process-related turns over five discussion sessions, *SD*=120.75, *Min*=133, *Max*=521. Over half of these process-related turns, 54.54% (*SD*=13.03), were coded as SMS talk. Those that were not coded as SMS focused on sharing information about how they were using technology or other forms of social -off-task talk. All teams engaged in RP, MO, RF, and PL/R acts in at least one discussion session, but only 7 out of 12 teams demonstrated RV act (Table 4).

<table><caption>Table 4: The mean percentages of teams' SMS patterns</caption><thead><tr><th rowspan="2"></th><th colspan="2">Main Discussions</th><th colspan="2">After-Discussion Reflections</th><th colspan="2">Total</th></tr><tr><th>M (%) *</th><th>SD</th><th>M (%) **</th><th>SD</th><th>M (%) ***</th><th>SD</th></tr></thead><tbody><tr><th>Reporting (RP)</th><td>2.65</td><td>4.96</td><td>18.37</td><td>14.65</td><td>11.52</td><td>5.06</td></tr><tr><th>Process Monitoring (MO)</th><td>12.27</td><td>4.77</td><td>8.45</td><td>7.34</td><td>10.39</td><td>4.76</td></tr><tr><th>Process Reflection (RF)</th><td>0.19</td><td>0.45</td><td>5.44</td><td>2.63</td><td>3.45</td><td>1.66</td></tr><tr><th>Process Planning/Regulation (PL/R)</th><td>43.93</td><td>11.05</td><td>15.14</td><td>11.96</td><td>28.64</td><td>9.71</td></tr><tr><th>Process Revising (RV)</th><td>0.58</td><td>1.15</td><td>0.45</td><td>0.67</td><td>0.54</td><td>0.62</td></tr></tbody></table>

Notes: * Mean frequency percentage out of the total P frequency in main discussions for each SMS pattern
** Mean frequency percentage out of the total P frequency in after-discussion reflections for each SMS pattern
*** Mean frequency percentage out of the total P frequency for each SMS pattern

The findings suggested that the teams engaged in PL/R act the most. In the main discussion these acts occur as regulation acts focused on proposing a discussion strategy or new direction for conversation, whereas in the after-discussion reflections these acts are planning acts focused on identifying a strength or weakness, or proposing and evaluating goals and strategies for future discussions. As shown in table 4, the high frequency of PL/R acts are due to the high number of regulation acts that occurred in the main discussions. For example, during the second discussion, a team was discussing whether all members understood the course content. One team member, Bill, stated that the content was new to them so they did not have full understanding of it. Upon hearing that Bill had specific questions, another member, Juan, said: “... we had some questions there, Bill, you can start” (R act in Main Discussion; Team 2, Session 2).
Figuring out how to discuss topics deeply, while keeping to an agreed upon time limit was a common topic discussed during the after-discussion reflection sessions. Team 2 discussed this topic as part of their reflection. Bill said, “Although we got very in depth, I feel like we could get just as in depth if we focus the conversation more and have strict time framing.” Another member, Jill, added, “We need to figure out how to keep to the timeframe without cutting off something important” (PL Act, After-Discussion Reflection; Team 2, Session 2,). Juan then proposed a suggestion for improvement, “We can do anything! we will just have to weigh our questions maybe before we start the conversation and start there as a base.” Bill responded by saying, “Right maybe ask the longer ones first” (PL Act, After-Discussion Reflection).

## (RQ2) Differences between low, medium, and high-quality discourse teams

Looking at the relationship between SMS talks and team performance, when ranking teams by SMS acts, we saw that the frequency of SMS acts was closely tied to the quality of collaborative discourse (see table 5).

<table><caption>Table 5: Frequencies of teams' collaboration patterns (SMS Lowest to Highest for Each Performance Category)</caption><thead><tr><th rowspan="2"></th><th colspan="4">Low Quality Discourse Teams</th><th colspan="4">Medium Quality Discourse Teams</th><th colspan="4">High Quality Discourse Teams</th></tr><tr><th>T4</th><th>T7</th><th>T9</th><th>T1</th><th>T6</th><th>T8</th><th>T12</th><th>T3</th><th>T11</th><th>T5</th><th>T10</th><th>T2</th></tr></thead><tbody><tr><th>P</th><td>348</td><td>365</td><td>196</td><td>217</td><td>142</td><td>133</td><td>247</td><td>200</td><td>216</td><td>521</td><td>166</td><td>412</td></tr><tr><th>SMS*</th><td>25.57</td><td>41.10</td><td>43.88</td><td>50.23</td><td>47.89</td><td>57.89</td><td>61.54</td><td>67.00</td><td>60.19</td><td>65.45</td><td>66.27</td><td>67.48</td></tr></tbody></table>

Notes: * Total frequency percentage out of the total P frequency for each team
P=Process-related talk, SMS=Socio-metacognitive sense-making talk, T=Teams

Comparing values on Table 5 with those on Table 3 (teams' mean discussion quality scores), we observed that low quality discourse teams tended to engage in least SMS acts while high quality discourse teams tended to engage in most SMS acts. However, as illustrated in Table 5, there were some exceptions. A Pearson correlation analysis on SMS percentage means and mean performances showed a strong positive correlation ($r = .749, n = 12, p = .005$). Analysis of variance showed a significant difference in the SMS percentage means between different quality discourse teams, $F(2,9)=10.66, p=.00$. Tukey HSD indicated a significant difference in SMS percentage means between low ($M=40.20, SD=10.47$) and high-quality discourse teams ($M=64.85, SD=3.22$), and between low ($M=40.20, SD=10.47$) and medium quality discourse teams ($M=58.58, SD=8.05$). No significant difference was found between medium and high-quality discourse teams.

We conducted one-way ANOVA to determine whether there is a significant difference in the five SMS patterns ($n=5$) between the low, medium, and high-quality discourse teams; no significant differences were found. However, descriptive analysis of the percentage values between teams yielded interesting trends. The average percentage of PL/R act increased from low ($M=19.76, SD=5.12$) to medium performing teams ($M=32.09, SD=9.92$), and from medium to high performing teams ($M=34.09, SD=7.77$). In RV act: low-performing teams engaged less ($M=0.27, SD=0.31$) than medium-performing teams ($M=0.52, SD=0.37$), who engaged less than high-performing teams ($M=0.83, SD=0.98$). The average percentage of RF act in low-performing teams was smaller ($M=2.86, SD=1.21$) than that of medium-performing teams ($M=3.62, SD=2.65$), which was smaller than that of high-performing teams ($M=3.87, SD=0.91$). Figure 2 below summarizes the distribution of the SMS acts by performance categories.

<figure><img src="image_140.png" /><figcaption>Figure 2. SMS percentage means of the teams by performance categories.</figcaption></figure>
Discussion

In this paper we examined teams’ sense-making patterns in the main content-based discussions and during after-
discussion reflections, where teams made sense of their collaborative process. We also examined the relationship
between sense-making patterns and collaborative discourse quality. Our findings showed that all the teams
engaged in collective socio-metacognitive sensemaking (SMS) talk during content-based discussions and after-
discussion reflections. Further supporting the design of the original intervention, which aimed at pushing students
to engage in SMS talk as part of reflections for the purpose of getting them to figure out regulate activity during
content-based talk and then actually regulate activity. We also found a strong positive correlation between the
frequency of socio-metacognitive sense-making talk and the quality of collaborative activity.

These findings add to the growing body of research that indicate that supporting collective regulation of group interactions may be the key to enhancing the quality of collaborative processes (Kwon, Liu, & Johnson, 2014; Rogat & Adams-Wiggins, 2015). Prior research has suggested that students do not have the ability to monitor and regulate individual and collaborative activities (Borge & White, 2016; Kwon, Liu, & Johnson, 2014; Gabelica et al., 2014; Winne & Nesbit, 2009), and if not provided with sufficient amount of guidance, they will likely to develop dysfunctional collaborative habits (Borge et al., 2018; Kozlowski & Ilgen, 2006; Webb & Palincsar, 1996). The current study suggests that guiding students to develop sense-making skills to monitor and reflect their ongoing/past collaborative processes can help teams to developing sophisticated collaborative competencies. What is more, this work suggests it is possible to enhance collaborative processes without over-scripting collaboration as it occurs or creating inauthentic collaborative environments (Dillenbourg, 2002; Fischer et al., 2013). As such, this paper also extends what is known about the impacts of socio-metacognition on teams’ ability to improve the quality of collaborative discussion processes.

Our findings imply that scripting socio-metacognitive sense-making activities before and after collaborative activity can help teams to regulate that ongoing collaborative activity, thereby developing socio-metacognitive competence and optimizing their ongoing collaborative activities. However, it also underlines the need for a more blended approach to scripting that supports collaborative process both during and after they occur. Nonetheless, it is crucial to note that further research is needed to support these findings and to explore how after-discussion reflection impact students’ sense-making of their collaborative process, their socio-metacognitive development, and collaborative process improvement.

One limitation of this study is the low number of the groups for each quality discourse category. Further
studies are needed to see if the patterns observed in the study remain once the number of observations increases.
Furthermore, the focus of this study did not fully address impacts of emotions on social interactions, which were
also prevalent forms of process-based talk. Building on our theoretical framework, collaboration emerges as a
product of interactions among individuals (Stahl, 2006), which entails the exchange of ideas, knowledge,
emotions, and feelings (Järvenoja & Järvelä, 2009). Thus, further research is needed to examine the dynamics
between socio-metacognitive expertise, socio-emotional interactions, and the quality of collaborative discourse.

References

Baker, M., Andriessen, J., Lund, K., van Amelsvoort, M., & Quignard, M. (2007). Rainbow: A framework for analyzing computer-mediated pedagogical debates. *International Journal of Computer-Supported Collaborative Learning, 2*(2), 315-357.

Barron, B. (2003). When smart groups fail. *The Journal of the Learning Sciences, 12*(3), 307-359.

Borge, M., & Carroll, J. M. (2014, November). Verbal equity, cognitive specialization, and performance. In *Proceedings of the 18th International Conference on Supporting Group Work*(pp. 215-225). ACM.

Borge, M., & White, B. (2016). Toward the development of socio-metacognitive expertise: An approach to developing collaborative competence. *Cognition and Instruction, 34*(4), 323-360.

Borge, M., Ong, Y. S., & Rosé, C. P. (2018). Learning to monitor and regulate collective thinking processes. *International Journal of Computer-Supported Collaborative Learning, 13*(1), 61-92.

Bruner, J. (1999). Postscript: Some reflections on education research. In E. C. Lagemann & L. S. Shulman (Eds.), *Issues in education research: Problems and possibilities* (pp. 399-409). San Francisco: Jossey-Bass Publishers.

Chi, M. T. H. (1997). Quantifying Qualitative Analyses of Verbal Data: A Practical Guide. *Journal of the Learning Sciences, 6*(3), 271-315.

Dillenbourg, P. (2002). Over-scripting CSCL: The risks of blending collaborative learning with instructional design. In P. A. Kirschner (Eds.), *Three worlds of CSCL. Can we support CSCL* (pp. 61-91). Heerlen: Open Universiteit Nederland.
Fischer, F., Kollar, I., Stegmann, K., Wecker, C., Zottman, J., & Weinberger, A. (2013). Collaboration scripts in computer supported collaborative learning. In C. Hmelo-Silver, C. Chinn, C. Chan, & A. O'Donnell (Eds.), *The International handbook of collaborative learning* (pp. 403-419). New York, NY: Routledge.

Gabelica, C., van den Bossche, P., De Maeyer, S., Segers, M., & Gijselaers, W. (2014). The effect of team feedback and guided reflexivity on team performance change. *Learning and Instruction*, 34, 86-96.

Hogan, K. (1999). Thinking aloud together: A test of an intervention to foster students' collaborative scientific reasoning. *Journal of Research in Science Teaching*, 36(10), 1085-1109.

Järvelä, S., & Hadwin, A. F. (2013). New frontiers: Regulating learning in CSCL. *Educational Psychologist*, 48(1), 25-39.

Järvenoja, H., & Järvelä, S. (2009). Emotion control in collaborative learning situations: Do students regulate emotions evoked by social challenges. *British Journal of Educational Psychology*, 79(3), 463-481.

Kerr, N., & Tindale, S. (2004). Group performance and decision making. *Annual Review of Psychology*, 55(1), 623-655.

Kozlowski, S., & Ilgen, D. (2006). Enhancing the Effectiveness of Work Groups and Teams. *Psychological Science in the Public Interest*, 7(3), 77-124.

Kozlowski, S. W., Watola, D. J., Jensen, J. M., Kim, B. H., & Botero, I. C. (2009). Developing adaptive teams: A theory of dynamic team leadership. In S. Eduardo, G. F. Goodwin, & C. S. Burke (Eds.), *Team effectiveness in complex organizations: Cross-disciplinary perspectives and approaches* (pp.113-155). New York, NY: Routledge.

Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: A review of the research. *Computers in Human Behavior*, 19(3), 335-353.

Creswell, J. (2015). *Educational research: Planning, conducting, and evaluating quantitative and qualitative research*. New York, NY: Pearson.

Kwon, K., Liu, Y. H., & Johnson, L. P. (2014). Group Regulation and Social-Emotional Interactions Observed in Computer Supported Collaborative Learning: Comparison Between Good vs. Poor Collaborators. *Computers & Education*, 78, 185-200.

Nesbit, P. L. (2012). The Role of Self-Reflection, Emotional Management of Feedback, and Self-Regulation Processes in Self-Directed Leadership Development. *Human Resource Development Review*, 11(2), 203-226.

Rogat, T. K., & Adams-Wiggins, K. R. (2015). Interrelation between Regulatory and Socioemotional Processes within Collaborative Groups Characterized by Facilitative and Directive Other-Regulation. *Computers in Human Behavior*, 52, 589-600.

Salomon, G., & Globerson, T. (1989). When teams do not function the way they ought to. *International Journal of Educational Research*, 13, 89-99.

Stahl, G. (2006). *Group cognition: Computer support for building collaborative knowledge*. Cambridge, MA: MIT Press.

Webb, N. M., & Palincsar, A. S. (1996). Group processes in the classroom. In D. C. Berliner & R. C. Calfee (Eds.), *Handbook of educational psychology* (pp. 841-873). New York, NY: Macmillan.

Winne, P. H., & Nesbit, J. C. (2009). Supporting self-regulated learning with cognitive tools. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), *Handbook of metacognition in education* (pp. 259-277). New York, NY: Routledge.

Zimmerman, B. J., & Schunk, D. H. (2001). *Self-regulated learning and academic achievement: Theoretical perspectives* (2nd Eds.). Routledge.
# The Binary Replicate Test:
Determining the Sensitivity of CSCL Models to Coding Error

Brendan R. Eagan, University of Wisconsin–Madison, beagan@wisc.edu

Zachari Swiecki, University of Wisconsin–Madison, swiecki@wisc.edu

Cayley Farrell, University of Wisconsin–Madison, cefarrell@wisc.edu

David Williamson Shaffer, University of Wisconsin–Madison and Aalborg University, Copenhagen,
dws@education.wisc.edu

**Abstract:** The process of labeling, categorizing, or otherwise annotating data—or *coding* in the computer-supported collaborative learning (CSCL) literature—is a fundamental process in CSCL research. It is the process by which researchers identify salient properties about segments of CSCL data: what they are, what they contain, or what they mean. Coding, like all processes in research, is subject to error. To reduce the potential impact of coding error, CSCL researchers typically measure inter-rater reliability (IRR). However, there is no extant method to determine what level of IRR would invalidate a CSCL result or model. One way of assessing the potential impact of such inaccuracies is by conducting *sensitivity analyses*, which measure the *level of error* that would need to be present in the data to invalidate a given inference. This paper introduces a new method for conducting sensitivity analyses in CSCL: the Binary Replicate Test.

## Introduction

Inaccuracies in measuring collaborative learning in *computer-supported collaborative learning* (CSCL) environments can result in poor learning outcomes, inappropriate or ineffective pedagogical responses, misguided design decisions, or misallocation of scarce resources. In order to effectively *control* for the impact of measurement error, researchers need to be able to *measure* it. One general approach to assessing the impact of inaccuracies in data is through *sensitivity analyses*, which measure the *level of error* that would need to be present in the data to invalidate a given inference: that is, the extent to which the data could be altered until the original result becomes invalid (Frank & Min, 2007) (1). Here, we develop and test a method to apply this general strategy to the specific case of *coding error*.

In CSCL research, one of the largest sources of error involves *coding*. At its most basic level, coding is the process used to classify data: to assert that it should be labeled with a given construct or thought of as containing a given property (Chi, 1997; Shaffer, 2017). There are, of course, many sources of coding error. For example, hand coding performed by qualitative researchers is subject to individual bias, misinterpretation, or even just human fatigue—ask any researcher rating their 250th excerpt of data! Similarly, automated coding processes introduce some level of error, no matter how good the algorithm or approach. To reduce the potential impact of coding error, CSCL researchers typically measure inter-rater reliability (IRR) (Eagan, et. al. 2017; Hammer & Berland, 2014). While the common method of establishing IRR in CSCL research requires some methodological refinement (Eagan, et. al. 2017; Hammer & Berland, 2014; Shaffer, 2017), even when applied correctly, there is no extant method to determine what level of IRR (and thus what level of coding error) would invalidate a result or model.

In this study, we propose and provide an example of a method for conducting a Monte Carlo rejective test, a form of null hypothesis significance test, that empirically measures the impact of coding error on the results of any CSCL analysis. We consider the specific case of *binary coding*—the assertion of the presence or absence of a given construct for some observation—because this is a commonly-used approach to coding in CSCL research, and we refer to the proposed technique as a Binary Replicate Test (BRT). And in this initial paper on the BRT method, we apply the BRT approach using one specific modeling tool and one specific set of CSCL data. Our results suggest, however, that the BRT method has the potential to empirically determine thresholds for IRR measurement in CSCL analyses more generally.

## Theory

### Coding error

Binary coding is the process of assigning one of two possible values (typically 1 or 0) to pieces of data, reflecting the presence or absence of some property of interest in the data: for example, whether a student is asking a question in a particular turn or talk (1) or not (0). It follows that there are two types of error in binary coding. *Type I errors*
(false positives) are when an observation *does not* contain a given construct but is *coded* for the construct—for example, a turn of talk coded with a 1, indicating that the student is asking a question, is a Type I error if the student is, in fact, not asking a question. *Type II errors* (false negatives) are when an observation *does* contain a construct but is *not coded* for the construct—for example, if a turn of talk is coded with a 0, indicating the student was not asking a question, when she actually was.

In most CSCL settings, however, determining Type I and Type II errors is complex, because there is not necessarily a clear ground truth: two observers may see the same event in the data and code it differently. Researchers typically use measures of IRR, such as Cohen's kappa, to determine the overall rate of agreement—that is, some combined level of Type I and Type II errors between two raters, taking one rater as the standard to which the second rater is compared. The assumption—often implicit—in using IRR in this way is that the ground truth lies somewhere between the two sequences of 1s and 0s that each coder assigns to the data <sup>(2)</sup>.

Put another way, the IRR between two raters establishes an *upper bound* on the distance that either rater can be from ground truth—assuming that both raters are conscientious and understand the concept being coded, of course. Therefore, to assess the sensitivity of a given model to coding error, we need to determine the minimum level of an IRR statistic for which the model remains valid: that is, some level at which we can change 1s to 0s and 0s to 1s without altering the result of a given analysis conducted on the coded data.

In the case of IRR, guidance for establishing such thresholds is scarce. Many statistics have no commonly accepted threshold for what constitutes an acceptable level of agreement. For example, researchers refer to levels of Precision, Recall, or F statistics as being “high” without ever defining what constitutes a “high” level for that statistic (Schwarm, & Ostendorf, 2005). For other metrics, there are commonly used levels of acceptable agreement that have no statistical or inferential foundation. For example, Viera & Garrett (2005) suggested that “substantial agreement” for Cohen’s kappa is above 0.61, and that has been used as a standard in many fields (Andrews, Leonard, Colgrove, & Kalinowski, 2011; Goh et. al., 2008; Lasorsa, Lewis, & Holton, 2012), despite the fact that neither they, nor Cohen, provided a justification for that choice.

## Thresholds of significance

This use of IRR, is, of course, a form of sensitivity analysis. Sensitivity analyses in general quantify the conditions under which the validity of a result could be called into question by establishing a *threshold*: the point at which a particular level of error would invalidate an inference based on that result. Such analyses are often conducted using *Monte Carlo* (MC) studies (Eagan et. al. 2017; Harwell, 1992). MC studies assess the performance and reliability of statistical tests in educational and psychological research by creating an *empirical sampling distribution*. That is, they create a large number of simulated datasets and calculate a test statistic for each one. The performance of the test can then be evaluated under different assumptions about the properties of the population from which the samples are drawn, or based on variations on the sampling procedure itself.

For example, Eagan and colleagues (2017) used MC studies to demonstrate the unacceptably high Type I error rates associated with using commonly applied methods for IRR measurement in CSCL, learning analytics, and related fields. Similarly, Harwell, Rubinstein, Hayes & Olds (1992) used MC studies to show that the Welch test, a variant of ANOVA, has inflated Type I error rates and lower power when used with non-normal distributions than with normal distributions. In the case of coding error, researchers can create simulated code sets by introducing different levels of error to an *actual* dataset to examine the performance of a model under different error conditions.

In this paper we conduct a sensitivity analysis on the results of one analytic technique used in CSCL and learning analytics, epistemic network analysis, but the BRT method is agnostic to the specific analytic approach.

## Model significance

Analytic models can be used to detect differences between groups, such as contrasting treatment and control, comparing different teams, highlighting instructor differences, and so on. In addition, analytic models can be used to assess learners at an individual level, which is useful for, among other things, providing feedback, grading, and selecting interventions. As a result, researchers need methods for establishing the accuracy of a learning analytic model for assessing both group differences and individual learners.

## MC rejective methods

Researchers can establish the robustness of analytic results using *rejective methods*. Using a criterion for rejecting a null hypothesis, such as employing p-value thresholds in hypothesis testing, is an example of a *rejective method*. Classically, when researchers observe a p-value under a critical value—often $\alpha < 0.05$—they reject the null hypothesis under the assumption that they would be doing so incorrectly 5% of the time if the null hypothesis were indeed true. While many rejective methods rely on a theoretical model, such as parametric tests using p-
values, others rely on empirical simulations to establish a criterion for rejecting a null hypothesis. For example, the MC rejective method Shaffer's rho generates an empirical null hypothesis distribution used to determine the likelihood of seeing a specific IRR measurement given the null hypothesis—that the actual IRR was below a threshold of interest (Shaffer, 2017). Similarly, an MC rejective method can be used to test a given analytical model's robustness to coding error when measuring group or individual differences.

## ENA

One technique that has been frequently used to analyze coded data produced in CSCL environments is *epistemic network analysis* (ENA) (see Shaffer, 2017, and references). To do so, ENA takes interaction data coded for elements of complex thinking, collaboration, learning, or any other phenomena of interest and creates weighted network models of the connections between those elements for individuals or groups.

As with any learning analytic technique applied to coded data, errors in coding can significantly affect statistical results and interpretations. In what follows, we present a study on the impact of coding error on one particular ENA result. However, we emphasize that our goal is less to show a method for conducting sensitivity analyses on one particular modeling technique than it is to present this as an example of a method that can be used to conduct sensitivity analyses on any CSCL or learning analytic result generated from coded data.

Specifically, we ask: *How much coding error can be introduced to a dataset before we are no longer confident in the statistical significance (at the group level) and accuracy (at the individual level) of a given CSCL result?* We address this research question by using a MC rejective method to examine the impact of coding error on one validated ENA result.

## Methods

### Gold-Standard ENA model

*General approach.* Our general approach to developing and testing a methodology for conducting sensitivity analyses was to create a “Gold-Standard” result: that is, an original, statistically significant result using coded data whose sensitivity we wanted to test. To do so we created an ENA model using a well-studied dataset (Arastoopour et. al. 2016; Chesler et al., 2015). We then evaluated the effects of making perturbations to the coded dataset upon which the Gold-Standard result was based. Specifically, we measured the sensitivity of the ENA model to coding error by randomly introducing coding error into the dataset used to create the Gold-Standard model, and then looked to see the level of error at which the original result was no longer statistically significant or accurate at the individual level.

*Data source.* To create our Gold-Standard result, we analyzed the discourse of students in the engineering virtual internship *RescuShell* (Chesler et al., 2015). In *RescuShell*, students work in project teams to design robotic exoskeletons for use by rescue workers in disaster situations. The Gold-Standard model has two conditions: (a) students in the first condition (relative novices, hereafter referred to as Novices) were participating in an engineering virtual internship for the first time; (b) students in the second condition (relative experts, hereafter referred to as Experts) had previously participated in a different engineering virtual internship.

*Model.* Previous research (Arastoopour et. al. 2016) found differences in the discourse patterns of Novices and Experts that were both statistically and interpretively significant (that is, meaningful). Chesler and colleagues (2015) found that Novices made connections mostly among basic skills and knowledge while Experts made additional connections with epistemological elements of engineering which are indicative of complex problem solving in engineering. To create the Gold-Standard model we replicated this previous research by applying Epistemic Network Analysis (ENA; Shaffer, 2017; Shaffer, Collier, & Ruis, 2016; Shaffer & Ruis, 2017) to data from *RescuShell* using the ENA Web Tool (version 1.5.2) (Marquart, Hinojosa, Swiecki, Eagan, & Shaffer, 2018).

The ENA technique has been described in detail elsewhere (Shaffer, 2017; Shaffer, Collier, & Ruis, 2016), including in previous papers presented at CSCL and ICLS (Collier, Ruis, & Shaffer, 2016; Csanadi et. al., 2017; Siebert-Evenstone et al. 2016; Swiecki, & Shaffer 2018). Briefly, ENA models connections in discourse as a network graph where nodes correspond to the codes, and edges reflect the relative frequency of co-occurrence, or connection, between two codes. These networks can be further compared both quantitatively and qualitatively to analyze and compare different discourse patterns in CSCL environments.

We are not providing the details of the data and coding scheme here, both because they are available in Chesler et al. (2015) and because the specific codes and details of the model are not the primary issue in testing the sensitivity to of the model to coding error. However, briefly: for the Gold-Standard result, we defined the units of analysis as all lines of data associated each individual student, and used a moving window to construct a network model for each line in the data, showing how codes in give line are connected to codes that occur within
the recent temporal context (Siebert-Evenstone et al., 2017), defined as 7 lines (each line plus the 6 previous lines) within a given conversation. The resulting networks were aggregated for all lines for each unit of analysis in the model. In this model, we aggregated networks using a binary summation in which the networks for a given line reflect the presence or absence of the co-occurrence of each pair of codes within the recent temporal context of the line. Our model included the following codes: CLIENT AND CONSUMER REQUESTS, COLLABORATION, DATA, DESIGN REASONING, PERFORMANCE PARAMETERS, and TECHNICAL CONSTRAINTS. We defined conversations as all lines of data associated with a single group in a single activity in each condition, Experts or Novices. We projected the resulting networks using a *means rotation*: a dimensional reduction technique that places the greatest differences between conditions on the x-axis in the ENA space.

In the resulting network graphs, the nodes were placed in ENA space so as to minimize the sum of the distances for each student between the point representing the student's network in the projected space and the centroid of the student's network graph. The result is two coordinated representations for each student: (1) a plotted point, which represents the location of that student's network in the low-dimensional projected space, and (2) a weighted network graph, which shows the pattern of connections the student made that led to his or her location in the ENA space. Because of this co-registration, the positions of the network graph nodes—and the connections they define—can be used to interpret the dimensions of the projected space and explain the positions of the plotted points in that space. It is also possible to interpret the differences between two groups by constructing their mean networks and subtracting them. The resulting *difference network* shows the connections that are stronger for each group compared to the other.

Our model had co-registration correlations of 0.99 (Pearson) and 0.99 (Spearman) for the first dimension and co-registration correlations of 0.99 (Pearson) and 0.99 (Spearman) for the second dimension. These measures indicate that there is a strong goodness of fit between the visualization and the original model, and thus indicates that the model has interpretive validity: we can use the network graphs to interpret the statistical results.

*Test result.* To test for discourse pattern differences between Novices and Experts, we applied a two-sample t-test assuming unequal variance to the mean location of points in the projected ENA space for each condition. Figure 1 shows the plotted points (dots), means (squares), and confidence intervals (dashed boxes) for the Novices (in red) and the Experts (in blue).

<figure><img src="image_141.png" /><figcaption>**Figure 1.** Plotted points (left) and difference network (right) for the Gold-Standard ENA result.</figcaption></figure>

Figure 1 also shows the difference network that can be used to interpret statistical differences between conditions. Where the difference network is red, the Novices had stronger connections, and where the network is blue, Experts made stronger connections.

Along the x axis, a two sample t-test assuming unequal variance showed the discourse pattern of Novices (mean = -0.37, SD = 0.44, N = 27) was statistically significantly different at the alpha=0.05 level from those of Experts (mean = 0.36, SD = 0.49, N = 28; t(52.78) = -5.76, p < 0.01, Cohen's d = 1.55) (see Figure 1).

This statistically significant result (Figure 1, left) can be interpreted using the subtracted network graph on the right side of Figure 1. The Novices made more connections between TECHNICAL CONSTRAINTS and COLLABORATION, TECHNICAL CONSTRAINTS and DATA, COLLABORATION and DATA and COLLABORATION and PERFORMANCE PARAMETERS when compared with the Experts. The Experts made more connections between
PERFORMANCE PARAMETERS, and DESIGN REASONING than Novices. While Experts did make connections to COLLABORATION, more of their discourse focused on the relationships between PERFORMANCE PARAMETERS, DATA, and DESIGN REASONING; in other words, they were more focused on the core elements of engineering design. This result aligns with previous expert-novice comparisons from engineering virtual internships (Arastoopour et. al. 2016; Chesler et al., 2015).

Our Gold-Standard model thus includes two components that we tested in this analysis: (a) the statistically significant difference between Experts and Novices in that model, and (b) the specific locations of individual students relative to the model.

## Monte Carlo approach

For our BRT analysis, we created 5,000 simulated datasets at each of the different levels of coding error we examined (5%–10%, inclusive, in 1% increments) (3). To create each individual simulated dataset, at each level of coding error, we randomly introduced error at that level to the Gold-Standard dataset. All other aspects of the subsequent ENA model creation process where the same (4).

In the original data, codes are represented as a 1 or 0, indicating the presence or absence of a particular construct, for each turn of talk as students collaborated in groups of four (5). We introduced coding error as follows: (a) for each code, we randomly selected a percentage of lines equal to the percent error rate; (b) we changed the coding of each line chosen, changing its value either from 1 to 0 or 0 to 1; and (c) we repeated this process for each code in the dataset.

## Sensitivity analyses

*RQ1: How much coding error can be introduced to a dataset before more that 5% of the 5,000 ENA simulated models do not find a statistically significant difference between the Novice and Expert conditions?*

To answer research question 1, at each level of error: (1) we produced an ENA model for each of the 5,000 simulated datasets; (2) for each of these models we conducted the same t-test used to analyze our Gold-Standard model; (3) we calculated the number of models where the t-test *did not* show a statistically significant difference between Novices and Experts; (4) we created a 95% confidence interval on the estimate of the number of models that were not statistically significant. This tested the null hypothesis at each level of error: for example, at 6% introduced error, the null hypothesis was $H_0$: The difference between Novices and Experts is not significant at 6% coding error. If the confidence interval for the number of non-significant tests was below 5%, we rejected the null hypothesis and concluded that, with $\alpha$ = 0.05, the group difference in the Gold-Standard result was robust to that level of introduced error.

*RQ2: How much coding error can be introduced to a dataset before the average correlation between the plotted points from the 5,000 simulated ENA models and the plotted points from the Gold-Standard model falls below 0.80?*

To assess the extent to which coding error affects the accuracy of assessing individual students, at each level of error: (1) we calculated the correlation between plotted points in the Gold-Standard model—which represent the results of the ENA analysis for each student—and the plotted points in each simulated ENA model; (2) we computed the average correlation between plotted points in the Gold Standard model and each simulated dataset; (3) we created 95% confidence intervals for the average correlations. This tested the null hypothesis at each level of error: for example, at 6% introduced error, the null hypothesis was $H_0$: The positions of students in is not different from Gold-Standard model with 6% coding error. There is no universally agreed upon threshold for significant correlations; some researchers propose 0.70–0.90 as a highly positive correlation (Hinkle, Wiersma, & Jurs, 2003), while others consider correlations of 0.80–1.00 very strong (Evans, 1996). We choose a conservative threshold of correlations greater than or equal to 0.80. If the lower bound of the 95% confidence interval was greater than 0.80, we rejected the null hypothesis and concluded that, with $\alpha$ = 0.05, the Gold-Standard result preserved the positions of students in the model at that level of introduced error.

## Results

The left side of Figure 2 shows the percentage of tests in the sensitivity analysis that were not statistically significant at each level of introduced coding error we tested. The upper bound of the confidence intervals is below 0.05 at error rates up to 8%; however the upper bound of the confidence interval is above 0.05 at a 9% error rate. Thus the group difference in the Gold-Standard result is robust up to an 8% rate of coding error.
<figure><img src="image_142.png" /><figcaption>**Figure 2.** BRT results group difference sensitivity (left), individual level accuracy (right). For each graph, if the 95% confidence interval is at all contained by the shaded area, we fail to reject the corresponding null hypothesis.</figcaption></figure>

The right side of Figure 2 shows the average correlation between the plotted points of the models with introduced error and the plotted points from the Gold-Standard model at the different levels of coding error. Because of the large number of simulated data sets (5000) at each level of error, the 95% confidence intervals are too small to be seen on the graph. The upper and lower bound of the confidence interval for error rates up to 6% are above 0.80; however; at a 7% error rate 7%, the confidence interval falls below 0.80. Thus, accuracy of individual measurements of the Gold-Standard model is robust to 6% error.

Discussion

Our results thus show that difference between the means of two groups of students in the result we tested was robust to 8% error, but the accuracy of the model for individual students was only robust to 6% error. We thus conclude that the BRT suggests the result overall was robust up to 6% error.

While this analysis was conducted with only one set of CSCL data and using only one CSCL modeling approach, we believe that our analysis provides an example of how the BRT method for conducting sensitivity analyses can be used in CSCL research. In particular, we argue that the BRT approach can be used to conduct a sensitivity analysis on any model that depends on binary-coded data. That is, BRT provides a method that could be used to measure how robust CSCL results are to coding error.

Measuring the sensitivity of CSCL results to coding error in this way would provide a useful tool for establishing confidence in CSCL models. For example, if there are two conflicting CSCL results and the first was robust up to 1% coding error and the other was robust up to 8% coding error, we might have more confidence in the more robust result. More important, however, the BRT approach potentially provides a method for determining what level of coding reliability would be required to draw conclusions from a model—that is, a way to determine empirically the appropriate IRR threshold for a given result.

There is, of course, more work that will need to be done to make it possible to use BRT in this way. In particular, this study has several clear limitations. First, while this study reports on the robustness of (a) discriminating between two groups and (b) the accuracy of individual assessment, our analysis did not investigate how coding error affected the *interpretation* of the model. While it is beyond the scope of this paper, we have conducted BRT analyses for the interpretation of this model. Briefly, ENA models are interpreted using the positions of the nodes in the network graphs. In a future paper we will show that the average correlation between ENA node positions from the Gold-Standard model and the node positions from each simulated ENA model was above our threshold of 0.80 up to a 10% error rate. This suggests that that model interpretation was robust to 10% introduced error, which aligns with previous methodological analysis of ENA (Ruis et. al. 2018).

A second limitation of this work is that we did not systematically account for properties of the codes, such as code frequency, which may impact sensitivity. Similarly, when researchers code data, errors may be systematic, rather than random. In future work we plan to account for these issues in the BRT method by
independently specifying rates of Type I and Type II coding errors. Third, we used only one model to explore the BRT method, and we deliberately chose a result with a large effect size. In future work we plan to test the BRT method using other results with a range of effect sizes, as well as other modeling techniques used in CSCL research. Finally, we used percent agreement as our IRR metric, which is a problematic measure (Cohen, 1960; Shaffer, 2017). In future work, we plan to base our error rates on more precise measures of IRR, including Cohen's kappa, precision, recall, and F statistics.

Despite these clear limitations, the results presented here suggest that the BRT method is a promising method for empirically deriving IRR thresholds.

## Endnotes

(1) In statistics, sensitivity analyses often focus on bias, or systematic error, but for the purposes of this study we are focusing on the effects of random error. If a method is shown to be negatively impacted by random error, it will most likely be even more negatively impacted by systematic bias. It follows that focusing on random error is a more conservative approach for this type of sensitivity analysis.

(2) Note that this approach does not require that an actual ground truth exists; only that we can say something about likely rates of disagreement that exist about a coding process that tries to locate it. This is an important conceptual issue, but beyond the scope of the current paper.

(3) We also created simulated datasets at coding error levels from 1% - 5% and above 10% but in this study we are only reporting where the sensitivity analysis demonstrated a change statistically significant impact of coding error.

(4) We projected each ENA model into our Gold-Standard ENA space by using the same Eigen vectors for the ENA rotation matrix in the dimensional reduction step of the ENA process. This resulted in the node positions for all of the ENA models in the sensitivity analysis to be identical. The only thing that varied was the coding and resulting connection strengths and plotted point locations for each unit in each ENA space. In other words, the creation of each of the ENA sets based on datasets containing introduced coding error was identical.

(5) ENA can handle non-binary or weighted data, but this study focused on binary ENA models.

## References

Arastoopour, G., Shaffer, D.W., Swiecki, Z., Ruis, A.R., & Chesler, N.C. (2016). Teaching and assessing engineering design thinking with virtual internships and epistemic network analysis. *International Journal of Engineering Education, 32*(3B), 1492–1501.

Andrews, T. M., Leonard, M. J., Colgrove, C. A., & Kalinowski, S. T. (2011). Active learning not associated with student learning in a random sample of college biology courses. *CBE—Life Sciences Education, 10*(4), 394–405.

Chi, M. T. (1997). Quantifying qualitative analyses of verbal data: A practical guide. *The journal of the learning sciences, 6*(3), 271–315.

Chesler, N. C., Ruis, A. R., Collier, W., Swiecki, Z., Arastoopour, G., & Shaffer, D. W. (2015). A novel paradigm for engineering education: Virtual internships with individualized mentoring and assessment of engineering thinking. *Journal of biomechanical engineering, 137*(2), 024701.

Cohen, J. (1960). A coefficient of agreement for nominal scales. *Educational and Psychological Measurement 20*(1): 37–46.

Collier, W., Ruis, A.R., & Shaffer, D.W. (2016). Local versus global connection making in discourse. *Paper presented at the 12th International Conference of the Learning Sciences*. Singapore.

Csanadi, A., Eagan, B., Shaffer, D. W., Kollar, I., & Fischer, F. (2017). Collaborative and individual scientific reasoning of pre-service teachers: New insights through epistemic network analysis (ENA). In B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (Eds.), *Making a difference: Prioritizing equity and access in CSCL: 12th International Conference on Computer-Supported Collaborative Learning* (Vol. I, pp. 215–222).

Eagan, B., Rogers, B., Serlin, R., Ruis, A. R., Arastoopour Irgens, G., Shaffer, D. W. (2017). “Can We Rely on Reliability? Testing the Assumptions of Inter-Rater Reliability,” *Making a Difference: Prioritizing Equity and Access in CSCL: 12th International Conference on Computer-Supported Collaborative Learning*, eds. B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (2017), II:529–532.

Evans, J. D. (1996). *Straightforward statistics for the behavioral sciences*. Brooks/Cole.

Frank, K., & Min, K. S. (2007). 10. Indices of Robustness for Sample Representation. *Sociological Methodology, 37*(1), 349–392.

Goh, N. S., Desai, S. R., Veeraraghavan, S., Hansell, D. M., Copley, S. J., Maher, T. M., ... & Bozovic, G. (2008). Interstitial lung disease in systemic sclerosis: a simple staging system. *American journal of respiratory and critical care medicine, 177*(11), 1248–1254.
Harwell, M. R. (1992). Summarizing Monte Carlo Results in Methodological Research. *Journal of Educational Statistics*, **17**(4), 297–313.

Hinkle DE, Wiersma W, Jurs SG. Applied Statistics for the Behavioral Sciences. 5th ed. Boston: Houghton Mifflin; 2003.

Marquart, C. L., Hinojosa, C., Swiecki, Z., Eagan, B., & Shaffer, D. W. (2018). Epistemic Network Analysis (Version 1.5.2) [Software]. Available from http://app.epistemicnetwork.org

Lasorsa, D. L., Lewis, S. C., & Holton, A. E. (2012). Normalizing Twitter: Journalism practice in an emerging communication space. *Journalism studies*, **13**(1), 19-36.

Ruis, A.R., Siebert-Evenstone, A.L., Pozen, R., Eagan, B., & Shaffer, D.W. (2018). A method for determining the extent of recent temporal context in analyses of complex, collaborative thinking. In Kay, J. & Luckin, R (Eds.) *Rethinking Learning in the Digital Age: Making the Learning Sciences Count, 13th International Conference of the Learning Sciences (ICLS), III*, (pp. 1625–1626).

Schwarm, S. E., & Ostendorf, M. (2005, June). Reading level assessment using support vector machines and statistical language models. In *Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics* (pp. 523-530). Association for Computational Linguistics.

Shaffer, D. W. (2017). Quantitative ethnography. Madison, WI: Cathcart Press.

Shaffer, D. W., Collier, W., & Ruis, A. R. (2016). A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. *Journal of Learning Analytics, 3*(3), 9–45.

Siebert-Evenstone, A.L., Arastoopour Irgens, G., Collier, W., Swiecki, Z., Ruis, A.R., & Shaffer, D.W. (2017). In search of conversational grain size: Modelling semantic structure using moving stanza windows. *Journal of Learning Analytics, 4*(3), 123–139.

Swiecki, Z. & Shaffer, D.W. (2018). Toward a taxonomy of team performance visualization tools. In Kay J. & Luckin, R (Eds.) *Rethinking Learning in the Digital Age: Making the Learning Sciences Count, 13th International Conference of the Learning Sciences (ICLS), III*, 144–151.

Viera, A.J., & Garrett, J.M. (2005). Understanding interobserver agreement: The kappa statistic. *Family Medicine, 37*(5), 360-363.

## Acknowledgements

This work was funded in part by the National Science Foundation (DRL-1661036, DRL-1713110), the Wisconsin Alumni Research Foundation, and the Office of the Vice Chancellor for Research and Graduate Education at the University of Wisconsin-Madison. The opinions, findings, and conclusions do not reflect the views of the funding agencies, cooperating institutions, or other individuals.
# Developing Relationships, Changing Participation:
## Computational Identity

Melissa Gresalfi, Vanderbilt University, melissa.gresalfi@vanderbilt.edu

Amanda Bell, Vanderbilt University, amanda.m.bell@vanderbilt.edu

Alexandria T. Cervantes, California University Monterey Bay, alecervantes@csumb.edu

**Abstract:** This paper examines a single case of an African American boy, Eric, who enrolled in a free week-long summer camp called *Art and Coding*. We consider how his participatory identity developed in relation to the design of tasks, the students and teachers with whom he interacted, and the tools that he leveraged. With the goal of better understanding the relationship between participation and the development of beliefs about oneself, we then connect our analyses with the students' self-reflections in interviews. For Eric, a novice programmer whose passion is art, the experience of being positioned routinely as a knowledgeable explorer supported him to feel capable of resolving problems and attaining goals that he set for himself, even as he encountered challenges and frustration.

## Introduction

As Computer Science (CS) becomes increasingly prevalent in K-12 contexts, and given what we know about participation with CS in higher education and beyond, there is a need to consider not only what students have opportunities to learn about CS, but also what kinds of identities students are developing in relation to the domain. As the subject of significant attention in STEM domains, identities are seen as inextricable from learning, decision-making, and persistence. However, because CS has, until recently, primarily been a discipline reserved for higher education, we know little about the ways that students develop relationships with the domain across the age span. We do know that the proportion of women in computing jobs has actually decreased since 1990, down to 25% in 2015 (NCWIT, 2017), and just 7% of workers in computing in 2014 identified as Black and 7% as Hispanic (Beckhusen, 2016). These patterns mirror those seen in Mathematics, a discipline to which much time has been devoted to better understand how to change persistent patterns of course taking, major selection, and career choice. Drawing from research in mathematics education, we can predict that CS, when introduced to students before Advanced Placement courses, could also have wildly disparate influences on learning, enjoyment, persistence, and identity, depending on how it is taught. Using the assumptions and frames developed in mathematics education as our starting point, we explore how situated identities can develop in relation to programming, one skill in CS. In this paper, we consider a single case of an African American boy, Eric, who enrolled in a free week-long summer camp called *Art and Coding*. We consider how his participatory identity developed in relation to the design of tasks, the students and teachers with whom he interacted, and the tools he leveraged. With the goal of better understanding the relationship between participation and the development of beliefs about oneself, we then connect our analyses with the students' self reflections in interviews.

## Theoretical framework

Identity has been conceptualized in many ways, some of which highlight narratives and beliefs (Larnell, 2016; Leyva, 2016; Sfard & Prusak, 2005), and some that highlight activity and behavior (Hand & Gresalfi, 2015; Nasir, 2002). There are good reasons to call both of these things “identity,” as the use of the same word helps to emphasize the interconnectedness in what we do, how others recognize what we do, and the stories we tell ourselves and others about ourselves. However, there is also utility in making distinctions between behavior and beliefs, particularly when it comes to making sense of the role that contexts play in shaping both. In this work we emphasize questions about how participation shifts over time, and then connect this to the ways that students talk about or make sense of those shifts.

Focusing on participatory identities as they are constructed in settings can contribute to our emergent understanding of the strategies and approaches that are useful in teaching students about computer science and computational thinking. To do so we draw on conceptions of situated identities that highlight how people participate with a set of practices (Hand & Gresalfi, 2015; Nasir, 2002). Early work on identity from a situative perspective sought to document students’ participation in relation to a set of practices, influenced by Lave & Wenger’s (1991) claim that identity and learning are intertwined. Analyses of identity therefore require documenting how students are participating in relation to a set of classroom practices, and how those practices serve to create opportunities for identity development. These analyses are tightly coupled with studies of
classroom norms and practices and focus on interaction, with students as the unit of analysis (Grant, Crompton, & Ford, 2015; Langer-Osuna, 2015; Sengupta-Irving & Enyedy, 2015).

We draw on this framing of identity with the goal of better understanding how student identities are co-produced through their experience with the domain. We see students’ decisions not to continue with a subject or not to pursue a major as co-produced through their experiences of the domain, and the resulting ways they view themselves in relation to the domain. Too often, differences in course taking are conceptualized as failures of confidence, of preparation, or of imagination. But research on identity suggests that continuing to persist with activities that are damaging come at great significant cost, and students are wise to bow out. Thus, a better understanding of how activities produce this damage—and how we might be able to change it—is essential to increasing participation in the domain in the long term.

## Designing to support identities in computer science

Although little research has focused on participatory identity in relation to CS, research from K-12 contexts have documented several design characteristics of CS learning environments that support the development of productive elements of identity, including communities, mentors and role models, collaborative work, and programming contexts like stories or games. First, research suggests that representation matters, both in terms of the content or models in activities, as well as through the images and objects in the room. For example, in *Digital Youth Divas*, researchers designed characters to imitate actual middle school girls with a variety of interests, body types, and stories (Pinkard, Erete, Martin, & McKinney de Royston, 2017). The *Digital Youth Divas* program also demonstrated the value of role models, whether real or fictional, for interest and personal identity construction. The relatable characters and situations offer ideational resources to support girls’ identification with CS. Likewise, even reducing stereotypical objects in computing classrooms (e.g. replacing Star Wars posters, electronic parts, and tech magazines with art, plants, and general magazines) can increase girls’ sense of belonging and interest in a high school computing course without lowering boys’ existing interests (Master, Cheryan, & Meltzoff, 2016).

It is not only *who* students see however, but also *what they can do* that seems to support productive engagement and identity development. Highlighting the social nature of computing in environments like Scratch, with its online community, can create positive, gender-inclusive educational experiences for newcomers (Resnick et al., 2009). Likewise, teaching CS through programming stories, dances, and games can connect to students’ interests and create gender-inclusive learning environments (Daily et al., 2014) that can increase the time spent persisting on programming projects (Kelleher, 2008).

## Context of the study

Data for this paper comes from a free five-day summer camp for middle school students, held in a southeastern U.S. city. The camp was titled “Code Your Art,” and was advertised as involving computer programming and art. There were two classrooms during the camp, with 16 students in each class. Each class was co-caught by two teachers. The teachers taught elementary or middle school mathematics during the regular school year. There were also six researchers and two teaching assistants present during the camp, rotating among the classrooms as needed.

The goal of the camp was two-fold. First, the project was funded to explore how mathematical thinking might co-develop or connect with the algorithmic thinking that is central to programming. Second, we wanted to understand how the design of tasks and activities invite participation from students who might not see themselves as programmers or have any prior experience with programming. To meet both needs, we designed a set of activities that were sufficiently open to be seen by students as spaces for design and self-expression, but sufficiently constrained to make algorithmic thinking useful. Image design, with its attunement to position and movement, was an ideal fit for that goal. Complimenting the focus on design was the appeal of deliberately naming and emphasizing art in the camp as a signal to a broad range of students that this camp was intended for them, and mindful of the ways programming, building, and engineering often signal masculinity (Butler, 2002; Master et al., 2016).

The five days ultimately included a variety of activities involving computers, physical materials, and embodied activities. The programming activities used NetLogo, a multi-agent environment developed by Uri Wilensky and his colleagues (Wilensky, 1999). We chose NetLogo because it had functionality to import and edit images and included multiple, accessible ways to think about modifying images (thinking about patches and turtles, the latter offering opportunities for dynamic animation). Because NetLogo uses text-based language for programming, we developed a set of introductory models that included buttons that allowed students to explore possible features and functions. Such buttons can be opened so that the code is shared, meaning that students can both explore what a button does and how it does it. During the camp students edited models, debugged existing models, and created projects from empty models, and then shared their work to a digital gallery that was curated
on a camp website. They also wrote daily blog posts and shared them on the camp website. Other activities included building with Perler beads and acting out programs with their bodies.

## Methods

For this first analysis, we focus on one student from the camp. Of course, this single case does not serve as a model for what happened to all students, nor is that the purpose of the analysis. Single case studies can be helpful when trying to understand a phenomenon in some depth; here, we seek to better understand how the activities of the camp served to position the student relative to content and to others. The focal student, Eric, was selected for further analysis because although he appeared to be engaged throughout the entire camp, he had no prior experience with computer programming, and he signed up for the camp because of his love of art. Eric sat in the same seat all days of the camp, primarily interacting with only one boy, Kyle. He tended to be quiet and focused, but was quick to share his thinking when the class was discussing ideas associated with art and artists. Eric was a rising 7th grader at the school where the camp was held, and he expressed that he enjoyed the camp and attended the camp all five days.

Multiple forms of data were collected during the camp; for this paper we rely primarily on screen captures of the students' programming, observation notes that were taken during the camp, and interviews that were conducted one-on-one at the beginning and end of the camp. Interviews were semi-structured, and the protocol focused on initial reactions to and ideas about the camp, their reflection about their experiences during the camp, and how they see themselves and others within computer science. To analyze these data, we first reviewed the screen capture software taken of Eric, with the goal of attempting to understand his approach to programming. We categorized his work by attending to the ways he was positioned relative to the activity, for example by noting how he explored the software, how he refined and changed his designs as they developed, and whether he persisted in the face of challenge. We also looked for the ways that Eric was positioned relative to others, for example whether he asked for help, was asked for help by peers, or made comments about himself or others that indicated some sense of relative ability or skill level.

## Findings

In what follows, we describe characteristics of Eric's shift in participation with the tools and resources over time, and also how his thinking about himself (or more typically, activities) changed over time.

### Positioning with respect to the activities: From exploration to intention

From the very first day of camp, Eric spent a long time exploring and modifying the NetLogo models he was given, and experimenting with their functionality in relation to the images he chose to use for his projects, which primarily involved logos and images of basketball. However, his exploration at the beginning of the camp was less obviously guided by the kinds of disciplinary understandings that characterized his later work.

#### Day 1

In one of the first models students saw, they were asked to select an image from a wide collection created by the research team, and consider how they might want to modify it in a way that was interesting to them. The instructions were intentionally open with respect to what interesting might mean, or how the image might be changed. The initial model included multiple buttons that operated on an image using color (changing all colors, changing some colors, changing colors randomly, etc.) These buttons had different functions but could also relate to one another. In creating buttons that were interactive, we sought to offer a resource for students to inquire into how the models worked in general, and how modifications to images could be made that built on one another.

As an example of how Eric's exploration changed, we examined his use of a *variable slider* on Day 1 and Day 4. Variable sliders are a type of interactive button that allows the user to change the value of a stored variable. The variable is referenced in other buttons in the model. The user can adjust the slider then click the button that references the variable, repeating the process to see how changing the variable changes the output. The models Eric used on days one and four both had variable sliders, but they referred to different variables. On day one, the teacher first introduced the model to the class and showed how to adjust the canvas size, and she explained that they should explore and edit an image. Eric then played with the model for 30 minutes before the below episode occurred.

We characterize Eric's initial use of the model as open (as opposed to systematic) exploration, as there is a lack of connection between the ways that he used buttons on the model. For example, in Figure 1 cell A, Eric clicked the button *ask patches with [pcolor = black] [set pcolor color#]*. At the time, the slider that read *color#* was set to 65. He might have expected the part of the canvas that appears to be black to change to a different color, but he clicked the button three times and nothing changed. This is probably because the patches were not
exactly black (as read by NetLogo). Eric's next move (cell B) suggests that he suspected the lack of change was due to the value in the slider for the *color#* variable, as he slid the variable to 115. Moving the slider had no effect, as the slider changed the variable but (as is designed to do), did not apply it to the canvas. Following this move, we might expect that Eric would then click a button that references *color#*, as in step 1, to see how the variable change affects the image. Instead, Eric clicked a button that did not use the *color#* variable. This button, *ask patches [set pcolor (140 - pcolor)]* changes the color of all patches. He clicked the button twice, so first the canvas changed to blue and red (cell C), then returned back to the original colors.

<figure><img src="image_143.png" /><figcaption>Figure 1. Three sequential buttons clicks of a model on day 1.</figcaption></figure>

This sequence of button clicking suggests that Eric either did not notice, or did not appreciate the significance of the connection between the label on the variable (*color#*) with buttons that reference that variable. Indeed, subsequent moves confirm that Eric's exploration did not seem to be attuned to this coupling. In cell D in Figure 2, Eric adjusted the *color#* variable slider a second time. Again, we would expect him to click a button that uses *color#* after this to see how changing the variable affects the output. Instead, (cell E) Eric clicked the same button again that did not use the variable: *ask patches [set pcolor (140 - pcolor)]*. Again, he clicked the button twice so it cycled the canvas back to the original colors. Since he repeated the same actions and nothing changed, it's not clear if he understood what the button was doing. Finally, Eric clicked another button that referenced *color#*: *ask n-of 100 patches [set pcolor color#]* and repeatedly clicks this button over 20 times (cell F). He might have noticed what the button was doing (changing random patches to a different color), but it is not clear if he understood how it connected to the *color#* variable.

<figure><img src="image_144.png" /><figcaption>Figure 2. Three subsequent button clicks of a model on day 1.</figcaption></figure>

Eric explored this model for almost 90 minutes, indicating his interest in, and commitment to, the activity. However, the way he explored the model suggests that he was not guided by the notion of variable, as he did not seem to yet have access to the appropriate disciplinary knowledge that would lead him to see connections between different buttons. This was just a short example, but during the whole hour and a half session with NetLogo on day one, we did not see systematicity develop in his approach to the model. However, it is important to note that
Eric stayed engaged with the model, repeatedly choosing basketball-related images and exploring how the image changed in relation to the buttons, suggesting that he was enjoying the activity, engaging in work that was in some way satisfying to him, and developing a relationship to the activity that emphasized his own agency and exploration.

## Day 4

Over the course of the week, Eric began to engage and explore models more systematically, even when the concepts in the model were entirely new. For example, on day 4 a researcher introduced several new models, to offer a vision of what some “cool models” could do that the students might find useful or inspiring for their final projects. As on day one, Eric had not used this model himself previously. However, Eric’s exploration of the model seemed more noticeably strategic. After opening the model for the first time, Eric imported an image, and then saved the canvas to a variable by clicking *save current colors in patch memory mem1* (Figure 3, cell A). In doing so, he makes use of something that he has learned in previous days. Eric then imported a second image (cell B) and saved the second canvas to another variable by clicking the *save current colors in patch memory mem2* (cell C). He used the two save buttons appropriately so that each image was saved in a separate variable. These moves suggest that he has learned how these two variables work, as he is leveraging concepts he has used on previous days. However, his next moves indicate his more intentional exploration of the model.

<figure><img src="image_145.png" /><figcaption>Figure 3. Four sequential buttons clicks of a model on day 4.</figcaption></figure>

<figure><img src="image_146.png" /><figcaption>Figure 4. Eric's exploration of a model on Day 4.</figcaption></figure>
In Figure 4, Eric adjusted the *percentage* variable slider (cell D). After noticing no change, Eric paused for 19 seconds, looking intently at the model, then said “Oh I have to merge it.” He clicked the *merge* button (cell E), suggesting that he deduced that the “merge” button used the *percentage* variable even though it was not written in the name of the button. With the merge button on, Eric adjusted the slider again and noticed the images fade from one to the other (cell F). Eric said “yeah that’s really cool. Alright I can do that.”

This episode suggests a shift in the way Eric approached this model, supported by both a change in his own understanding of the models and the affordances of the programming environment. The strongest characterization of his participation is that of intentionality. This can be seen in the model he chose to investigate, the images he selected, and his exploration of the model itself. Eric selected an image blurring model, and then chose the images he imported with a purpose—the images in cells 2 and 3, when blended, create an animation that looks like a player dunking the basketball. Eric found these images online himself. His use of the model indicates a shift in his understanding of variables: Eric was methodical as he imported an image, saved it, then imported another image and saved it. This systematicity might seem simple and straightforward, but amongst camp participants, understanding what “patch memory” was, and how to use that feature appropriately, was often confusing and required multiple iterations. Next, Eric adjusted the slider variable. When he used the variable slider on day 1, he subsequently clicked several buttons that did not reference the variable, and he repeated that process multiple times. It appeared that Eric was exploring without really understanding what the buttons were doing or how they related to one another. On this day, Eric paused after he adjusted the slider and noticed no change. He looked around the screen, appearing to take the time to read the other buttons to see what he needed to do next. After pausing, he clicked the “merge” button. With the merge button pressed, he adjusted the slider again and noticed the image change. It appeared that he noticed the connection between the variable slider and the merge button in this model, unlike the way he used the slider on day 1. His exploration of this model was much more methodical and goal-oriented than on day 1, since he imported particular images and had an idea of what he wanted to happen, so Eric just had to figure out how to reach his goals with the buttons he was given. On day 1, he imported an image and spent the rest of the time clicking around different buttons to see what they did, but he did not appear to draw connections between any of the buttons given the order in which he clicked on different things throughout the session.

## Positioning with respect to others: From keeping quiet to mobilizing assistance

A second resource that supported and demonstrated Eric's change in participation could be seen in the ways he marshalled different forms of assistance. In day one, Eric seemed to attempt to be as inconspicuous as possible: he was generally quiet and talked very little, and did not speak to an adult until directly approached and being asked if he needed help. He replied by saying he was alright or "just playing." This kind of exchange happened between him and adults in the camp several times early on. Below is an example of one of those instances; Eric was focused on his computer, looking at a NetLogo model:

Teacher: *Do you need some buttons, Eric?*

Eric: *I was just ah, I don't know.* (Eric navigates to the camp website)

Teacher: *Just playing?*

Eric: *Yeah. I'm trying to figure out where that, okay.* (Eric re-opens his NetLogo file)

(the Teacher stands behind him for a few seconds, then walks away)

In contrast, Eric made strategic use of adult expertise on the fourth day. Below, Eric appeared to know what he wanted to accomplish with the model, and he asked for help when he was stuck along the design process:

Eric: *(talking to himself)* *That's really cool. I'm still gonna need help, though.* *(calling out to the room)* *Could you help me please? (a researcher comes over)* *So I want to add more than one image here. How would I do that?*

Researcher: *Yeah! So...* *(continues to explain how to add a third image; Eric does the clicking or typing on the computer while the researcher gives instructions)*

Overall, over the course of the week we see Eric's participatory identity evolving, from engaged compliance to intentional design. This shift appeared to be supported by a set of tasks that were open and positioned students to be the ones to determine what they wanted to make, what “story” they wanted to tell, or what effect they were trying to create. For Eric, such openness allowed him to explore and use images that