*use files*. Still, none of them was used neither only once nor by only one person. Such result both confirms our first hypothesis and evidence collective elaboration of metaphors (research question 2). As figure 1 shows, during the whole discussion preceding the closing sequence, the students experienced 21 episodes of focus on one single *model* (9 on *to remember*, 2 on *a blurred picture*, 3 on *to imagine*, 3 on *cogwheels*, and 3 on *to use files*). The co-construction of metaphors supporting these *cognitive models* was not linear: the students did not fully elaborate one *model* before considering another one, but rather tended to switch from one *model* to another in a heuristic way. Contrasting different metaphorical views and their ability to apprehend the target concept, such alternation may be a pattern necessary to foster the deepening of each *cognitive model*.

## Metaphorical dialectical opposition between two models: *to remember* versus *to imagine*

Such heuristic process is made visible through a bimodal analysis of each of these episodes. We created 5 collections of episodes focusing on the same *cognitive model* and paid a special attention to the corresponding metaphorical gestures. Such analysis revealed strong specificities associated to each *model*:

1. *To remember*: pictures from the outside are coming up to the head (44 s collection)

2. *A blurred picture*: thinking is a flow of blurred pictures (10 s collection)

3. *To imagine*: pictures from the inside are coming out of the head (15 s collection)

4. *Cogwheels*: each thought is a cogwheel which move causes another cogwheel's move (17 s collection)

5. *To use files*: we can create files, open them, modify them, delete them, retrieve them (53 s collection).

This gestural description of the models reveals that a strong binary opposition was co-constructed along the discussion between to think as *to remember* or as *to imagine*. The expert moderator himself makes this opposition explicit and gesturally expresses it three times during the discussion, by the 15th, 18th and 28th minute. Our interpretation is that confronting these to metaphors worked as a dialectical tool to gradually sophisticate students' understanding of the *target concept*. Such advancement in reasoning also proceeded through metaphorical thinking, by elaborating what we called a 'conceptual synthetizing metaphor' (CSM) that overcame such opposition and conciliated the strength of several *cognitive models* to better apprehend the *target concept*.

## *To use files* emerging as a conceptual synthetizing metaphor (CSM)

The cognitive model *to use files* is the latest to emerge (from episode 15th onwards), and is co-elaborated by Nourra and Jean-Luc. Finally, Jean-Luc reuses it as a full complex analogy during the closing sequence. For its late emergence in the discussion and its ability to conciliate several key features previously introduced in other metaphors, this *cognitive model* plays the role of a *conceptual synthetizing metaphor* (CSM). With respects to our third research question, a CSM is defined as a metaphor associated to advancement in reasoning by encompassing several aspects of the *target concept* that could so far only be apprehended by multiple metaphors. In this case, *to use files* is clearly a CSM allowing the participants to overcome the opposition *to remember* vs *to imagine*.

<table><caption>Table 1: Features of the three *cognitive models* 'to remember', 'to imagine', and 'to use files' (final round table)</caption><thead><tr><th></th><th>Fundamental unit</th><th>Dynamic process ?</th><th>Internal structure</th><th>Location</th><th>Size</th></tr></thead><tbody><tr><td>To remember</td><td>Evan sometimes at thinking one is uh: like remembering<br/>Nourra take an object, a place and:: recording uh: this place this object in our head and:<br/>remembering it but uh: [...] not as clear as in the real world</td><td></td><td></td><td></td><td></td></tr><tr><td></td><td>Pictures<br/>- of objects<br/>- of places</td><td>Registering<br/>Reminding</td><td>Blurred aspect</td><td>Head</td><td></td></tr><tr><td>To imagine</td><td>Sofia thinking is like [...] imagining something</td><td></td><td></td><td></td><td></td></tr><tr><td></td><td>Pictures</td><td>Active cognizer<br/>creating pictures</td><td></td><td></td><td></td></tr><tr><td>To use files</td><td>Jean-Luc [...] it's a movement in a huge archive where some files are created sometimes<br/>by chance mixing some up [...] thinking is [...] moving uh: // using the files that we have in<br/>our head</td><td></td><td></td><td></td><td></td></tr><tr><td></td><td>Files created<br/>- randomly<br/>- from previous ones</td><td>Creation<br/>Displacing<br/>Intentional use</td><td>Chronology</td><td>Head</td><td>Big</td></tr></tbody></table>
Table 1 summarizes key characteristics of the latter three *cognitive models* that appeared during the closing sequence. Both *to remember* and *to imagine* are described verbally and gesturally as dynamic processes based on pictures. *Remembering* is depicted as a double process taking place in the head, consisting in a first step in registration of pictures from the outside, and in a second step in reminding them, which is to say accessing these pictures within brain memory. *Imagining*, in contrast, depict a more active subject, constructing and sharing mental pictures created in his head. The CSM *to use files* is the only final analogy fully described into all the 5 parameters of size, localization, structure, degree of dynamism, nature of fundamental units. In this model, the fundamental units of thoughts are depicted as files, which can either be created ‘by chance’ or from a ‘preexisting file’. As a whole, the archive room located in our head is big and well chronologically structured. It hosts either intentional or random processes of file creation, displacing and deletion. We understand the idea of random processes encompassed by this CSM as an integration of the idea that thoughts can come from pictures from the outside randomly registered and reminded – a key aspect of the cognitive model *to remember*. Nevertheless, such CSM also includes the idea of intentionality that was brought from the model *to imagine*. The image that one file might be created from a previous one is a way to conciliate the idea of subject agency with the consideration that a thought is not created from scratch; such causal link can also be understood as integrating the *cogwheels* model.

**Conceptualization: bimodal collective complexification of the CSM at the micro level**

Applying a micro-analysis based on the source concept coding scheme presented above, we then show that the CSM to use files first emerged through an extended bimodal metaphor about forgetting and remembering. This metaphor was collectively built through three main phases of complexification: “metaphor introduction”, “metaphor specification” and “metaphor exploitation”, corresponding to three distinct stages of conceptualization.

**Metaphor introduction: *forgetting* is like a file that “we don't open”**

Figure 2 below corresponds to the verbal transcript and the key gesture of the metaphor introduction phase.

367 : Nourra : (...) about forgetting it's quite like a file that (...) we keep except that::: /// [1] {Ulrick:
we don't open it} / yes that's it we'll open it later (...)

<figure><img src="image_165.png" /><figcaption>Figure 2. Metaphor introduction phase transcript and illustration of gesture 1.</figcaption></figure>

During this phase, Nourra first formulated a verbal comparison for defining the target concept of *forgetting*. Then Nourra added a gesture to her speech (see illustration), which expressed the *source concept* 'to open a file'. While this gesture was synchronous with the words "except that::: ", which conveyed Nourra's hesitation, Ulrick immediately suggested a verbal formulation for it: "we do not open it". A bimodal metaphor was co-constructed about the notion of *forgetting*, and our view is that the process of monitoring synchronization described by Semin and Cacioppo (2008) allowed Ulrick experience the gestural *source concept* represented by Nourra and helped him access to the associated verbal concept. In addition, we note that the moderator illustrated the child's speech with one gesture that expressed the *source concept* 'to put a file aside'. We also note that the child imitated this gesture, immediately and many other times, which can also be explained as a monitoring synchronization process. On the reflective plane, Nourra was defining the concept of *forgetting*, exemplifying Tozzi's *conceptualization*.

**Metaphor specification and conceptual distinction: *remembering* is like “opening the file”**

Figure 2 presents the verbal transcript and illustration of four key gestures of the metaphor specification phase.

379 : Nourra: Well, remembering it's:: it/ what I was talking about // here it's opening the file //
[2a, moderator gesture]

380: moderator: rather than *putting it aside* [2b] it's {Nourra : yeah} to open it

381: Nourra: *it was* [2c] put aside *and* [2d] now we open it
<figure><img src="image_166.png" /><figcaption>Figure 3. Metaphor specification phase verbal transcript and illustration of gestures 2a, 2b, 2c and 2d.</figcaption></figure>

During this phase, Nourra created an extended the metaphor previously introduced. She used the same verbal *source concept* ('to open a file') as for defining the notion of *forgetting*, but this time for specifying the opposite concept of *remembering*. Even though the reflexive task was still conceptualization, it was now more specifically a task of distinguishing concepts. Subsequently, our coding scheme allowed us to reveal a collective synthesis. The moderator synthesized and rephrased the conceptual distinction between *forgetting* and *remembering* through a sequence including two bimodal metaphors. In the first bimodal metaphor, he repeated his gesture done during the metaphor introduction (see illustration of gesture 2b above) but adding the words "rather than putting it aside". This first bimodal metaphor was opposed to a second one, in which he created a palm up gesture representing the *source concept* of *openness* to the words "we open it". Again illustrating the monitoring synchronization process mentioned above, Nourra immediately validated this bimodal synthesis of conceptual distinction made by the moderator. She said: "it (the file) was put aside but now we're opening it", while doing gestures similar to those of the moderator, conveying the *source concepts* 'to put a file aside' and 'to integrate something'. Hence, during this phase, the student and the facilitator through verbal and gestural modalities were gradually and collectively constructing a bimodal metaphor.

## Metaphor exploitation and conceptual network elaboration: moving files in a big archive

Finally, in speech turn 391, another child used the file extended metaphor for creating a complex conceptual network. Figure 4 below shows the verbal transcript and two key gestures of this metaphor exploitation phase.

391: Jean-Luc: (...) actually according to me remembering well I'm going to cite this example again it's like // memories are like a huge archive and when you think you move from one file [3a] to another and therefore // heum sometimes when you try to think about something // you can't open this file and / and this is when you forget something and sometimes // uh when you succeed in remembering something you succeed in opening the file and this is because you have moved from one file [3b] to another

<figure><img src="image_167.png" /><figcaption>Figure 4. Metaphor exploitation phase verbal transcript and illustration of gestures 3a and 3b.</figcaption></figure>

During this third phase, Ulrick created a new bimodal synthesis of the conceptual distinction between *forgetting* and *remembering* by taking up speech and gestures of his classmate. He added two similar bimodal metaphors through which *thought* was apprehended in the terms of the *source concept* 'moving from one file to another'. In addition, he also expressed the target concept of *memory* successively through the gestural *source concept* of 'cycle' and the verbal one of 'archive'. Jean-Luc created a conceptual network connecting the concepts of *forgetting, remembering, memory and thought* by taking up Nourra's relevant speech and gestures, and adding complementary bimodal elements in order to contribute to the collective advancement in conceptualization of the notion of *forgetting* and *remembering*. This result could illustrate the complementarity of monitoring synchronization and goal-mediated synchronization implied in the Semin and Cacioppo's Social Cognition model.

## Discussion and concluding remarks

Studying the bimodal co-construction of metaphors along a CPI reveals how strongly embodied metaphorical reasoning plays a role in collaborative learning and conceptual change. Using the resources of their environment
and the interactions with each other, the participants used many source concepts in their exploration of the abstract concept of 'thinking', and were able to build heuristically a *conceptual synthetizing metaphor* through the *cognitive model* 'to use files'. The feature of the model that they used the most throughout the bimodal construction of this metaphor was 'to open a file', which corresponds to the only feature that was expressed both verbally, gesturally and collectively from its introduction phase onwards. This result supports and complete the bimodal metaphors salience hypothesis: metaphorical gestures, especially if produced by a hearer and not the speaker, plays a great role in making reasoning visible and therefore fostering collaborative learning. Further research might include multi-level studies of bimodal metaphorical reasoning in other collaborative learning contexts. Gathering more data with, when relevant, statistical treatment on the frequency and re-use of some metaphorical features may result in a theoretical generic modelisation of metaphorical reasoning together. For now, practical implications of this exploratory study still need to be addressed: how can the CSCL community seriously take into account the strongly embodied nature of metaphorical reasoning together at designing online educational interactions? A key might be to paid attention to the contextual resources offered by such environments for embodied metaphorical thinking.

## References

Charmaz, K. (2006). *Constructing grounded theory: a practical guide through qualitative analysis*. London: Thousand Oaks, Calif.

Colletta, J.-M., Capirci, O., Cristilli, C., Goldin-Meadow, S., Guidetti, M. & Levine, S. (2010). Multimodal Data Transcription and Annotation with ELAN. Coding Manual, ANR-05-BLANC-0178-01 & -02, Lidilem, University of Grenoble Alpes.

Dyke, G., Adamson, D., Howley, I., & Rosé, P. C. (2012). Towards Academically Productive Talk Supported by Conversational Agents. In S. Cerri, W. Clancey, G. Papadourakis, & K. Panourgia (Eds.), *Intelligent tutoring systems*, 7315 (pp. 531-540). Springer.

Goldin-Meadow, S. (2004). *Hearing gesture: How our hands help us think*. Cambridge: Harvard University Press.
Grize, J. B. (1997 [1990]). *Logique et langage*. Ophrys.

Lakoff, G., Johnson, M. (1980). *Metaphors We Live By*. University of Chicago Press.

Lipman, M. (2003). *Thinking in Education*. Cambridge University Press (2nd Edition).

McNeill, D. (1992). *Hand and Mind: What Gestures Reveal about Thought*. University of Chicago Press.

Menary, R., (2010). Introduction to the special issue on 4E cognition. *Phenomenology and the Cognitive Sciences*, 9, 459-463.

Mercer, N. (1996). The quality of talk in children's collaborative activity in the Classroom. *Learning and instruction*, 6(4), 359-377.

Müller, C. (2008). What gestures reveal about the nature of metaphor. In A. Cienki & C. Müller (Eds.), *Metaphor and Gesture* (pp. 219-247). Amsterdam/ Philadelphia : John Benjamins Publishing Company.

Plantin, C. (2018). *Dictionary of Argumentation*. London: College Publications.

Polo, C. (in press). La participation des gestes métaphoriques à la construction collective d'analogies cognitives en CRP. In E. Auriac-Slusarczyk (Ed.) *Les ateliers de philosophie. Une pensée collective en marche*. Sphères Educatives.

Polo, C., Lund, K., Plantin, C., Niccolai, G. P. (2016). Group Emotions: The Social and Cognitive Functions of Emotions in Argumentation. *International Journal of CSCL* 11(2), 123-156.

Roschelle, J., & Teasley, S. D. (1995). The construction of shared knowledge in collaborative problem solving. *Proceedings of CSCL (Bloomington, IN). In C. O'Malley (Ed.) *Computer-Supported Collaborative Learning* (pp. 69-97). Berlin: Springer Verlag.

Semin, G.R. & Cacioppo, J. (2008). Grounding social cognition: Synchronization, coordination, and co-regulation. In Semin, G.R. & Smith E.R. (Eds.), *Embodied grounding: Social, cognitive, affective, and neuroscientific approaches* (pp. 119-148). Cambridge University Press.

Simon, J.-P. & Tozzi, M. (Eds.) (2017). *Paroles de philosophes en herbe*. Grenoble: UGA Editions.

Stahl, G. (2006). *Group cognition*. Cambridge: MIT Press.

Tozzi, M. (2009). *Apprendre à penser ensemble*. *Diotime*, 42.
# Tracing Teacher Collaborative Learning and Innovation Adoption:
A Case Study in an Inquiry Learning Platform

María Jesús Rodríguez-Triana, École Polytechnique Fédérale de Lausanne and Tallinn University, mjrt@tlu.ee

Luis P. Prieto, Tallinn University, lprisan@tlu.ee

Tobias Ley, Tallinn University, tley@tlu.ee

Ton de Jong, University of Twente, a.j.m.dejong@utwente.nl

Denis Gillet, École Polytechnique Fédérale de Lausanne, denis.gillet@epfl.ch

**Abstract:** Social processes play an important role in teachers' ongoing professional learning: through interactions with peers and experts to solve problems or co-create materials, teachers internalize knowledge developed in their communities. However, these social processes and their influence on teachers' learning (i.e., adopting new practices) are notoriously hard to study, given their implicit and informal nature. In this paper, we apply the Knowledge Appropriation Model (KAM) to trace how different social processes relate with the implementation of pedagogical innovations in the classroom (as a marker of professional learning), through the analysis of more than 20,000 artifacts from Go-Lab, an online community to support inquiry-based learning. Our results not only show how different social processes like sharing or co-creation seem to be related to increased classroom implementation. Also, they provide insights on how we can use traces from digital co-creation platforms, to better understand the social dimension of professional learning.

## Introduction

Despite the benefits that educational innovations may entail, they require the adoption of new teaching practices (i.e., a form of professional learning), which is known to be challenging (Webb & Cox, 2004). The adoption of innovations among professionals builds on processes of knowledge creation that span individual, group and collective levels (Nonaka, von Krogh, & Voelpel, 2006): interacting with peers, seeking help from more expert colleagues, etc. Indeed, communities of practice (Wenger, 1999) have a crucial role on workplace learning, as professionals get access, adapt and internalize knowledge that has been developed in the community (often reified as documents, materials, lesson plans and other artifacts).

From a research standpoint, however, a challenge in studying this kind of professional learning is that many of these processes and knowledge are informal, implicit, and hard to trace (often happening face-to-face, across multiple spaces and timeframes). One opportunity to do so is the use of digital platforms to embody and facilitate many of these social processes. In the concrete case of teacher professional development, the use of co-creation platforms to develop materials has a long history (e.g., Vuorikari et al., 2011). However, the fact that professional development actions (e.g., training workshops) still remain quite separate from classroom implementation of the target innovations (which is the hallmark of teachers actually learning to use the innovation professionally), makes tracing this kind of complex ongoing professional learning rather elusive.

Recent advancements in research about professional and workplace learning aim at clarifying this complex system of social processes and practices. For instance, the Knowledge Appropriation Model (KAM) (1) identifies different kinds of knowledge maturation, scaffolding and appropriation practices at different social levels, as crucial in such learning process.

In this paper, we explore how the relationship between such knowledge appropriation practices (which happen at the individual, group and community levels) and the adoption of educational innovations (as a marker of professional learning) can be traced, through the study of Go-Lab, an online community aimed at supporting teaching and inquiry-based learning. Our ultimate goal is to better understand which social processes seem to be more strongly related with actual classroom implementation, to design interventions and further develop such platforms to encourage teacher adoption and learning.

An important aspect that enables this kind of study is the fact that Go-Lab's ecosystem supports not only the co-creation (with peers or experts) and sharing of materials and plans reifying the inquiry learning pedagogy (called Inquiry Learning Spaces, or ILSs). It also supports the implementation of these lessons with students in the classroom (hence making innovation adoption somewhat traceable). Hence, in this paper we trace the social practices defined in KAM as these ILSs are co-created, shared, etc. Our analysis of 24,485 such learning artifacts, created between November 2015 and November 2018, through descriptive and exploratory analyses, try to answer the following research question: *How are the knowledge appropriation collaborative practices related with the teacher adoption of inquiry-based pedagogy in the Go-Lab community?*
The rest of the paper is structured as follows: the next section summarizes related research on knowledge appropriation and professional learning theories. Then, we describe the methodology we followed in our study and, later on, the context (including how we operationalized the KAM in Go-Lab) and the results of our study. We end the paper with implications and future work.

## The role of social learning and knowledge appropriation in innovation adoption

In learning technologies research and many adjacent fields (like CSCL), the adoption of educational innovations is often studied in terms of acceptance models, such as TAM (technology acceptance model - see, e.g., Teo, 2009). This kind of models assume that the adoption of innovations in teaching is influenced by teacher's competence, self-efficacy and perceived ease of use and usefulness of the innovation. However, these models treat "technology" (or whatever other innovation) as an object that either is -or is not- appropriated. As such, they do not take into account processes of co-creation in which new practices emerge. In contrast, sociocultural learning theories look at innovation adoption as a form of social learning that is connecting individual learning to the emergence of common practices and the development of knowledge in groups and communities. Examples of this view include Trialogical Learning (Paavola & Hakkarainen, 2014) or Knowledge Maturation (Maier & Schmidt, 2015) models.

Indeed, among the many factors for the adoption of pedagogical innovations, social learning has been shown to have an important, positive impact: the more teachers are involved in professional networking and collaboration, the more likely they are to apply such pedagogies (e.g., OECD, 2014). Similarly, teacher collaboration and efficacy are correlated, and seem to predict student achievement (e.g., Lee, Dedrick, & Smith, 1991). Moreover, different kinds of collaboration have also been shown as important: understanding develops and new classroom practices emerge in the context of teachers' collaboration with peers (Blumenfeld, Krajcik, Marx, & Soloway, 1994), but also with experts (e.g., Schenke, van Driel, Geijsel, Sligte, & Volman, 2016).

It is however an open question "how such collaborative processes can best occur, what makes them effective and what are the limits of their efficacy" (Billett & Choy, 2013, p. 268). The recently proposed Knowledge Appropriation Model (KAM) (1) describes those collaborative practices that are important in the context of adopting innovations. The model identifies informal learning practices (in organizations, communities, groups and individuals) that can be observed in the context of innovation adoption. The model is based on existing sociocultural models of learning (knowledge maturation and scaffolding) and explains how these processes are interconnected in workplace learning.

In the context of KAM, knowledge maturation is understood as a social learning process that transforms knowledge (often embodied as artifacts) from the individual level into communities of interaction (by participating in activities of social groups or communities) (Maier & Schmidt, 2015). Scaffolding describes a related social learning process whereby individuals develop expertise through guided experience with experts or more advanced peers who help them to internalize knowledge that has been developed.

In previous research on workplace learning, 12 collaborative learning practices have been derived, and are assumed responsible for the successful adoption of an innovation. In the context of teacher professional learning, they would be:

* *Knowledge maturation* practices lead to the transformation and maturation of knowledge (Figure 1, left). Starting from a teacher who takes up materials for new teaching and learning methods (appropriating an idea), this knowledge is made accessible to a group of people (sharing) and further transformed (co-creation) into more mature knowledge so that they are reusable for other teachers outside the narrower community that has created it (formalization). Eventually, knowledge might reach a status in which it becomes a standard, e.g., in terms of national curricula or entering into widely accepted training material for beginning teachers (standardization).

* *Knowledge scaffolding* practices explain how this knowledge is applied in concrete working situations (Figure 1, right). A teacher may request help regarding a certain problem, and peers or experts provide guidance towards a solution. Then, as the teacher acquires competence, peers and experts fade support.

* *Knowledge appropriation* practices (Figure 1, center) ensure that the adoption of innovations is successful, sustained and scaled. These practices describe how individuals are made aware of knowledge about typical problems in the domain and possible solutions (create awareness), and how the community maintains a shared understanding about these problems and the solution. Later, teachers can adapt those solutions to new situations according to the local circumstances, and establish some form of validation (e.g., gathering experiences or formal evidence about the success and impact of the solution). Knowledge appropriation practices are assumed to underlie both maturation
and scaffolding, and these and related practices have widely been discussed in the collaborative
learning literature to be conducive to social learning.

As collaborative online platforms are increasingly being used in the collaborative design of innovative teaching and learning scenarios, it is possible to find evidence of these practices in those online platforms, and to gain an understanding of their role in the process to adopting an innovation. These insights may also help us suggest ways to improve such collaborative platforms.

In this paper, we use KAM as an analytical framework to guide the analysis of how knowledge appropriation practices and adoption of an educational innovation (as a marker of collaborative professional learning) can be traced in a (large-scale) collaborative online platform. The following section shows how the KAM helped us describe the case of the Go-Lab online community (to support IBL) and identify indicators to follow these practices and the adoption of IBL.

## Context: The Go-Lab ecosystem to support Inquiry-Based Learning

Inquiry-based learning (IBL) has long been associated with positive student outcomes like an improved ability to apply scientific thinking, or increased retention rates (Seymour, Hunter, Laursen, & DeAntoni, 2004). However, it has proven a major challenge to get large numbers of teachers to use it (Fairweather, 2008; Henderson & Dancy, 2011): since effective support needs to be offered to students in the process of inquiry (de Jong, Linn, & Zacharia, 2013; Kirschner, Sweller, & Clark, 2006), it is a notoriously demanding practice for teachers. In that sense, the support to the design, or creation, of IBL activities is one of the most-often cited (e.g. Slotta, Tissenbaum, & Lui, 2013).

The Go-Lab initiative aims to support teachers in the adoption of IBL pedagogies and on-line labs, by offering a technological ecosystem around the notion of Inquiry Learning Spaces (ILSs): pedagogically-structured learning environments that can contain labs, apps and resources (Rodríguez-Triana et al., 2014). Typically, an ILS is visualized as a web space with a set of folders (or tabs, in the student view) for each of the phases of an inquiry (from hypothesis formulation to data interpretation). These sub-spaces then contain different apps, online labs or other learning resources for students to perform the inquiry.

The Go-Lab ecosystem is composed of two platforms: a) Golabz (2), a repository where teachers can find ready-to-use apps, labs, ILSs, and support materials; and b) Graasp (3), a platform for ILS authoring and monitoring, and community gathering. Additionally, there is a help desk where teachers can request support, and specific face-to-face training events that are organized at the regional, national and international level.

To understand how the Go-Lab ecosystem mediates teachers’ social learning of the inquiry-based pedagogy, let us look at a typical example of usage. After a teacher has become acquainted with inquiry-based learning (e.g., in an online or face-to-face training event, or on her own), the teacher may decide to put it in practice in her classroom. She can, for instance, browse, find and *reuse* one of the ILSs available in the Golabz repository. If she decides to *create* the lesson from scratch, or needs to *adapt* the original ILS, she is directed to Graasp, where the ILS can be modified. At any point in this process, the teacher can *share* her work-in-progress lesson with other teachers or experts in the domain, who can *co-create* the ILS through Graasp as well. Communication during this co-creation can happen face-to-face, or through Graasp’s built-in chat (associated to each ILS). When the teacher is satisfied with the ILS and deems it ready for use in the classroom, she can direct her students to the *student-view* of the ILS: a simpler interface following the structure designed by the teacher, where students can follow the activities, input their hypotheses, data, conclusions, etc. Later on, the teacher(s) can create further *copies* of the ILS (e.g., for use with other student groups of hers). Eventually, if the teacher thinks the lesson design may be useful for other teachers as well, she can submit her ILS for *publication* in the Golabz repository, for use by the larger teacher community. During this publication process, experts will curate the ILSs, collaborating with the teacher, suggesting further enhancements or modifications.

Go-Lab has been the object of several EU projects, and as of November 2018, it has reached more than 26,000 teachers and 73,000 students all over the world, receiving on average more than 300 visits per day. While teachers can use isolated apps or labs, 63.11% have been involved in the (co)creation of more 37,000 ILSs. However, according to the number of “student-views”, only 5.51% of these ILS may have been implemented in the classroom.

It would thus be interesting to understand what social processes seem to be most associated with classroom implementation as a signal for the effectiveness of the collaborative professional learning of teachers involved in the platform. Or, said differently: *what are the social practices most followed by Go-Lab's classroom adopters?* In the study section below, we detail how we have applied the KAM to guide the identification of metrics about teacher's collaborative learning practices in Go-Lab, and explore how teachers' ILSs created in different (collaborative) ways are associated with their classroom implementation – as a marker of the teachers actually learning the IBL pedagogy they represent.
# Research questions, hypotheses, and methodology

Against the aforementioned research background and context, this paper tries thus to answer the following research question: *To what extent are collaborative professional learning practices (as defined by KAM) related to teacher adoption of IBL pedagogy in Go-Lab?* According to KAM, two hypotheses can be derived from it:

* Hypothesis 1: the ratio of adopted ILSs will increase across the categories of maturation

* Hypothesis 2: the ratio of adopted ILSs will be higher for ILSs that had been reused than for those that had not been reused

In order to validate these hypotheses, we have looked at the data available in the Graasp database and logs. In our study below, we use the ILS as the main unit of analysis, as it is the smallest entity in the system that has a pedagogical meaning, and on which knowledge appropriation actions (like sharing or co-creation) can be performed. More concretely, we have analyzed the data about the ILSs created from November 2015 (when authoring and implementation actions started to be tracked in the platform) to November 2018. Since we are interested in tracing the collaborative practices of teachers, only teacher-created spaces have been considered in the analysis (i.e., ILSs created by experts or other project members have been discarded).

Several assumptions have been made to interpret the data available in the logs. For the purposes of the description of results and discussion that follow, we considered as *experts* those registered users that are tagged as Go-Lab and Next-Lab project members (as they are assumed to be more knowledgeable about IBL than other teachers that may have entered the platform through trainings and other means). The rest of the users registered in the authoring platform are considered potential *teachers*. Also, an ILS has been considered *implemented* in the classroom (i.e., adopted) when more than 10 “student-mode users” are registered to it (this empirical rule of thumb stems from conversations with teachers using the platform, as their classes have on average around 20 students, and teachers often report their students using the platform in groups of 2 or more people).

Regarding data analysis, basic descriptive and exploratory statistics have been used. First, indicators have been drawn for each ILS as markers for the presence of different KAM social practices (see the following section). Then, counts of ILSs (and proportions of their being implemented in classrooms) have been calculated, for different levels of maturation and social involvement. Furthermore, Chi-squared tests of independence and logistic regression modelling have been performed to ascertain the potential association between these KAM-related practice indicators and ILS implementation (and their relative strength).

# Study

## Knowledge appropriation and adoption in Go-Lab

In order to better understand how KAM practices are materialized in Go-Lab, this section describes how teachers deal with knowledge maturation, scaffolding and appropriation, reflecting on where these practices take place (inside or outside of the technical ecosystem), as well as whether and how they can be monitored. Figure 1 provides a graphical overview of this, and more concrete metrics are listed in Table 1.

**Knowledge maturation practices.** With the help of the technical ecosystem, teachers can appropriate existing IBL templates when creating their ILSs from scratch. Teachers can invite other peers into their ILSs either for *sharing* or for *co-creation* purposes. Once an ILS is ready, teachers can publish it to make it available for other teachers. This process also involves an expert review that provides feedback via email about how to formalize the ILS. *Standardization* happens when ILSs are widely uptaken by other teachers at the national or international level to address specific parts of the curriculum.

**Knowledge scaffolding practices.** Teachers may *request help* from experts using the help desk available in the ecosystem, where a set of experts is available to answer doubts and provide *guidance*. Additionally, teachers often address their trainers or national coordinators face-to-face, via email, video conference, social media, or inviting them into their ILSs, so that they can have a look, comment or even edit the ILS. In addition, there are many cases where experts accompany teachers while implementing ILSs for the first time. Then, to encourage teachers to solve their own problems and be autonomous, expert support *fades* progressively reducing the face-to-face support and then interventions in the ILSs.

**Knowledge appropriation practices.** Teacher *awareness* comes from four different sources: the teachers themselves, searching and browsing through the repository and the support material; the technical ecosystem, which provides with recommendations about relevant apps, labs, resources, etc.); peers, who share their knowledge or experience using communication channels outside the ecosystem (e.g., face-to-face or in the social media); and, experts, who keep teachers updated about the new apps, labs, ILSs, functionalities and events through the newsletter, the online community and, especially, through training events. Once aware of the
available support, teachers can *adapt* existing ILSs to their own educational contexts. Thus, we consider the ILS reuse as a general proxy for the fact that knowledge appropriation has taken place.

Although the ecosystem supports communication between users, most of the discussions between peers to reach a common *understanding* about the learning context and the ILS take place outside of the platform. The review process before the ILS publication triggers a conversation between experts and teachers that not only leads to ILS refinements, but also improves the experts’ *understanding* about teacher needs, usability issues, misconceptions and challenges for ILS adoption in the real classroom.

Regarding the ILS *validation*, this practice usually takes place during the training events (out of the technological ecosystem) where practitioners are invited to share their experiences implementing ILSs in the classroom. Also, through iterative refinements due to the reuse, the validation of the content of an ILS increases copies (and copies-of-copies) of it are used and implemented.

**Innovation adoption in Go-Lab.** While there are different degrees of adoption (e.g., teachers can use IBL practices, labs, or apps independently), in this paper we focus on the potential usage of ILSs in the classroom since they represent the overall Go-Lab proposal: the IBL pedagogical approach combined with apps, labs, and multimedia resources that contribute to scaffold the students in the learning process.

<figure><img src="image_168.png" /><figcaption>**Figure 1.** Knowledge Appropriation Model (in italics) and the corresponding practices in Go-Lab. Those practices that cannot be mapped with monitorable metrics in Graasp appear in lighter color.</figcaption></figure>

While these knowledge appropriation practices involve both face-to-face and computer-mediated interventions, in this paper we focus on those supported (and traceable) through Go-Lab's technological ecosystem. For instance, since the Go-Lab repository does not require logging in, no evidence can be obtained about the “create awareness” actions carried out by specific teachers there. Therefore, only the practices mediated by the authoring platform, i.e., Graasp, will be taken into consideration in the analysis.

# Results and discussion

## Hypothesis 1: The ratio of adoption increases across knowledge maturation categories

First, we explored whether the adoption of an ILS was dependent on knowledge maturation practices. Specifically, we checked whether the fact that an ILS was assumed to be in higher stage of maturity had an effect on its eventual adoption in the classroom. In order to test the first hypothesis, *the ratio of adopted ILSs would increase across the categories of maturation* (individual, shared, co-created, formalized), each ILS created by a teacher was assigned to one exclusive knowledge maturation category according to the indicators scheme shown in Table 1. Then, for each maturation category, the rate of adoption was calculated.
<table><caption>Table 1: Monitorable knowledge maturation levels versus adoption (ILS implementations) in Go-Lab</caption><thead><tr><th>Maturation level</th><th>Indicator</th><th>Adopted</th><th>Not adopted</th><th>% Adopted</th></tr></thead><tbody><tr><td>Individual</td><td>ILS was created and neither shared nor published</td><td>651</td><td>21631</td><td>2.92%</td></tr><tr><td>Shared</td><td>ILS was shared with peers but not co-created nor published</td><td>85</td><td>560</td><td>13.18%</td></tr><tr><td>Co-created</td><td>ILSs co-created with peers, but not published</td><td>293</td><td>745</td><td>28.23%</td></tr><tr><td>Formalized</td><td>ILS was published for the community</td><td>188</td><td>332</td><td>36.15%</td></tr></tbody></table>

Our results show that knowledge maturation is strongly related to later adoption of the ILS in the classroom. While it is true that the accessibility is restricted, i.e., the individual spaces are not widely accessible in Graasp, we still find it remarkable that collaboration seems to be strongly related to adoption. Especially outstanding is the difference between sharing and co-editing. In this case, the accessibility would be typically the same, i.e., the same amount of users would have access, but the adoption more than doubled between sharing and co-editing. Aside from these descriptive results, a Chi-squared test of independence also indicates such strong association between both traits of an ILS ($\chi^2=2549.2$, p<0.001).

## Hypothesis 2: The ratio of adoption increases when knowledge is appropriated

Since the KAM assumes knowledge appropriation to be a prerequisite for knowledge maturation and eventually adoption, we hence hypothesized that knowledge appropriation (in this case the reuse of an ILS) should be one of the determinants of maturation. In the current setup of the system, it was not feasible to gather data about individual practices of appropriation (creating awareness, shared understanding, adapting and validating), and thus for this analysis we considered whether a particular ILS was reused (copied and adapted from another ILS), as a proxy for whether knowledge appropriation had taken place. Then, as shown in Table 2, we compared the rates of adoption for reused versus non-reused ILSs.

We found that knowledge appropriation in Go-Lab also is strongly associated with later adoption. In this case, we assumed that the reuse of an ILS would be indicative of several appropriation practices: to reuse existing material, there should be first awareness of its existence; then, to implement the ILS in the classroom, it is necessary to refine and adapt it to the contextual needs, contributing to build sharing understanding; and the adoption shows not only the interest of the community but also validates the feasibility to apply it in real scenarios. A Chi-square test of independence also rejects strongly the null hypothesis (i.e., appropriation and adoption being independent): $\chi^2=2035.9$, p<0.001.

<table><caption>Table 2: Monitorable knowledge appropriation levels versus adoption (ILS implementations) in Go-Lab</caption><thead><tr><th>Appropriation level</th><th>Indicator</th><th>Adopted</th><th>Not adopted</th><th>% Adopted</th></tr></thead><tbody><tr><td>Not reused</td><td>ILSs of which no copies have been made</td><td>868</td><td>22642</td><td>3.69%</td></tr><tr><td>Reused</td><td>ILSs of which no copies have been made</td><td>349</td><td>626</td><td>35.79%</td></tr></tbody></table>

## Relative associations between knowledge maturation and appropriation

In order to perform a first exploration of the relative strength of the two variables targeted by our hypotheses (i.e., knowledge maturation and appropriation practices), we built a logistic regression model that predicts whether an ILS will be adopted or not, as a function of its levels of maturation (from individual to formalized) and appropriation (i.e., whether it is a re-use of a previous one). The results are summarized in Table 3. The results seem to indicate that both maturation and appropriation have a significant positive effect (as indicated by the analysis of the hypotheses above): each level of maturation or appropriation increases the odds of an ILS being implemented by a factor of 3.7 and 3, respectively. Moreover, the fact that the quadratic coefficient of the knowledge maturation is negative (i.e., the positive association decreases in higher levels of maturation) may point to the fact that maturation is especially important in the early phases of the process, where the artifacts still need a lot of collaborative effort in order to be understood and usable for others.
<table><caption>Table 3: Coefficients of a logistic regression model of adoption as function of appropriation and maturation</caption><thead><tr><td>Coefficients</td><td>Estimate</td><td>Std. Error</td><td>z value</td><td>p-value</td></tr></thead><tbody><tr><td>Intercept (Individual, Not reused)</td><td>-1.463</td><td>0.045</td><td>-32.23</td><td>&lt;0.0001</td></tr><tr><td>Maturation (linear)</td><td>1.298</td><td>0.097</td><td>13.32</td><td>&lt;0.0001</td></tr><tr><td>Maturation (quadratic)</td><td>-0.978</td><td>0.089</td><td>-10.92</td><td>&lt;0.0001</td></tr><tr><td>Appropriation (linear)</td><td>1.097</td><td>0.076</td><td>14.36</td><td>&lt;0.0001</td></tr></tbody></table>

## Conclusion and future work

While social processes have been considered crucial in teachers' ongoing professional learning, their implicit and informal nature make their study challenging. In this paper, the Knowledge Appropriation Model (KAM) helped us to define traceable social practices that could then be related with the adoption of IBL practices in the classroom through Go-Lab. This focus on social practices at work makes our study closely related to the computer-supported collaborative work domain. However, we should not forget that the main focus is on the teachers learning to integrate IBL in their practice, and how informal (computer-supported) collaborations around shared artifacts (in our case ILS) facilitate this learning. Then, the usage of these artifacts in the classroom should be looked at as one of the few reliable markers to assess whether teachers have learned to integrate such pedagogy in their practice.

Based on the ILSs created between November 2015 and November 2018, out of 24,485 ILSs created by teachers, 1270 (4.99%) were potentially implemented. The descriptive and exploratory analyses of the dataset support our initial hypotheses about how collaborative professional learning practices (as defined by KAM) relate to teacher adoption of IBL pedagogy in Go-Lab: the higher the maturation and the appropriation, the higher the adoption – as a marker of teachers learning the pedagogy. This positive association between knowledge maturation, appropriation and IBL adoption, could help to enrich existing strategies for teacher professional development in IBL (Maaß & Doorman, 2013). For example, mentoring and collaborative practices among teachers could be promoted in trainings, and be facilitated even further in IBL digital platforms.

While these findings are encouraging, other factors may well be playing a role in the adoption (e.g., participation in trainings, previous IBL teacher experience, experts' interventions, etc.). Indeed, the present study focuses on social processes mediated by an on-line platform, whereas additional collaboration may have taken place through other face-to-face or digital channels. Thus, further research (e.g., in the form of follow-up qualitative studies) is needed to examine the assumptions and conclusions of our present analysis, and the influence that other factors may have had on the adoption of Go-Lab.

It should be also noted that the filter applied to detect ILS adoption was purely superficial (based on the average number of students expected per classroom implementation) and did not take into consideration the qualitative characteristics of the ILS. Therefore, future studies should analyze more thoroughly the work done by the students as well as the ILS structure and content. This includes not only content/structure analyses of the ILSs, but also time-based models of the knowledge maturation and appropriation processes (e.g., using process mining techniques on the different stakeholders' actions in the platform).

Another feature of this initial work towards disentangling the different social processes that play a role in the adoption of IBL, is our present focus on the professional learning artifacts (i.e., the ILSs), rather than focusing on the teachers themselves as the unit of analysis. In future work we expect to find models that help us understand not only artifacts but also skill adoption among practitioners, so that we can identify and promote efficient learning paths for teachers.

While the overall results described in this paper (i.e., the fact that more mature artifacts/lessons are implemented more often in the classroom) may seem somewhat self-evident, our main contribution is rather the operationalization of this hypothesis in terms of concrete, traceable metrics in a digital platform. This is especially valuable in the realm of (collaborative) professional learning, where empirical evidence of such learning and surrounding collaborative processes tends to be scarce. The first steps presented here open thus the door to further investigations of how collaboration around shared, “implementable in practice” artifacts mediate informal professional learning.

## Endnotes

(1) Knowledge Appropriation Model: http://results learning-layers.eu/scenarios/knowledge-appropriation

(2) Go-Lab initiative: www.golabz.eu

(3) Graasp: graasp.eu
## References

Billett, S., & Choy, S. (2013). Learning through work: emerging perspectives and new challenges. *Journal of Workplace Learning, 25*(4), 264–276.

Blumenfeld, P. C., Krajcik, J. S., Marx, R. W., & Soloway, E. (1994). Lessons Learned: How Collaboration Helped Middle Grade Science Teachers Learn Project-Based Instruction. *The Elementary School Journal, 94*(5), 539–551.

de Jong, T., Linn, M. C., & Zacharia, Z. C. (2013). Physical and virtual laboratories in science and engineering education. *Science, 340*(6130), 305–308.

Fairweather, J. (2008). *Linking evidence and promising practices in science, technology, engineering, and mathematics (STEM) undergraduate education*. The National Academies National Research Council Board of Science Education. Retrieved from

Henderson, C., & Dancy, M. H. (2011). *Increasing the impact and diffusion of STEM education innovations*. Presented at the White Paper commissioned for the Characterizing the Impact and Diffusion of Engineering Education Innovations Forum, Retrieved from https://www.nae.edu/File.aspx?id=36304

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching. *Educational Psychologist, 41*(2), 75–86.

Lee, V. E., Dedrick, R. F., & Smith, J. B. (1991). The Effect of the Social Organization of Schools on Teachers' Efficacy and Satisfaction. *Sociology of Education, 64*(3), 190–208.

Maaß, K., & Doorman, M. (2013). A model for a widespread implementation of inquiry-based learning. *ZDM: The International Journal on Mathematics Education, 45*(6), 887–899.

Maier, R., & Schmidt, A. (2015). Explaining organizational knowledge creation with a knowledge maturing model. *Knowledge Management Research & Practice, 13*(4), 361–381.

Nonaka, I., von Krogh, G., & Voelpel, S. (2006). Organizational Knowledge Creation Theory: Evolutionary Paths and Future Advances. *Organization Studies, 27*(8), 1179–1208.

OECD. (2014). *TALIS A Teachers’ Guide to TALIS 2013 Teaching and Learning International Survey: Teaching and Learning International Survey*. OECD Publishing.

Paavola, S., & Hakkarainen, K. (2014). Trialogical Approach for Knowledge Creation. In S. C. Tan, H. J. So, & J. Yeo (Eds.), *Knowledge Creation in Education* (pp. 53–73). Singapore: Springer Singapore.

Rodríguez-Triana, M. J., Govaerts, S., Halimi, W., Holzer, A., Salzmann, C., Vozniuk, A., ... Gillet, D. (2014). Rich open educational resources for personal and inquiry learning: Agile creation, sharing and reuse in educational social media platforms. In *2014 International Conference on Web and Open Access to Learning (ICWOAL)* (pp. 1–6). IEEE.

Schenke, W., van Driel, J. H., Geijsel, F. P., Sligte, H. W., & Volman, M. L. L. (2016). Characterizing cross-professional collaboration in research and development projects in secondary education. *Teachers and Teaching, 22*(5), 553–569.

Seymour, E., Hunter, A.-B., Laursen, S. L., & DeAntoni, T. (2004). Establishing the benefits of research experiences for undergraduates in the sciences: First findings from a three-year study. *Science Education, 88*(4), 493–534.

Slotta, J. D., Tissenbaum, M., & Lui, M. (2013). Orchestrating of Complex Inquiry: Three Roles for Learning Analytics in a Smart Classroom Infrastructure. In *Proceedings of the 3rd International Conference on Learning Analytics and Knowledge* (pp. 270–274). Leuven, Belgium.

Teo, T. (2009). Modelling technology acceptance in education: A study of pre-service teachers. *Computers & Education, 52*(2), 302–312.

Vuorikari, R., Berlanga, A., Cachia, R., Cao, Y., Fetter, S., Gilleran, A., ... Others. (2011). ICT-based school collaboration, teachers' networks and their opportunities for teachers' professional development-a case study on eTwinning. In *International Conference on Web-Based Learning* (pp. 112–121). Springer.

Webb, M., & Cox, M. (2004). A review of pedagogy related to information and communications technology. *Technology, Pedagogy and Education, 13*(3), 235–286.

Wenger, E. (1999). *Communities of Practice: Learning, Meaning, and Identity*. Cambridge University Press.

## Acknowledgments

This research has been partially funded by the European Union in the context of Go-Lab, Next-Lab, CEITER (FP7 grant agreement no. 317601, and Horizon 2020 Research and Innovation Programme, grant agreements no. 731685 and 669074).
# A Scaled Analysis of How Minecraft Gamers Leverage YouTube Comment Boxes to Participate and Collaborate

Justice T. Walker, Stefan Slater, and Yasmin B. Kafai

justicew@upenn.edu, slater.research@gmail.com, kafai@upenn.edu

University of Pennsylvania Graduate School of Education

**Abstract:** Minecraft—a commercially successful and popular sandbox game—is increasingly used in formal educational spaces. While there is significant research on how online gaming can support positive outcomes when used as a pedagogical tool, little research has investigated how gamers collaborate to learn in informal spaces. We conducted an analysis of a curated subset of more than 540,000 comments of Youtube videos to explore how users participate and practice in discussions about Minecraft where comment boxes are the primary method used to communicate about video content. Our findings reveal that gamers use online comment sections for multiple reasons, from participating in shared social experiences unique to Youtube as a community to giving myriad forms of feedback to learn and further content development. In our discussion, we address the ways other online game platforms can better support learners seeking out information and collaborating in ways that are constructive and inclusive.

**Keywords:** Gaming, Metagaming, Feedback, Participation, Minecraft

## Introduction

Much research has focused on learning and collaboration while playing (Connolly et al., 2012; Clark et al., 2016) and making digital games (De Freitas, 2006; Basawapatna et al., 2010), which has collectively pointed to the influence games have in supporting positive outcomes in academic content areas (e.g., language arts, computer science, mathematics) and skills (e.g., literacy, problem solving and argumentation). These outcomes have been especially promising for education efforts designed to support broader access to science fields where social and intellectual participation disparities exist. Alongside these efforts is research that attends to learning and collaboration in massive multiplayer games where game elements and social interactions are complex and more difficult to ascertain (Paraskeva et al., 2010). Examples include commercially successful games like the role playing-based World of Warcraft (Clark et al., 2009), the simulation-based SimCity (Tanes and Cemalcilar, 2010), and the open narrative-based Minecraft (Nebel et al., 2016) where gamers can interact on massive server networks that are globally distributed.

With more than 100 million registered users, Minecraft is an example of a game where users are able to collaborate to make and play games (Makuch, 2014). In Minecraft, players use tools and materials to build complex structures that can be automatized or programmed, especially through the use of third-party game modifications (mods). YouTube has become a popular metagaming environment for Minecraft by providing an online open access video platform where content creators have posted more than 70 million Minecraft tutorials videos publicly. In fact, Minecraft-related searches on Youtube represent the second most commonly searched term on the video platform (Thompson, 2016). Prior research of this environment examined a small number of Minecraft player Youtube video producers to understand the nature of user interactions (Niemeyer and Gerber, 2015). Findings suggested that this online video platformed represented—in broad terms—a “collaborative learning community” that supported constructive engagement among players. While these findings offer important justifications for further research on how collaboration takes place in this metagaming space, few research (Niemeyer and Gerber, 2015; Anderson et al., 2017) studies have focused on learning outside of gaming environments and in digital spaces that develop around games— the so-called metagaming spaces (Gee, 2008). Gee and others have argued that more learning happens in these contexts and communities than in traditional learning spaces. However, research of these spaces is difficult: metagaming environments generally operate on private web servers (e.g., personal networks or in-game chat systems) or are distributed across physical environments (e.g., conferences, camps, festivals, etc) that represent only a small fraction of the total number of community members that participate in a given game. As a result, it has been difficult to research these contexts at scale. In this study, we use natural language processing (NLP) approaches to characterize and make sense of the complex ways Minecraft gamers participate and provide feedback when using video tutorials and comment sections. We ask: (1) what modes of participation are used by Minecraft players in the comments section of Youtube? (2) what forms of feedback do Minecraft users provide when using Youtube? and (3) what
are the affordances and constraints of using online video platforms to participate in feedback? In the discussion, we address the ways online game platforms can better support accessible and equitable learning and collaboration in game environments that are self-directed as well as directions for future research.

# Background

## Collaboration in gaming

In learning, collaboration has been described as moments when learners and/or experts come together to share ideas, feedback, and attention toward a shared understanding or goal (Davis et al., 2018). The relationship between collaboration and learning has been conceptualized using constructivist perspectives that describe learning as a social interaction where so-called novices and experts come together to learn from one another. In this theoretical trajectory, digital games can be understood as communities of practice as players of different mastery levels convene around a common interest (Lave and Wenger, 1991). It is within these communities where collaborations guide interactions and support learning. There are a number of collaborative arrangements in digital game environments that positively support these outcomes and that have been shown to be instrumental in important cognitive processes including reflection.

One collaborative arrangement that has been shown to be especially effective involves situations where learners provide each other feedback (Butler and Winne, 1995; Plass et al., 2015). This has been effective in traditional learning research in literature, mathematics, and computer science. Feedback has also been examined in digital game platforms. Research in this area has shown that collaborative feedback has supported (civic) engagement, knowledge building, and skill building and across a range of subject areas. While most studies examine collaborative feedback while playing games, this research explores a new genre of game platform that combines game play and construction. In this form, gamers not only engage in play, but they also have opportunities to build and modify game environments, characters and objectives.

Minecraft is an example of this type of game and is the frame within which this research examines collaboration and feedback. Platforms like Minecraft have previously been described as among the most promising game-based approaches to support learning (Gee, 2008; Plass, 2015). However, education research that explores the ways Minecraft can support learning is scant (Nebel et al., 2015). Existing research has described ways the ways Minecraft can be utilized in K-12 learning environments to teach mathematics concepts (Short, 2012), in-game collaboration tactics (Marklund et al., 2013) and collaborative design (Cipollone et al., 2014; French et al., 2016). Research has also assessed collaboration differences in underrepresented minority groups (Ames & Burrell, 2017), and small-scale analysis of collaboration techniques for learning (Wernholm & Vigmo, 2015; Davis et al., 2018) among young players.

## Collaboration in metagaming

While these studies provide important insights into the nature of collaborations among diverse players, much less is known about how gamers collaborate in spaces outside the actual game platform—Minecraft's metagame environments. Because these environments play a significant role in supporting learning outside of Minecraft (Bebbington, 2014), understanding the ways gamers collaborate in these spaces provides perspectives that are not always apparent in the game environment. Although research has shown that feedback benefits collaborative learning arrangements, adverse outcomes have also been identified. Some of these outcomes include inequitable participation where only a few learners benefit from learning outcomes associated with feedback-driven collaboration (Ames and Burrell, 2017). Other adverse outcomes occur when feedback is not constructive or is not used to drive reflection. Because of challenges related to access and scale, little research investigates the myriad ways in which learners engage in feedback in digital games or their associated metagaming environments.

Videos on Youtube can provide an important insight into the nature of collaboration in the metagaming space. Minecraft players post millions of videos on YouTube showcasing their play experience or explaining particular concepts. Minecraft related topics on Youtube include instructions for making and applying 'skins' (changes to the physical appearance of a Minecraft avatar), strategies for mining and obtaining resources, redstone networks based on computational circuits and logic, and 'modding' (third-party code that adds additional game functionality).

Previous research that examined thousands of comments in the online youth programming community Scratch (Fields et al., 2015) found that players were able to provide constructive feedback. However, within the Scratch online community, the commenting feature is integrated within the environment and thus not a traditional metagaming location. Youtube videos are a traditional metagaming location, but the traffic served by the website is multiple orders of magnitude larger. To overcome this challenge, we use a bag-of-words
approach—specifically keyword spotting (Cambria and White, 2014)—to curate, collect and analyze moments of user collaboration in order to provide a scaled insight into collaborations that occur in these comment spaces, the collaboration affordances and constraints that exist using online video platforms, and ways to support constructive collaborations in similar online video-based meta game environments.

## Methods

Data were collected in September 2018, from the comment sections of a sample of videos posted to Youtube. In order to generate our sample of videos, we first developed a list of search terms related to different actions and behaviors within Minecraft that users might want to engage with. We used Youtube's public application program interface (API) to write a Python script that returned the top 50 video results for each of our search terms, detailed in the next section. We used the video identification (IDs) obtained from this process to collect every comment left on each of the sampled videos. This process yielded more than 540,000 unique comments. From this sample we performed qualitative coding using an inductive approach (Ravitch and Carl, 2015) in order to identify a subset of terms relevant to learning and social engagement. This yielded a sample of more than 27,000 unique comments. Of this sample, we randomly selected a subset of about 100 and developed a ground-up codebook to describe how Youtube users participate and collaborate with one another when using the comments section to learn. We then applied these codes to a random set of 518 unique comments drawn from the 27,000 comment sample. In the following section, we detail our specific video sampling criteria and subsampling techniques.

## Video selection

We searched Minecraft-related videos based on nine search terms encompassing a number of popular approaches to playing. For each search, the top 50 video results according to Youtube's relevance algorithm were selected. From these videos, every comment was recorded along with the author, the date posted, the content of the comment, and the number of likes/replies the comment received. To collect comments related to 'skins,' we searched for videos using the search terms 'minecraft how to skins' and 'minecraft skins tutorial.' For mining, we used search terms 'minecraft how to mine' and 'minecraft mining tutorial'. For 'Redstone,' we used the search terms 'minecraft how to redstone' and 'minecraft redstone tutorial.' Finally, we used the search terms 'minecraft best mods', 'minecraft how to mod', and 'minecraft mod tutorial' to search for videos related to the modding process in Minecraft.

## Subsampling the dataset

Including several videos collected during our code testing, our dataset included 433 unique videos and 546,034 unique comments. Due to the size of this dataset, we generated several smaller datasets to make qualitative coding approaches feasible. We performed this dataset reduction in two steps. We first selected all comments from the dataset that were either liked or replied to at least once. These criteria allowed us to select for comments that showed some level of social engagement. This yielded two datasets—52,775 'liked' comments and 37,930 replied comments, respectively. In the first dataset, 21,441 comments were liked more than once, and the maximum number of likes received by any comment was 9,228. In the reply dataset, 18,334 comments were replied to more than once, and the maximum number of replies received by any comment was 7,460.

We used these two datasets to perform preliminary inductive coding, in order to identify how Youtube users engage with Minecraft, and how collaboration and learning within the comments might be operationalized. From this preliminary coding, we identified a set of 11 key words and phrases that appeared to be associated with a deeper level of engagement in the content. These keywords included 'figure out' (433 comments), 'turn out'/'turns out' (55), 'I found'/'I have found' (1,849), 'because' (8,440), 'I think' (4,381), 'I can' (6,791), 'how do' (4,371), 'how can' (656), and 'learn to'/'learn how'/'learn about' (176).

Finally, we performed both inductive and deductive coding on each of these keyword datasets. For datasets containing more than ~100 comments, we selected random samples of 100 comments from within them to code. We used an inductive coding approach to identify both common themes of discussion on Youtube, as well as social norms and platform-specific memes that users engage in. Deductive coding was used to identify the degree to which a comment was emblematic of involving feedback or social participation.

## Findings

We report finding about the forms of participation Minecraft gamers use when using the comments section on Youtube tutorial videos. We also report on the various ways Minecraft gamers use feedback to collaborate and participate. Unless otherwise stated, these codes were not mutually exclusive and could co-occur.
# Finding 1: Minecraft gamers mostly: use comments to participate, reference in-game activities, and engage Youtube video producers

Of 518 Youtube comments assessed, 346 involved some form of social participation with other people. This could involve either sharing personal experiences from playing Minecraft, or it could involve conversations with video authors or other users on Youtube.

101 of these participation comments were coded as involving *ways that Minecraft gamers participated*. These results are summarized in figure 1a. Unsurprisingly, in-game references (88%) represented the largest form of participation. These were moments when gamers made reference to experiences either playing Minecraft or engaging with Minecraft. For example, one commenter explained, “So aswome because I don’t mine at night any more—but when I go in the day, it turns out night when I get out an have to sprint to get to my house.” Here the user is using the comment box to describe their strategy during play after viewing a related tutorial. References to personal experiences (12%) also occurred and represented instances when gamers mentioned personal life experiences that were not directly related to Minecraft. For example, a video reminded a commenter of sleepovers with a friend, “I first heard this 4 years ago when my friend came over and introduced me to minecraft. 4 years later and I have found the song. OMG so much nostalgia. Plus also nostalgic because it reminds me of all the sleepovers I had with the friend.” In this example, it can be seen that personal stories often overlapped with nostalgia (11%), which was represented in moments when gamers made reference to or recollected a time in the past. Finally, the smallest set of comments represented instances when gamers used comment boxes to reference to other games (2%).

201 comments were addressed to a particular audience, and explicitly social in nature. The majority of these comments were directed toward Youtube video producers (53%). For instance, as one commenter used the comments to solicit help from the video author directly, saying “I really wanted a mod but it turns out ur tutorial doesn’t work for me when I Vick ur link it takes me to a different forge pls answer me.” Other commenters (13%) and Minecraft game players broadly (2%) were also focal points of engagement in comment boxes as commenters would ask others to describe their own experiences or refer to other gamers on Youtube about Minecraft experiences. Finally, we coded for unclear or ambiguous participatory comments, instances where it was not possible to determine to whom the engagement was directed. For instance, the comment “you know what do you have a server ? because there is a cool game its called egg wars:D” could be directed either at a video author or another commenter.

96 comments referenced Youtube as a platform and its various design affordances. Comment codes (85%) included references to using Youtube's comments section as a site of shared participation, such as requests to “start a comment chain” around a particular idea or behavior in the video. This represented the largest percentage of participation forms gamers used on Youtube. References to comment or video “likes” (7%) are instances where the commenter refers to the number of likes a video has or encourages other users to like/dislike a particular video. Technical references (6%) were also observed and were instances when gamers referenced a technical aspect of Youtube such as lag on a video or a missed notification. References to subscribing (5%) to a video were also observed and involved mention of the subscribe feature of a Youtube video channel. These comments were often solicited by the author (e.g., “SUBSCRIBE!!!”) in order to grow a viewer audience base. Notifications (1%) were infrequently observed and represented instances when a gamer refers to being notified (or not) about video content on Youtube. Similar to subscribe references, these typically represented moments when commenters signaled wanting to be a part of or being a part of a viewer audience and stay up to date on new content.

Finally, 80 comments included digital artifacts of participation beyond simply leaving a comment: poems (68%), hashtags (14%), stories (10%), emojis (4%), memes (4%), and links to external resources (3%). These results are summarized in figure 1b. These represent instances when gamers used atypical representations to participate, such as collaboratively posting lyrics to a song parody, or telling elaborate and embellished stories about a particular behavior.
<figure><img src="image_169.png" /><figcaption>**Figure 1.** Participation Forms (a) and Digital Artifacts (b) Used.</figcaption></figure>

## Finding 2: Minecraft gamers mostly use affective and neutral feedback and primarily to assess Youtube video usefulness and functionality

261 of the 518 coded comments involved some form of feedback, with users sharing their thoughts and opinions on video content, video authors, or other commenters, among other topics

Of this sample, 134 involved comments that referenced specific in-game game features including ‘mods’ (41%), redstone (37%), mining (19%), building (18%), general play (16%), and ‘skins’ (7%).

Also, from this sample, 168 comments reflected instances when gamers used the Youtube comment section to provide either affective or social feedback. Feedback that was evaluative (49%) included judgements that included affective features like sarcasm, insults, compliments, or encouragement. Feedback that was suggestive (41%) included moments when commenters suggested ideas for future Minecraft-related tasks, strategies, and/or video content. An illustrative example of this occurred when a commenter remarked, “can you do a customstom command series were you learn to make a command block machine and make minecraft a better place to play in :).” Here the commenter is suggesting through request that the Youtube video producer make a specific series of tutorial content. Neutral feedback (29%) reflected moments when commenters provided feedback that did not have an affective quality. This was illustrated when a commenter remarked about the audio being played during a tutorial “shoulve been learn to fight at night,” which is a critique about the audio that best represents the tutorial and yet does not explicitly connote any affective judgement. Feedback that was appreciative (21%) reflected moments when commenters asserted thanks because of something related to video tutorial content or the video producer. This type of feedback was evident when a commenter noted, “Thanks for this series seth it is really clear and instructive and i’m really looking up to learn to code turtles makes FTB way easier.” The commenter’s feedback also goes further to explain the reasons why they appreciated the tutorial. Feedback that was collaborative (3%) in nature demonstrated moments when video tutorials prompted a commenter to solicit video producers or other commenters to collaborate on a common Minecraft related endeavor. Findings are summarized in figure 2a.

Content focused feedback occurred in 112 feedback-related comments. Of these, the majority emphasized video tutorial usefulness (72%) in helping viewers achieve some in-game outcome as illustrated when a commenter remarked, “Hey Mumbo I love your vids and watch them every day! Your redstone tutorials have helped me sooo much and I have actually have been able to build some complicated redstone contraptions because of you! [emojis]” In this case the commenter is underscoring the idea that the video producer’s tutorial helped the content viewer achieve a goal - in this case learning to use redstone in complicated ways. These observations are summarized in figure 2b. In addition, 172 involved comments that reflected feedback that was directed toward: content (88%), Youtube producers (11%) and technical video features (3%).
<figure><img src="image_170.png" /><figcaption>Figure 2. Feedback Focus (a) and Forms (b).</figcaption></figure>

There were also instances when commenters provided feedback about the strategy (30%), solution (26%), and efficiency (15%) of proposed game play. An illustrative example of these occurred when one commenter asserted, “Dude searching all this stuff is slow as hell and less reliable, sure mining sometimes turns out slow but trust me after like 40 minutes I have stacks of materiel and sometimes enough for Diamond Armour.” Here, the commenter is providing feedback about the efficiency of a mining strategy and the benefits of using the approach in reference. Feedback related to functionality (15%) reflected moments when video commenters made reference to the function of an approach such as the ability to produce a functional redstone-based product. Aesthetics (11%) related feedback were moments when commenters referred to the aesthetic quality of a video or Minecraft artifact such as—for example—the sound of a song or look of a building as exampled when one commenter noted “I like the song, but i think the vid would be better with the default textures....” The commenter is pointing out the mod used to alter the game environment would have been visually better if the game’s default settings were maintained.

## Discussion and conclusion

Here we advance a mixed-methodological approach that uses NLP and qualitative methods to overcome challenges associated with analyzing qualitative data from metagaming platforms at scale. By doing so, we're able to assess the myriad ways Minecraft users participate and collaborate in tutorial-based video content on Youtube. Our analyses show several trends involving the types of feedback and collaboration that occur on this platform.

First, we observed that Minecraft gamers use Youtube to participate in multiple forms of discourse. While comment section discourse represented the most frequent form of engagement in our sample, we also observed song remixing and parodying. This suggests that online platforms support unique cultural norms and practices. We also observed moments where Youtube supported just-in-time, on-demand access to quasi-experts—video content producers that have specialized domain knowledge of a set of Minecraft related features (e.g., redstone, mods, skins, etc.). This underpins important implications for learning at scale in environments where access to domain experts is limited. Here, we observe that metagame communities provide an important space for learners to not only engage with experts in order to achieve learner-centered goals in the game environment, but also to provide feedback on content which these experts should cover.

We also observed that Minecraft gamers frequently use forms of feedback to assess Youtube content. Notably, feedback about Youtube videos primarily focused on game-related content, suggesting that this metagame environment is an important space for (Minecraft) learning—harkening back to Gee's characterization of the ways these spaces support meaningful engagement. Furthermore, commenters most often provided feedback that was evaluative and suggestive. This underpins the prevalence of affective social engagement on Youtube. It suggests that Youtube acts as a place for learners to advance not only their own learning repertoires, but also that of others. Learners very often made unprompted suggestions (both to video authors and other commenters) to advance tutorial content.

Importantly, this research provides important first attempts to elucidate insights into the myriad ways gamers use a metagaming environment experience to participate and productively collaborate while learning.
While Youtube affords opportunities for diverse participation and engagement, important constraints exist. These constraints include the ability for users to participate in discourse beyond those afforded by Youtube's technical features. We observed, for instance, users participating and collaborating in ways that were marginal compared to prominent comment-based and evaluative forms. These instances represent an important need to create space for participants to engage in more broad forms of engagement. In addition, our research uncovers an important risk that exists when social engagement takes on a predominantly affective orientation. This risk could create adverse outcomes if, for instance, dominant voices or practices unintentionally place groups at risk of marginalization in this space, an outcome that could exacerbate education disparities. Nevertheless, we report on methodological approaches and perspectives that support research on metagame environments at scale.

In conclusion, we argue that metagaming environments are spaces where important learning processes take place. The affordances of these environments can be used by educators as well as learning systems designers in order to produce more effective and fulfilling collaborative learning practices.

## References

Ames, M.G. & Burrell, J. (2017). Connected Learning and the Equity Agenda: A Microsociology of Minecraft Play. In Proceedings of the 20th ACM Conference on Computer-Supported Cooperative Work and Social Computing. ACM, New York, NY, 446-457. DOI: https://dx.doi.org/10.1145/2998181.2998318

Anderson, E., Walker, J., Kafai, Y. B., & Lui, D. (2017). The gender and race of pixels: an exploration of intersectional identity representation and construction within minecraft and its community. In Proceedings of the 12th International Conference on the Foundations of Digital Games (p. 17). ACM.

Basawapatna, A. R., Koh, K. H., & Repenning, A. (2010). Using scalable game design to teach computer science from middle school to graduate school. In *Proceedings of the fifteenth annual conference on Innovation and technology in computer science education* (pp. 224-228). ACM.

Bluemink, J., Hamalainen, R., Manninen, R., & Jarvela, S. (2010). Group-level analysis on multiplayer game collaboration: How do the individuals shape the group interaction? *Interactive Learning Environments, 18*(4), 365-383. doi:10.1080/10494820802602444

Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. *Review of educational research, 65*(3), 245-281.

Callaghan, N. (2016). Investigating the role of Minecraft in educational learning environments. Educational Media International 53, 4 (2016), 244-260. DOI: http://dx.doi.org/10.1080/09523987.2016.1254877

Cambria, E., & White, B. (2014). Jumping NLP curves: A review of natural language processing research. *IEEE Computational intelligence magazine, 9*(2), 48-57.

Cipollone, M., Schifter, C. C., & Moffat, R. A. (2014). Minecraft as a creative tool: A case study. *International Journal of Game-Based Learning (IJGBL), 4*(2), 1-14.

Clark, D., Nelson, B., Sengupta, P., & D'Angelo, C. (2009). Rethinking science learning through digital games and simulations: Genres, examples, and evidence. Paper presented at the Learning science: Computer games, simulations, and education workshop sponsored by the NAS, Washington, DC.

Clark, D., Tanner-Smith, E. E., & Killingsworth, S. S. (2016). Digital games, design, and learning: A systematic review and meta-analysis. *Review of educational research, 86*(1), 79-122.

Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. *Computers & Education, 59*(2), 661-686.

Crook, C. (1994). Computers and the collaborative experience of learning. London: Routledge & Kegan Paul.

Davis, K., Boss, J. A., & Meas, P. (2018). Playing in the Virtual Sandbox: Students' Collaborative Practices in Minecraft. *International Journal of Game-Based Learning (IJGBL), 8*(3), 56-76.

De Freitas, S. (2006). Learning in immersive worlds: A review of game-based learning.

Dezuanni, M., O'Mara, J., Beavis, C., Potter, J., & Gilje, Ø. (2015). 'Redstone is like electricity': Children's performative representations in and around Minecraft. E-Learning and Digital Media, 12(2), 147-163. doi:10.1177/2042753014568176

Dieterle, E., & Clarke, J. (2006). Multi-user virtual environments for teaching and learning. In M. Pagani (Ed.), *Encyclopedia of multimedia technology and networking* (2nd ed.). Hershey, PA: Idea Group, Inc.

Evans, A. C., Wobbrock, J. O., & Davis, K. (2016). Modeling collaboration patterns on an interactive tabletop in a classroom setting. In ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW '16) (pp. 860-871). New York: ACM Press. doi:10.1145/2818048.2819972

Fields, D. A., Pantic, K., & Kafai, Y. B. (2015). "I have a tutorial for this": The language of online peer support in the Scratch programming community. In Proceedings of the 14th International Conference on Interaction Design and Children (pp. 229-238). ACM.
French, D. J., Stone, B., Nysetvold, T. T., Hepworth, A., & Red, W. E. (2016). Collaborative Design Principles From Minecraft With Applications to Multi-User Computer-Aided Design. *Journal of Computing and Information Science in Engineering, 16*(2).

Gee, J. P. (2008). Learning and games. In K. Salen (Ed.), The ecology of games: Connecting youth, games, and learning (pp. 21–40). Cambridge, MA: MIT Press

Hickey, D.T., Ingram-Goble, A.A., & Jameson, E.M. (2009). Designing assessments and assessing designs in virtual educational environments. J Sci Educ Technol 18(2):187–20

Lave, J. & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. Cambridge.

Lenhart, A., Kahne, J., Middaugh, E., Macgill, E. R., Evans, C., & Vitak, J. (2008). Teens, video games, and civics. Washington, DC: Pew Internet & American Life Project.

Li, M. C., & Tsai, C. C. (2013). Game-based learning in science education: A review of relevant research. *Journal of Science Education and Technology, 22*(6), 877-898.

Makuch, E. 2014. Minecraft passes 100 million registered users, 14.3 million sales on PC. Retrieved from www gamespot com.

Marklund, B. B., Backlund, P., & Johannesson, M. (2013). Children's collaboration in emergent game environments. In FDG (pp. 306-313).

Nebel, S., Schneider, S., & Rey, G. D. (2016). Mining learning and crafting scientific experiments: a literature review on the use of minecraft in education and research. *J.Ed.Tech & Soc.*, 19(2), 355.

Niemeyer, D.J., & Gerber, H. R. (2015). Maker Culture and “Minecraft”: Implications for the Future of Learning. Educational Media International, 52(3), 216–226. doi:10.1080/09523987.2015.1075103

Paraskeva, F., Mysirlaki, S., & Papagianni, A. (2010). Multiplayer online games as educational tools: Facing new challenges in learning. *Computers & Education, 54*(2), 498-505.

Plass, J., Homer, B., & Kinzer, C. (2015). Foundations of game-based learning. *Educational Psychologist, 50*(4), 258–283. doi:10.1080/00461520.2015.1122533

Ravitch, S. M., & Carl, N. M. (2015). Qualitative research: Bridging the conceptual, theoretical, and methodological. Sage Publications.

Repenning, A., Webb, D. C., Brand, C., Gluck, F., Grover, R., Miller, S., & Song, M. (2014). Beyond minecraft: Facilitating computational thinking through modeling and programming in 3d. *IEEE Computer Graphics and Applications*, 34(3), 68-71.

Rogat, T., & Linnenbrink-Garcia, L. (2011). Socially shared regulation in collaborative groups: An analysis of the interplay between quality of social regulation and group processes. *Cognition and Instruction*, 29(4), 375–415. doi:10.1080/07370008.2011.607930

Short, D. (2012). Teaching Scientific Concepts Using a Virtual World: Minecraft. Teach.Sci. 58(3), 55–58.

Shute, VJ. (2008). Focus on formative feedback. Rev Educ Res 78:153–189. doi:10.3102/0034654307313795

Steinkuehler, C. A. (2004). Learning in massively multiplayer online games. In *Proceedings of the 6th International Conference of the Learning Sciences (ICLS '04)* (pp. 521-528). New York: ACM Press.

Tanes, Z., & Cemalcilar, Z. (2010). Learning from SimCity: An Empirical study of Turkish adolescents. Journal of adolescence, 33(5), 731-739. doi:10.1016/j.adolescence.2009.10.007

Thompson, C. 2016. The Minecraft Generation. Retrieved from www.nytimes.com.

Wernholm, M., & Vigmo, S. (2015). Capturing children's knowledge-making dialogues in Minecraft. *International journal of research & method in education, 38*(3), 230-246.
# How Teachers Implement Active Learning: Typologies of Orchestrational Flow

Elizabeth S. Charles, Dawson College, echarles@dawsoncollege.qc.ca

James D. Slotta, University of Toronto (OISE), jim.slotta@utoronto.ca

Robert Cassidy, Concordia University, rob.cassidy@concordia.ca

Michael Dugdale, John Abbott College, michael.dugdale@johnabbott.qc.ca

Kevin Lenton, Vanier College, lentonk@vaniercollege.qc.ca

Chao Zhang, McGill University, chao.zhang2@mail.mcgill.ca

**Abstract:** The term, Active Learning (AL) is commonly used in describing pedagogical approaches that engages learners actively in classrooms, with an emphasis on problem solving, inquiry and reflection. While a growing body of evidence supports the effectiveness of such an approach, there is great variation in defining the specific strategies and approaches, making it difficult to advance the field. We examined 19 college instructors for over 220 hours using AL methods. We coded the observed classroom activities according to teacher- and student-centered behaviours. On average, these teachers allocated over 50% of class time to group work. A cluster analysis revealed four distinct patterns of student-centred activity: (1) frequent, short duration; (2) longer duration; (3) less frequent, short duration; and, (4) mixed. Two approaches to workflow were identified: 1) tightly orchestrated and 2) front-loaded with less structured periods of work. Results suggest typologies of instructional patterns, growth trajectories and new directions for examining AL.

## Introduction

Most educators now recognize the need to foster “21st century knowledge skills”, such as critical thinking, collaborative problem solving, and evidence-based reasoning (Hargreaves, 2003; Pellegrino & Hilton, 2012). Science educators have responded to this challenge, exploring new modes of learning and instruction, such as peer instruction (PI), where students in large lecture courses respond to multiple choice conceptual problems using “clickers”, with the tallied results introduced as a powerful mediator of whole class discussions (Crouch & Mazur, 2001). While PI methods hold promise for improving large lecture courses, a movement is now underway to reduce large lectures, emphasizing smaller, recitation-sized sections, led by TAs or instructors. In the “flipped classroom” approach, students spend time at home preparing for class by watching video lectures and reading texts, and class time engaging in active forms of problem solving, small group work, tutorial and recitation (Brookfield, 2012).

Referred to broadly as “Active Learning” (Bishop & Verleger, 2013; DeLozier & Rhodes, 2017), this approach has now engaged many STEM educators, resulting in professional societies (e.g., SALTISE.ca) and university-based centers to support the design of active learning courses. Pioneered by Beichner and his colleagues (e.g., Beichner, Saul, Abbott, Morse, Deardorff, Allain & Risley, 2007), the Student-centered Activities for Large Enrolment Undergraduate Physics, or SCALE-UP method, emphasizes small group tables for student engagement with hands-on activities, physical or digital manipulatives, structured and conceptual problems). SCALE-UP was adapted by MIT for its introductory physics curriculum, known as Technology Enhanced Active Learning (TEAL; Dori et al, 2003), which has provided an important referent for many new initiatives in undergraduate science education. Ruiz-Primo, Briggs, Iverson, Talbot and Shepard (2011) summarize active learning (AL) as comprising four dimensions: conceptually oriented tasks, collaboration, technology, and inquiry based projects. Several studies have measured the benefits of AL (e.g., Dori & Belcher, 2005; Linton, Pangle, Wyatt, Powell & Sherwood, 2014). A large meta-analysis of AL in STEM, shows that exams scores improved by 6% and students were 1.5 times less likely to fail compared with traditional lecture approaches (Freeman, Eddy, McDonough, Smith, Okoroafor, Jordt & Wenderoth, 2014). Moreover, AL has been shown to improve critical thinking, motivation and communication skills (e.g., Kim, Sharma, Land & Furlong, 2013), and increase students’ use of metacognitive processes such as decision making and questioning (Lin, Hmelo, Kinzer & Secules, 1999).

Despite this evidence of success, however, AL remains largely ill-specified and difficult to study with any control (Ruiz-Primo et al., 2011; Brownell, Price & Steinman, 2013). For example, while specific group strategies are often invoked (e.g., group work, gallery walks, collaborative projects, or problem solving) very little definition is provided about the learning processes within those groups, the materials or assessments, or the instructor's role during the activities (Henderson & Dancy, 2007). Simply naming collaborative strategies fails to
provide sufficient detail about the content, structure or sequencing of activities, nor how they must be configured to ensure efficacy. What makes a hands-on lab activity effective? When should it be used within the AL sequence? How will students collaborate, and to what end? How should design projects be structured, and how should they be assessed? Additionally, AL instruction places new demands on teachers' management and responsibilities for learning, referred to by Dillenbourg (2013) as *classroom orchestration*. Practitioners and researchers require more detail about AL designs, in order to develop a deeper understanding of how to create or adapt them for their own learning designs.

The recent interest in building innovative “Active Learning classrooms” (ALCs) offer an opportunity to gain insight into the instructional patterns used by instructors who purport to engage in AL. Instruction that takes place in these classrooms relies heavily on collaborative learning and student projects, providing opportunities to observe the corresponding teaching practices in a “natural habitat”. By examining the nature of teaching and learning in such environments, we can gain an understanding of what instructors are doing and how they integrate AL strategies into their courses. Are there meaningful patterns and/or trends? Are they intentionally designed to produce specific learning and/or instructional outcomes?

We conducted a study of AL “in the wild” to inform our understanding of active learning, and to improve our own approaches to AL design. Our observations and survey of 19 instructors, reported in this paper, can serve as an important referent, informing how we make sense of AL in terms of its patterns of activity structure, discourse and interactions. In many ways, examination of these authentic implementations are akin to what Bereiter (2014) calls for when he described principled practical knowledge, “a type of knowledge that has characteristics of both practical know-how and scientific theory” (p. 5). This research thus aims to inform theoretical perspectives on AL, as well as instructional design and principled instructional practices (authors, 2010).

## Methods

### Design and data collected

This research used a case study design (Yin, 2013), involving mixed methods and ethnographic approach (Denzin & Lincoln, 2000), and part of a larger study involving student group practices and artifact production. The data used included classroom observations consisting of field notes, video recordings, teacher interviews and the pedagogical commitment self-reporting survey - i.e., Post-secondary Instructional Survey (PIPS; Walter, Henderson, Beach & Williams, 2016). PIPS consist of 24 items categorized into student-centred and teacher-centred statements, that indicate a score describing a teacher's practice along the two factors. Two indices were generated from PIPS scores: A student-centred index (SCI) was created from the ratio of the instructor's student-centred score and teacher-centred score; and classification of SC (student-centeredness) based on the SCI such that Level 1 included SCI less than 2, Level 2 included SCI from 2 to 3.5, and Level 3 included SCI greater than 3.5.

### Participants and setting

The study used a purposeful sampling to recruit a total of 19 instructors from three colleges, situated in a metropolitan city in eastern Canada. This sample was selected from a total population of approximately 70 teachers across the three colleges, who voluntarily select to teach their one or more of their courses in active learning classrooms. Their data corpus represents 33 course sections, from 13 courses, and eight disciplines - Physics, Chemistry, Biology, Mathematics, Psychology, History, Humanities and English. Class sizes ranged from 15 to 47 students. All 19 participants had taught in these types of classrooms multiple times, ranging from 5 - 12 semesters (Mean = 9 semesters). Additionally, each institution has some level of support and expectation that teachers using these classrooms will use a student-centred approach and engage with an appropriate professional learning community.

### Procedure and analysis

Observations were conducted by members of the research team, carefully following a common protocol that included field notes and video recordings using GoPro cameras to document the teacher's activity. Multiple class sessions were observed for each teacher, based on an established schedule that set conditions - e.g., accommodation of the teacher preferences, minimum number of classes to observe, timing between observations. In all instances, efforts were made to schedule observations to ensure obtaining a representative sampling of the teacher's practices and their instructional implementations (avg. # of observations = 8.3).

Quantitative data analysis was performed using R (v3.4.1), an open source analytic software environment. Analysis of the classroom observations was performed using StudioCode and a protocol similar to
the COPUS approach (Classroom Observation Protocol for Undergraduate STEM; Smith, Jones, Gilbert & Wieman, 2013). However, our coding sought to elaborate on the types of classroom behaviours during the class session, as well as to document the amount of time spent in each kind of activity. *Capturing the distinct kinds of instructional interaction* -- including various forms of lecture, small group, and individual learning activities -- was a primary goal, with the aim of providing a clear account of the frequency, duration, sequence and mix of interactions that occur. Each class session was coded for presence of the following categories: (1) teacher-centered (lecture/demo); (2) student-centered (group/individual/whole class/student presentation); and, (3) other (administrative work). Actual time spent in each activity was then calculated and recorded for each teacher. We also recorded the type of classroom as being either high-tech “active learning space” (e.g., interactive writable surfaces) versus low-tech “active learning space” (e.g., traditional whiteboards and writing surfaces), in order to investigate whether this variable influenced the nature of classroom interactions.

## Results

Entering into this effort, we recognized that instructors would vary appreciably in how they put the ideas of AL into practice, and for purposes of this paper, we include any student-centred instruction occurring within these classrooms as a form of active learning. A total of 157 observations were obtained, with average class times of 85 minutes, and an average of 8.3 observations per teacher.

We found that instructors spent 59% of their time in student-centred activities (ranging from a low of 24% to a high of 76%), and 40.8% in teacher-centred activities (see Figure 1). A box-plot of these observational data (Figure 2) reveals that student-centred approaches were the primary instructional mode for 15 of the 19 teachers observed. This is in clear contrast to other studies such as Lund et al. (2015) whose sample of 73 teachers spent less than 20% of class time in student-centred activity, with lecture as the primary instructional mode. Even in cases when professional development (PD) training was provided (e.g., Stains, Pilarz & Chakraverty, 2015), lectures remained above 60%, and returned to 80% within two years of the PD intervention. Thus, it is encouraging that AL-oriented instructors exhibit such a clear pattern of student-centred learning designs.

<figure><img src="image_171.png" /><figcaption>Figure 1. Average proportion of class time spent on different activities by the 19 teachers.</figcaption></figure>

<figure><img src="image_172.png" /><figcaption>Figure 2. Box plot comparisons of the 19 instructors, ranking by time spent in student centered activities within their observed classes. Horizontal lines are grand means for each activity type.</figcaption></figure>
## What factors impact instructional patterns?

To determine whether the observed patterns of instruction depended on particular characteristics of the teacher or classroom, a MANOVA was performed with activity type (lecture, group work, individual work, student presentation, whole-class discussion) as the dependent variable. Independent variables were teacher, classroom type (high-tech ALC or low-tech ALC), course section, course content and semester (fall or winter session). There were no main effects of classroom type (low-tech or high-tech), course section, or semester, nor significant interaction effects, however there was a significant main effect of teacher, with Pillai's Trace = 1.23 ($F_{(18, 131)}$ = 2.37, p < 0.05). This is unsurprising, as different teachers would likely have characteristic instructional patterns, which would give rise to such a significant effect. To further test whether teachers' instructional patterns are stable across contexts, we examined the 7 teachers who were observed multiple times in different contexts. For each of these teachers, a MANOVA was performed to examine whether or not the different contexts led to different instructional patterns. In all but one teacher, no statistically significant difference was found, suggesting that teachers within our cohort use similar instructional patterns regardless of the course context. Post-hoc analysis of the one exception indicated that the difference is attributable to a small dependence of "student presentation" activities in the observed course. This overall lack of variation across contexts suggests that it is safe to aggregate our observations across context for the remaining analyses.

We sought to further understand the characteristics of these active learning designs by focusing on the most commonly occurring form of student-centered activity: group work. One question that could be addressed from our observation data was concerned with the average duration of groupwork. We performed a *partitioning around medoids* cluster analysis (Reynolds, Richards, de la Iglesia & Rayward-Smith, 1992). Findings show that 18 of the 19 teachers fall into one of four clusters (Figure 3). Cluster 1 indicates very frequent but short activity sessions; cluster 2, moderately frequent but long activity sessions; and Cluster 3 shows moderately frequent but shorter activity sessions; finally the two teachers in Cluster 4 employed moderately frequent and moderately long activity sessions.

<figure><img src="image_173.png" /><figcaption>**Figure 3.** Cluster analysis revealing four clusters of student-centred pedagogy organized by % of class time devoted to group work and average length of group work assigned within class session.</figcaption></figure>

Looking within clusters, we found different orchestrational flows. In Cluster 1 teachers were generally observed as designers of many short duration activities engaging students in collaborative tasks that included group problem solving, peer review and editing, peer instruction (reciprocal teaching), orchestrated by task termination (or interruption) after 10-15 minutes to provide feedback at the class-level. Observations of teachers in Cluster 2 revealed longer group activities with orchestration involving the teacher circling around the classroom, providing feedback to individual groups. Generally, such activities engaged students in project work, sometimes with a distribution of labour and sometimes working jointly to build and/or add to a shared artifact. Cluster 3 teachers orchestrated with shorter activities and fewer of them, appearing to have less experience with
student-centred instruction or teaching in disciplines with lots of content to learn (e.g., biology, history). Much like the earlier cluster, these tasks engaged students in problem solving and peer instruction. Cluster 4 is more difficult to describe because the two teachers appear to engage in different orchestrational flows or styles, teacher 8 being more aware of using group work but hampered because of the student under preparedness (described in interviews).

## What characteristics are shared by teachers within the clusters?

To understand the factors that might account for these clusters, we examined characteristics of the 19 teachers (see Table 1). We found that Cluster 1 teachers were predominantly from a common discipline (physics), as compared with the other clusters, which were more heterogeneous. Cluster 1 teachers also generally shared a high commitment to student-centred interactions like small group work, with 4 out of the 6 cluster members at Level 3 on the SCI classification. We examined other factors, such as years of teaching or experience in AL classrooms, but those did not vary systematically (i.e., in any way that could offer explanation for the clusters). Still, in Cluster 3 we recognized that two of the disciplines found there (biology and chemistry) have a strong tradition of focusing on content at the undergraduate levels, particularly as these were introductory courses with multiple sections, where teachers expressed feeling external pressures to prepare students for the common final. This could may explain why their SC index scores are high compared to their actual implementation patterns – i.e., cluster allocation.

<table><caption>Table 1: Breakdown of individual teachers information organized by their association to the cluster analysis</caption><thead><tr><th>cluster</th><th>Teacher ID*</th><th>Teaching experience</th><th>AL experience</th><th>SCI</th><th>SCI Level</th><th>Discipline</th></tr></thead><tbody><tr><td>1</td><td>9</td><td>11</td><td>3</td><td>6.1</td><td>3</td><td>physics</td></tr><tr><td>1</td><td>10</td><td>9</td><td>2</td><td>3.5</td><td>3</td><td>physics</td></tr><tr><td>1</td><td>11</td><td>9.5</td><td>2</td><td>4.5</td><td>3</td><td>physics</td></tr><tr><td>1</td><td>12</td><td>17</td><td>3</td><td>10.4</td><td>3</td><td>physics</td></tr><tr><td>1</td><td>15</td><td>11</td><td>2</td><td>3.4</td><td>2</td><td>physics</td></tr><tr><td>1</td><td>18</td><td>n/a</td><td>n/a</td><td>n/a</td><td>n/a</td><td>math</td></tr><tr><td>2</td><td>13</td><td>15</td><td>1</td><td>4.6</td><td>3</td><td>english</td></tr><tr><td>2</td><td>14</td><td>7.5</td><td>2</td><td>2.5</td><td>2</td><td>sociology</td></tr><tr><td>2</td><td>16</td><td>13</td><td>3</td><td>2.8</td><td>2</td><td>physics</td></tr><tr><td>3</td><td>2</td><td>7.8</td><td>2</td><td>1.8</td><td>1</td><td>chemistry</td></tr><tr><td>3</td><td>3</td><td>6</td><td>1</td><td>1.6</td><td>1</td><td>psychology</td></tr><tr><td>3</td><td>4</td><td>9.5</td><td>2</td><td>2.0</td><td>1</td><td>physics</td></tr><tr><td>3</td><td>6</td><td>16</td><td>2</td><td>10.6</td><td>3</td><td>biology</td></tr><tr><td>3</td><td>7</td><td>23</td><td>2</td><td>3.1</td><td>2</td><td>sociology</td></tr><tr><td>3</td><td>17</td><td>9</td><td>1</td><td>n/a</td><td>n/a</td><td>history</td></tr><tr><td>3</td><td>19</td><td>11</td><td>2</td><td>n/a</td><td>n/a</td><td>biology</td></tr><tr><td>4</td><td>5</td><td>15</td><td>2</td><td>1.8</td><td>1</td><td>chemistry</td></tr><tr><td>4</td><td>8</td><td>10</td><td>2</td><td>5.4</td><td>3</td><td>physics</td></tr></tbody></table>

Notes: Teaching experience, numbers of years teaching; AL experience (proxy used: request to teach in AL classrooms) 1 = low (>2 yrs), 2 = medium (3-5 yrs), 3 = high (> 5 yrs); SCI, student-centred index, see Methods section for details; SCI Level, see Methods section for details. *Teacher 1 not presented here, b/c not part of cluster analysis.
## Characterizing orchestrational flows

Several studies have focused on teachers' pedagogical implementation, which is sometimes referred to as "pedagogical flow" (Schmidt, 2007), and more recently subsumed within the broader notion of classroom orchestration (Dillenbourg, 2013). Inspired by these notions, we explored the implementation patterns of our cohort of teachers, selecting one teacher from the three main clusters (i.e., Clusters 1, 2 and 3). In an effort to minimize differences in implementation that might be associated with content, we selected the three teachers who teach the same discipline (physics) and specific course (Mechanics), but who fell into different clusters. Because we conducted multiple observations for each of these teachers, and the timeline descriptions for each observation were somewhat distinct (i.e., based on topic of the day and what the students were asked to do), we had to develop some measure of a teacher's "characteristic implementation," which we obtained by selecting the most representative of their respective implementations. This is captured through a pair of graphs - a pie chart to show total proportions of time spent in different forms of interaction (i.e., small group, lecture, or demonstration), combined with a timeline showing the distribution of those interactions over a representative class period. Figure 4 shows the three sample teachers, whose respective pedagogical flow can be seen reflecting the three distinct clusters.

<figure><img src="image_174.png" /><figcaption>**Figure 4.** The orchestration of AL by three teachers, each representing one of the three main clusters identified. Teacher Phy_9, is representative of cluster 1, teacher Phy_16, of cluster 2, and teacher Phy_4, of cluster 3. Colors, blue=group work; red=lecture; red stripe=demonstration; black=administrative work.</figcaption></figure>

Interestingly, while teachers Phy_9 and Phy_16 allocate over 60% of their class time to group work there is a stark difference in their implementations. Phy_9 has a "stop and go" pattern that is a heterogeneous mix of lecture and student group work – creating a rhythm almost like an up-tempo musical piece. Meanwhile, Phy_16 shows a two-stage pattern that is "front-loaded" with the teacher lecture and demo followed by a prolonged period of group work. Teacher Phy_4 illustrates the near inverse of the Cluster 1 teacher. This pattern too is "stop and go" but substantially less time is allocated to group work.

Interviews with these teachers also revealed differences in their instructional objectives and pedagogical commitments. Teacher Phy_9 stated that he had a pair of aims: (1) to guide students through a series of activities that engage them in exploring the concepts, but also (2) to maintain control over the student's interaction with the content as a means of giving them an opportunity to apply their knowledge but then moving them along. Teacher
Phy_16 stated that his aim was to allow students to explore the content, and preferred to work with the groups themselves to better understand the individuals and their concerns. The interview with teacher Phy_4 revealed that while he was also interested in having students try out what they were learning, he was more concerned with moving things along and controlling “the chaos that comes with group work” in active learning classrooms.

## Discussion

Under the assumption that this cohort of 19 college teachers is adequately representative of student-centred teachers, we set out to examine their practice and implementation of AL strategies. Examining the allocations of time to different instructional activity revealed four patterns: high group work with frequent interruptions (Cluster 1), high group work with few interruptions (Cluster 2), moderate group work with interruptions (Cluster 3), mixed (Cluster 4). The Cluster 2 pattern suggested a front-loading of lecture before handing over longer durations of time for students to work on activities. Such implementations appear similar to inquiry-based approaches. On the other hand, Cluster 1 and 3 patterns revealed types of “stop and go” tempos, or what might be considered a heterogeneous mix of lecture and group work – though clearly Cluster 1 is more student-centred.

These identified patterns offer new opportunities to extend the CSCL research on orchestration in empirical ways. For instance, the two types of AL patterns call for very different management of resources and learning – group level versus class level. Additionally, the patterns of Clusters 1 and 3 offer opportunities to explore how and what teachers do in interweaving the lecture and group work and how they integrate AL strategies. The limits of this paper do not allow for the expansion of this investigation but interviews with these teachers reveal clearly different intentions behind these designed implementations. Lastly, the inverse of patterns identified between Cluster 1 and 3 raises the question of whether or not these might not be representative of growth trajectories along the continuum of AL adoption. As such, we might further ask whether or not teachers adoption of this new approach takes on such trajectories and could provide insights on professional development. This study has begun the process of shaping what is to be explored as CSCL and the Learning Sciences begins to consider AL as an instructional practice. We argue that by examining authentic implementations we gain a better understanding of how collaborative learning is orchestrated to meet the needs of integrating group work and lecture, and more importantly, how our findings might be adopted by everyday practitioners. Such investigations also bring us closer to understanding how we might follow up on Bereiter’s challenge of designing with *principled practical knowledge* (Bereiter, 2014).

## References

* Bereiter, C. (2014). Principled practical knowledge: Not a bridge but a ladder. *Journal of the Learning Sciences*, 23(1), 4-17.

* Beichner, R. J., Saul, J. M., Abbott, D. S., Morse, J. J., Deardorff, D., Allain, R. J., & Risley, J. S. (2007). The student-centered activities for large enrollment undergraduate programs (SCALE-UP) project. Research-based reform of university physics, 1(1), 2-39.

* Bishop, J. L., & Verleger, M. A. (2013, June). The flipped classroom: A survey of the research. In ASEE National Conference Proceedings, Atlanta, GA (Vol. 30, No. 9, pp. 1-18).

* Brookfield, S. D., & Preskill, S. (2012). Discussion as a way of teaching: Tools and techniques for democratic classrooms. John Wiley & Sons.

* Brownell, S. E., Price, J. V., & Steinman, L. (2013). Science communication to the general public: why we need to teach undergraduate and graduate students this skill as part of their formal scientific training. *Journal of Undergraduate Neuroscience Education*, 12(1), E6.

* Coburn, C. E., & Penuel, W. R. (2016). Research-practice partnerships in education: Outcomes, dynamics, and open questions. *Educational Researcher*, 45(1), 48-54.

* Crouch, C. H., & Mazur, E. (2001). Peer instruction: Ten years of experience and results. American journal of physics, 69(9), 970-977.

* DeLozier, S. J., & Rhodes, M. G. (2017). Flipped classrooms: a review of key ideas and recommendations for practice. *Educational psychology review*, 29(1), 141-151.

* Denzin, N. K., & Lincoln, Y. (Eds.). (2000). Handbook of qualitative research (2nd ed.). Thousand Oaks, CA: Sage Publications.

* Dillenbourg, P. (2013). Design for classroom orchestration. *Computers & Education*, 69, 485-492.

* Dori, Y. J., & Belcher, J. (2005). How does technology-enabled active learning affect undergraduate students' understanding of electromagnetism concepts?. The journal of the learning sciences, 14(2), 243-279.

* Dori, Y. J., Belcher, J., Bessette, M., Danziger, M., McKinney, A., & Hult, E. (2003). Technology for active learning. *Materials Today*, 6(12), 44-49.
Edelson, D. C., Gordin, D. N., & Pea, R. D. (1999). Addressing the challenges of inquiry-based learning through technology and curriculum design. *Journal of the learning sciences*, 8(3-4), 391-450.

Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. *Proceedings of the National Academy of Sciences*, 111(23), 8410-8415.

Hargreaves, A. (2003). Teaching in the knowledge society: Education in the age of insecurity. Teachers College Press.

Henderson, C., & Dancy, M. H. (2007). Barriers to the use of research-based instructional strategies: The influence of both individual and situational characteristics. Physical Review Special Topics-Physics Education Research, 3(2), 020102.

Kim, K., Sharma, P., Land, S. M., & Furlong, K. P. (2013). Effects of active learning on enhancing student critical thinking in an undergraduate general science course. *Innovative Higher Education*, 38(3), 223-235.

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. *Educational psychologist*, 41(2), 75-86.

Lin, X., Hmelo, C., Kinzer, C. K., & Secules, T. J. (1999). Designing technology to support reflection. *Educational Technology Research and Development*, 47(3), 43-62.

Linton, D. L., Pangle, W. M., Wyatt, K. H., Powell, K. N., & Sherwood, R. E. (2014). Identifying key features of effective active learning: the effects of writing and peer discussion. CBE-Life Sciences Education, 13(3), 469-477.

Lund, T. J., Pilarz, M., Velasco, J. B., Chakraverty, D., Rosploch, K., Undersander, M., & Stains, M. (2015). The best of both worlds: building on the COPUS and RTOP observation protocols to easily and reliably measure various levels of reformed instructional practice. CBE-Life Sciences Education, 14(2), ar18.

Pellegrino, J. W., & Hilton, M. L. (2012). Committee on Defining Deeper Learning and 21st Century Skills. Center for Education.

Reynolds, A., Richards, G., de la Iglesia, B. and Rayward-Smith, V. (1992) Clustering rules: A comparison of partitioning and hierarchical clustering algorithms; Journal of Mathematical Modelling and Algorithms 5, 475–504 (http://dx.doi.org/10.1007/s10852-005-9022-1).

Ruiz-Primo, M. A., Briggs, D., Iverson, H., Talbot, R., & Shepard, L. A. (2011). Impact of undergraduate science course innovations on learning. Science, 331(6022), 1269-1270.

Schmidt, W. H. (Ed.). (2007). *Characterizing pedagogical flow: An investigation of mathematics and science teaching in six countries*. Springer Science & Business Media.

Smith, M. K., Jones, F. H., Gilbert, S. L., & Wieman, C. E. (2013). The Classroom Observation Protocol for Undergraduate STEM (COPUS): a new instrument to characterize university STEM classroom practices. *CBE-Life Sciences Education*, 12(4), 618-627.

Stains, M., Pilarz, M., & Chakraverty, D. (2015). Short and long-term impacts of the Cottrell scholars collaborative new faculty workshop. Journal of Chemical Education, 92(9), 1466-1476.

Walter, E. M., Henderson, C. R., Beach, A. L., & Williams, C. T. (2016). Introducing the Postsecondary Instructional Practices Survey (PIPS): a concise, interdisciplinary, and easy-to-score survey. CBE-Life Sciences Education, 15(4), ar53.

Wieman, C. E., Adams, W. K., & Perkins, K. K. (2008). PhET: Simulations that enhance learning. Science, 322(5902), 682-683.

Yin, R. K. (2017). *Case study research and applications: Design and methods (6th ed.).* Thousand Oaks, CA: Sage Publications.

## Acknowledgments

This research was supported by a grant from le Ministère de l'éducation et de l'enseignement supérieur, Programme d'aide à la recherche sur l'enseignement et l'apprentissage (PAREA), Quebec, Canada (grant number PA2014-013). And, in collaboration with our colleagues from the SALTISE network <https://www.saltise.ca/>.
# Cross-Boundary Interaction for Sustaining Idea Development and Knowledge Building with Idea Thread Mapper

Guangji Yuan, Jianwei Zhang, and Mei-Hwa Chen

gyuan@albany.edu, jzhang1@albany.edu, mhc@cs.albany.edu

University at Albany, State University of New York

**Abstract:** This study investigated the cross-classroom interaction in four grade 5 knowledge building communities as the third iteration of design-based research. The four classrooms studied human body systems with the support of the Idea Thread Mapper platform. As the students conducted focused inquiry and discourse within their own community, they posted reflective syntheses of their “Journey of Thinking” to a cross-community space accessible to the broader peers from other classrooms. Students from the four classrooms further participated in a live “Super Talk” to collaboratively address a challenging question. Qualitative analyses of classroom videos, online discourse, and classroom observation notes provide a rich description of how the students interacted within and across communities to develop scientific understandings.

## Introduction

Educational researchers have made substantial advances to develop collaborative learning environments and support collaborative knowledge building among students (Engle & Conant, 2002; Scardamalia & Bereiter, 2006; Slotta, Suthers, & Roschelle, 2014). Further research is needed to extend collaborative interaction to higher social levels and scales (Stahl, 2013), so students can build on the knowledge of other communities across school years for sustained knowledge building (Zhang, Bogouslavsky & Yuan, 2017). This requires designs for cross-boundary interaction and collaboration, which has gained interests in the broader fields (Star & Griesemer, 1989; Start 2013) but still lacks systematic investigation in the field of computer-supported collaborative learning. The purpose of this design-based research is to test and refine designs of cross-classroom interaction for knowledge building with the support of a new collaborative platform: Idea Thread Mapper (Zhang et al., 2015, Zhang et al., 2018).

Existing explorations of cross-community interactions for knowledge building adopted a single layer interaction design. Each classroom gave other classrooms access to their online discourse space so they could read their notes, respond, and supplement with periodic video conferences (Lai & Law 2006; Laferrière et al., 2012). The findings suggest productive classroom changes. Meanwhile, difficulties arose for students to understand other communities' ideas and discourses without a clear sense of the contexts. New strategies of boundary-crossing support are needed to make knowledge progress accessible across communities.

This research designs cross-community interaction using a multi-level emergent interaction approach, focusing on interactions mediated through boundary objects. “Boundary objects” are artifacts (e.g. reports, tools, models) used to bridge the boundaries (discontinuities) between different social worlds (Star & Griesemer, 1989). Objects from a community often have contextual meanings that are not accessible to other communities. What makes boundary objects effective for bridging different communities of practice is their interpretative flexibility as a “means of translation” (Star & Griesemer, 1989): they have a structure that is common enough to make them recognizable across the different social worlds and allow different communities to interact and work together. Through interacting with shared boundary objects, members from different communities can identify, understand, and reflect on their different practices, leading to enriched understandings within each community and potentially the creation of new, in-between practices (Akkerman & Bakker, 2011).

As noted above, raw distributed online discourse records are hard to use as boundary objects to bridge different knowledge building communities. Therefore, we developed a multi-level emergent interaction approach (Zhang et al., 2017). Students in each community engage in focused inquiry and interactive discourse within their own community's space. As progress is made, students selectively synthesize fruitful threads of inquiry emerged from their discourse. The reflective reviews and syntheses can facilitate peer learning and build-on within each classroom (Zhang et al., 2018); they may further be shared as boundary objects to enable cross-community interaction. Students from other partner classrooms (or a subsequent student cohort) can use the syntheses of idea threads to view into the progress of inquiry and build connections.

We have conducted a multi-year design-based research to test and refine this multi-level, emergent interaction design (Zhang et al., 2017, Yuan, Zhang & Luo, 2018). The earlier iterations/designs were supported by purposeful uses of existing tools offered by Knowledge Forum (Scardamalia & Bereiter, 2006).
On the basis of the findings, we created and upgraded a new collaboration platform: Idea Thread Mapper (ITM) (Zhang et al., 2018), which interoperates with Knowledge Forum. ITM supports knowledge building interaction in a network of “buddy classrooms.” Students in each classroom co-organize “juicy” wondering areas (shared inquiry foci) as they pursue interactive discourse to deepen their understandings in a domain area. They compose Journey of Thinking (JoT) syntheses to review progress in each area, focusing on (a) overarching topics and problems, (b) we used to think...now we understand... and (c) deeper research is needed. The wondering areas and syntheses are shared in a cross-classroom space that has an easy search tool. By viewing other classrooms’ JoT syntheses, students can get a sense of their inquiry directions and learn from their “big ideas” and deepening questions. As a newly designed feature, ITM allows different classrooms to initiate and participate in live “Super Talk” threads to collaboratively address challenging problems.

This paper reports on a new iteration of this design-based research to refine cross-classroom interaction in a set of four grade 5 classrooms supported by ITM. Our research questions ask: (a) How did students generate JoT syntheses for cross-classroom sharing? (b) How did the four classrooms initiate and pursue “Super Talk” to address challenging problems? And (c) in what ways did the within-classroom discourse and cross-classroom “Super Talk” interact to support deep inquiry and understandings?

## Method

### Classroom design and contexts

This study was conducted in four grade 5 classrooms (with a total of 89 students who were 10-to-11 years old) that studied human body systems over a six-month period using ITM. The four classrooms, labeled as Class 1-4, were taught by two teachers each teaching two classes. Students in each classroom generated interest-driven questions, co-organized wondering areas focusing on various human body systems, and conducted research using various resources. They conducted reflective knowledge-building conversation (called “metacognitive meetings”) in their classroom to build on one another’s questions and ideas while reviewing their progress. The conversation continued on ITM in their online discourse space organized as various idea threads each addressing an overarching problem/theme. As progress is made in each idea thread, students co-created and edited JoT to reflect on their knowledge (Figure 1). The JoT was then shared with all the other classrooms. Drawing upon their knowledge built about the various body systems, students in Class 3 proposed a challenging problem for “Super Talk” across the classrooms: How do people grow? (Figure 3). The proposal was supported by the other classrooms. Students from the four classrooms worked together to discuss this overarching question. Near the end of the unit, each class had a metacognitive meeting to review knowledge gained from the “Super Talk” and build connections with the different human body systems.

### Data sources and analyses

The data sources included classroom observation notes, video records and transcriptions of classroom conversations, and students' online discourse on ITM and JoT shared in the cross-classroom space. Guided by each of the three research questions, we conducted detailed qualitative analysis of student inquiry process in their home class in connection with student interaction across classrooms. To analyze the quality of students' JoT syntheses, we conducted content analysis (Chi, 1997) based on the scientific levels (from pre-scientific to scientific) and complexity of ideas (from unelaborated facts to elaborated explanations) (see coding scheme in Zhang et al., 2007). To understand how the Super Talk took place, we read/re-read our detailed field notes and examined the classroom videos to understand the classroom processes. To understand how the within-classroom discourse and cross-classroom Super Talk interacted to support further inquiry, researchers selectively transcribed two metacognitive meeting videos from each class, one from the 5th month, and the other from the sixth/last month of the inquiry in which students discussed what they had learned from the Super Talk. We analyzed the classroom discussion to trace the formation of cross-topic connections based on the betweenness centrality measure (Oshima, Oshima & Matsuzawa, 2012).

## Results

### How did students generate syntheses of Journey of Thinking for cross-classroom sharing?

Students engaged in kick-off activities that elicited their interests about the human body in January 2018. Students in each room generated various interests and questions and formulated overarching “wondering areas.” These areas became the focus of the subsequent inquiry by individuals and groups using books, online resources, and models. With inquiry progress made in the next two months, students were introduced to the JoT
function in ITM. Focusing on each area of inquiry, students added reflections on their research individually, and then compiled the reflection entries into whole group JoT (Figure 1). The syntheses of JoT were then shared with all the other classrooms in the cross-classroom sharing space.

<figure><img src="image_175.png" /><figcaption>**Figure 1.** The synthesis written for the area of bones and muscles from Class 1. Journey of Thinking</figcaption></figure>

To examine the quality of students' reflections in their JoT, researchers coded the ideas summarized under “We used to think” and “We now understand” based on scientific sophistication and epistemic complexity (Zhang et al., 2007) (see Table 1).

<table><caption>**Table 1:** Coding of ideas in the JoT summarized under “We used to think” and “Now we understand” based on data from the current (2018) and the previous iteration (2017) of this research</caption><thead><tr><th rowspan="2"></th><th colspan="2">2018</th><th colspan="2">2017</th></tr><tr><th>We used to think</th><th>Now we Understand</th><th>We used to think</th><th>Now we Understand</th></tr></thead><tbody><tr><td><strong>Scientific Sophistication</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Pre-scientific</td><td>33%</td><td>0%</td><td>38%</td><td>0%</td></tr><tr><td>Hybrid</td><td>50%</td><td>0%</td><td>48%</td><td>0%</td></tr><tr><td>Basic</td><td>16.7%</td><td>0%</td><td>14%</td><td>18%</td></tr><tr><td>Scientific</td><td>0%</td><td>100%</td><td>0%</td><td>82%</td></tr><tr><td><strong>Epistemic Complexity</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Unelaborated Facts</td><td>83%</td><td>%</td><td>95%</td><td>9%</td></tr><tr><td>Elaborated Facts</td><td>0%</td><td>16.7%</td><td>0%</td><td>32%</td></tr><tr><td>Unelaborated Explanations</td><td>16.7%</td><td>0%</td><td>5%</td><td>9%</td></tr><tr><td>Elaborated Explanations</td><td>0%</td><td>83.33%</td><td>0%</td><td>48%</td></tr></tbody></table>

As Table 1 shows, ideas summarized under “Now we understand” show a higher level of scientific quality and complexity than those summarized under “We used to think.” Compared against the data from the previous (2017) iteration of this design-based research in which we tested a similar design without ITM, the ideas summarized under “Now we understand” in the current iteration showed higher levels of scientific and complex quality. A more detailed analysis revealed that the JoT syntheses in 2018 had more words on average than those in 2017 (362.3 versus 170.8 words). These changes suggest the potential improvement made by students in their reflective inquiry with the support of ITM.
## How did the four classrooms initiate Super Talk to address challenging problems?

In the beginning of May, at a metacognitive meeting, students in class 3 noticed the new ITM feature of “Super Talk.” The teacher explained that this function was for all the classrooms to explore big challenging questions and put their knowledge together. Then Class 3 started to discuss possible challenging questions for Super Talk. Three questions were proposed in total: How are all the systems connected? Which two systems are most connected? And why do people grow? A few students reflected on what they knew about how muscles grow, and several other students showed interests in the growth topic as they had grown a lot during the school year. Then they agreed to focus on one topic for the Super Talk, and decided to have to vote for the one that they felt was most challenging and exciting. The topic of “How do people grow?” was selected. This Super Talk topic was proposed and added in ITM and made visible to other classrooms (Figure 2).

<figure><img src="image_176.png" /><figcaption>**Figure 2.** Students’ collective wondering areas about human body systems and the Super Talk area.</figcaption></figure>

In the following week, students from Class 3 first started to contribute knowledge about how the brain, bones and muscles grow drawing upon what they had learned about these body systems. The teachers then advertised the “Super Talk” question to the other classrooms. In each case, the teacher read the notes already posted by Class 3, and discussed what might count as a good note for the Super Talk. In the following week, students from the other classrooms started to build on the ideas in the Super Talk (Figure 3).

<figure><img src="image_177.png" /><figcaption>**Figure 3.** The cross-classroom Super Talk about how people grow. Each dot shows a note, and a line between two dots shows a build-on connection. Each note is positioned based on the date of creation (x-axis) and author (y-axis).</figcaption></figure>

Student A from Class 4, studying the digestive system and energy, found the connection between growth and muscles, so he added more detailed information to the Super Talk. He mentioned that muscles use energy from ATP (Adenosine Triphosphate) for muscle placement. And then Student B from Class 2 built on this idea, saying “Muscles grow by when you stress muscle fibers by lifting heavy weights or doing motions you’re not used to, they rip which lets out a chemical called cytokines which activates your immune system which repairs it bigger than it was earlier which makes your muscles grow. Hypertrophy is how your muscles say you need to work more to make your muscles grow.” At the same time, a new angle of viewing was presented by Student C from Class 1, who wrote about how sleep affects growth as related to her inquiry of the
brain. Her post highlighted that during the Non-Rapid Eye Movement stage (NREM), the body is actually repairing damaged tissues and growing. New detailed information about bones was further expanded by Student D from Class 2 who was studying bones. He mentioned that "bone grows from cartilage, they fuse together and go through a process called ossification." Later, his peer who studied the same topic built on this note and added more detailed accounts of the process of ossification: "Over time, a different type of cells called osteoclasts head to the middle of the bone to help in. Now, inside osteoclasts there are hydrolytic enzymes and acids. These enzymes and acids will help dissolve the temporarily bone (the cartilage) to make room for the permanent bone (marrow)." Toward the end of the online discussion, Student E from Class 1, who was studying the endocrine system, gave her explanation from the angle of the endocrine system, because the pituitary gland releases a hormone that controls the growth as it plays a huge role in puberty and metabolism. A cross-cutting connection was further built when Student F from Class 2, who was studying cells, found that humans start as cells and all organs are made of cells, and the way cells grow is from mitosis; therefore, cell growth is key to how humans grow. He imported his earlier note about mitosis from his home class discussion into the Super Talk space. This note was read by other students and triggered deeper conversation in other classrooms' face-to-face meetings.

The Super Talk extended till the end of May with a total of 22 students from the four classrooms participating in the discussion. Students collaboratively explained how people grow involving bone and muscles, brain and nervous systems, cells and genetics, and digestive systems. Approximately 50% of the notes are build-ons, reflecting a higher level of collaborative responses. Student notes were coded based on epistemic complexity (Zhang et al., 2007), with 86% of the notes offering elaborated explanations beyond simple facts.

After students shared their Journal of Thinking and participated in the Super Talk collaboration on ITM, each classroom community continued its inquiry activity in June. The teachers named the last month as the "Month of Connection," and students extended their learning with the focus on the potential connections between different human body systems. Metacognitive meetings were held to discuss how the different body systems work together, integrating what students had learned from their own inquiry as well as from their peers. As specific body systems and connections were mentioned by students, the teachers took visual notes of the discussion, leading to the co-generation of the concept maps shown in Figure 4.

<figure><img src="image_178.png" /><figcaption>**Figure 4.** Concept maps of four classrooms metacognitive meetings about connections between human body systems (Class 1, Class 2, Class 3, Class 4, respectively).</figcaption></figure>

During the metacognitive meeting about connections between different human body systems in the 5th month, students made solid reflections, build-ons, and had deep conversations about how different systems connect to each other. As their discourse expanded, students started to understand the concept of the cell as the basic unit of the human body, however, they did not mention the mechanism of cells multiplication. The excerpt below shows how students from Class 3, which was taught by Teacher K, tried to understand collaboratively the cell as the start and basic unit of human body systems and the connections between various systems. Below is an excerpt from Class 3' metacognitive meeting conducted in the 5th month of this inquiry.

K21: I think the cells in the immune system should be connected to cells that fight off the germs to fight cells

K8: Actually the immune system is made up of cells.
Teacher K: What else is made up of cells?

K8: Blood

K5: Isn't it everything

K8: Yeah, well everything is made up of cells.

K5: Well there are all different kinds of cells

Teacher K: Ah uh!

K1: Yeah, but everything is made up of cells.

K8: It's the beginning of everything

K5: We could have different specific types of cells, maybe we can go to a bunch of different kinds of cells. Like the brain cells just put it next to the brain?

Teacher K: So basically cells is kind of like there is a reason why I drew it in the center. Is it the beginning of every part? Do we have anybody studying cells?

K12: I know something about cells. oh, it's that, it's my dad told me, we all started out as one little cell and we start to grow.

Teacher K: What? Isn't growth being the big question that we put out up there (on ITM)?

K1: Yeah, so we grow teller just by see cells grow in our body

K5: and then it just multiply, so there are tons of different types of cells. and that's why there are the differences, because there is a different amount of cells of certain types.

K15: I think cells are connected to the growth. because as you grow more cells in your body

Teacher K: Every kind of cells

K15: If you get a lot of cells in your body at one type, that's how we growth spurts I think

Teacher K: I think someone may have some growth expert spurts. Is there anyone want to say the big connections that we are missing?

K10: Gene and heredity

K8: So the immune system is connected to the digestive system, skin, eyes.

Teacher K: Really? Tell us why?

K8: So, it is connected to skin cause skin is actually one huge defense against pathogens.

## In what ways did the cross-classroom "Super Talk" leverage deeper discourse in each classroom?

In early June, each class held a face-to-face metacognitive meeting to revisit the question of how people grow. Discourse analysis showed that students brought what they had learned from the "Super Talk" back to their home class discussion and made further connections with the growth of other organs, leading to deepened understandings of the human body systems. Below is an excerpt from Class 3's metacognitive meeting.

Teacher K: Your brain cells are dying? or not making new ones

Student K8: You are not making new ones, but they do start out, they do die as you get older

K5: When you run out of brain cells you die?

K12: I saw something on ITM about chromosomes, it is kind of related to growth.

Teacher: What is it, can you reiterate it? What are chromosomes are related to?

K7: Mitosis?

K5: DNA?

Teacher K: Oh, Mitosis? what about DNA? what's that related to?

K5: DNA and RNA.

Teacher K: What is that related to?

K8: RNA is just half of DNA

K12: Mitosis is the process of one cell splitting into two new cells, it is a complex process of many steps. One prophase. In prophase the structures called centrioles move to opposite ends of the cell and fibers come out of them and enclose the cell. And in metaphase chromosomes line up in the center of
the cell. Each attaches to two fibers. Chromosome halves pull apart the cell and divide the membrane.
*Step three is Anaphase and step 4th telophase.*

Teacher K: He is talking about really deep science that's behind this (pointing to the drawing) where the one cell is splitting into two equal parts. So when you cut an apple, you know, in the center of the apple gets really cut in half, it really does, that's not the same as what is going on here. With mitosis, it gets cut in half but each half gets exactly the same the central part. Like the same center of the apple grows into both pieces, when it splits apart, and then that's...they split apart to make two identical, and it still has that center of the apple, and what's in the center in the apple, or the center of the cell?

K1: The DNA

K2: Chromosomes

Teacher: DNA and Chromosomes, and what can you tell us about heredity or DNA

K1: Hair color, eye color.

K17: Your genes there are like the blueprint.

To specifically examine the conceptual connections built between different topical concepts, researchers traced co-occurrence of the main domain concepts (e.g. brain, cell, bones, lungs, and heart etc.) in each conversation turn in the two metacognitive meetings mentioned above before and after the “Super Talk.” Using the network analysis tool KBDex (Oshima, Oshima & Matsuzawa, 2012), we compared the betweenness centrality of the key concepts, which shows which concepts act as ‘bridges’ between concepts in a network. Figure 5 shows the changing centrality of each key concept over time in each of the two meetings. Before the “Super Talk”, the concept “Brain” stood out as having the highest betweenness centrality among the discussed concepts, suggesting that students’ discourse positioned brain as the central topic connected with other systems. In the meeting after the “Super Talk” activity, the concept of “Cell” had the highest gain in betweenness centrality. Student K12, who acted as broker, brought back the concept of cell mitosis from the “Super talk” and triggered extended discussion related to cells in the home class. According to the science standards, the concepts of cell and mitosis are required by Grade 8 and Grade 9-12 respectively.

<figure><img src="image_179.png" /><figcaption>**Figure 5.** Betweenness centrality of the key concepts discussed in the metacognitive meeting before (Left) and after the “Super Talk” meeting (Right).</figcaption></figure>

## Discussion and conclusions

This study tested a multi-level emergence design of cross-community interaction, with students engaging in focused inquiry and discourse within their own classroom while generating reflective JoT syntheses as boundary-crossing objects, and participating in the “Super Talk” to investigate a challenge problem together. The results provided an elaborated account of the processes to generate reflective syntheses and pursue Super Talk. Students showed solid reflections in JoT with the help of ITM compare with last year and students built on knowledge building interactions through the cross-classroom collaboration to extend and enrich the discourse in their home class. More specifically, with collaboration going on in the cross-boundary space, students further expanded their social connections, engaged in addressing challenging research questions and demonstrated higher level collaboration and understanding beyond their classrooms’ walls. The cross-community collaboration also enabled students to bring back valuable insights to their home class and triggered further build-on and leveraged the home class’ understanding to “rise above” (Scardamalia& Bereiter, 2006). Cross-classroom interactions offer a multi-layer structure which enable the information flow across various sites in a
mutually understanding approach that suggested by Stahl (2013). The analysis of the classroom discourse
reveals the benefits of a higher level of interaction enabled by the updated design of ITM as a knowledge
infrastructure. Expanding our previous year's work (Zhang et al, 2017. Yuan et al., 2018), the findings shed
light on the possible designs and processes to enable collaborative knowledge building across a network of
classrooms in a larger learning environment and ongoing learning process. These research results also provide
insights that may guide teachers' adoption of cross-classroom collaboration to support student inquiry and
extend knowledge building. During the learning process, the teachers played a crucial role in guiding students'
cross-classroom collaborations, embracing the ideas from other communities to extend students' thinking, and
using child-friendly language to leverage students' understanding, and making connections with other topics in
the current classroom.

## References

Akkerman, S. F., & Bakker, A. (2011). Boundary crossing and boundary objects. *Review of educational research, 81*(2), 132-169.

Chen, J., & Zhang, J. (2016). Design Collaborative Formative Assessment for Sustained Knowledge Building Using Idea Thread Mapper. In: *Proceedings of the International Conference of the Learning Sciences (ICLS 2016). Singapore: International Society of the Learning Sciences*.

Chi, M. T. H. (1997). Quantifying qualitative analysis of verbal data: A practical guide. *Journal of the Learning Sciences, 6*, 271-315.

Engle, R. A., & Conant, F. R. (2002). Guiding principles for fostering productive disciplinary engagement: Explaining an emergent argument in a community of learners classroom. *Cognition and instruction, 20*(4), 399-483.

Laferrière, T., Law, N., & Montané, M. (2012). An International Knowledge Building Network for Sustainable Curriculum and Pedagogical Innovation. *International Education Studies, 5*(3), 148-160.

Stahl, G. (2013). Learning across levels. *International Journal of Computer-Supported Collaborative Learning, 8*(1), 1-12.

Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In R. K. Sawyer (Ed.), *Cambridge handbook of the learning sciences* (pp. 97-115). Cambridge University Press.

Scardamalia, M. (2004). CSILE/Knowledge forum®. *Education and technology: An encyclopedia*, 183-192.

Slotta, J., Suthers, D., & Roschelle, J. (2014). CIRCL primer: Collective inquiry and knowledge building. *CIRCL primer series*.

Star, S. L., & Griesemer, J. R. (1989). Institutional ecology, “translations” and boundary objects: Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907–39. *Social Studies of Science*.

Oshima, J., Oshima, R., & Matsuzawa, Y. (2012). Knowledge Building Discourse Explorer: a social network analysis application for knowledge building discourse. Educational technology research and development, 60(5), 903-921.

Yuan, G., Zhang, J.,& Luo, C. (2018). Cross-Classroom Interaction Among Four Grade 5 Knowledge Building Communities. Conference on American Educational Research Association. New York.

Zhang, J., Bogouslavsky, M., & Yuan, G. (2017). Cross-Community Interaction for Knowledge Building in Two Grade 5/6 Classrooms. International Society of the Learning Sciences.

Zhang, J., Chen, M.-H., Tao, D., Lee, J. Sun, Y., & Judson, D. (2015). Fostering Sustained Knowledge Building through Metadiscourse Aided by the Idea Thread Mapper. *International Conference on Computer Supported Collaborative Learning* (Vol. 1, pp. 166-173).

Zhang, J., Scardamalia, M., Lamon, M., Messina, R., & Reeve, R. (2007). Socio-cognitive dynamics of knowledge building in the work of nine- and ten-year-olds. *Educational Technology Research and Development (ETR&D), 55*(2), 117-145.

Zhang, J., Tao, D., Chen, M.-H., Sun, Y., Judson, D., & Naqvi, S. (2018). Co-organizing the collective journey of inquiry with Idea Thread Mapper. *Journal of the Learning Sciences, 27*, 390-430.

## Acknowledgments

This research was sponsored by the Cyberlearning Program of National Science Foundation (#1441479). The views expressed in this article are those of the authors and do not necessarily reflect the views of the funding agency. We thank our collaborating teachers and their students for their creative classroom work enabling this research.
# Embodiment of Computational Thinking During Collaborative Robotics Activity

Theodore J. (TJ) Kopcha, Ceren Ocak

tjkopcha@uga.edu, ceren.ocak@uga.edu

University of Georgia

**Abstract:** This case study explored how young children's thinking was embodied during collaborative robotics activity. Two 5th grade learners were recorded as they worked together to program and navigate a robot across a 3'x3' grid of obstacles. The video was then analyzed using a grounded theory approach (Glaser & Strauss, 1967) to determine how cognition manifested in an embodied manner (e.g., gesturing, use of conceptual metaphor) through the tools and affordances employed by the learners. Four primary tools were used by learners in solving their robotics task: (1) basic robot moves, (2) gestures, (3) the map and the computer, and (4), the grid and the robot. These tools provided symbolic access to direct experiences, supporting an emergent and reactive process in which perception and action were intimately linked. Implications for computer science and improving the design of computer-supported collaborative learning are discussed.

## Introduction

Educators and policymakers currently are interested in integrating computer science into K-12 curriculum. At the heart of computer science is computational thinking, a problem-solving process in which both abstract and automated processes are involved in formulating problems and developing solutions during programming activity (Cuny et al, 2010; Wing, 2006). Learners who engage in computational thinking decompose the larger task into smaller subgoals, use algorithmic thinking and pattern recognition to create new and novel programs that address those subgoals, and debug programs that are not performing optimally (Yadav, Mayfield, Zhou, Hambrusch, & Korb, 2014). These skills are important for developing a competent 21st century workforce, making computational thinking in K-12 settings a critical and current issue (Israel, Pearson, Tapia, Wherfel, & Reese, 2015; Sengupta, Kinnebrew, Basu, Biswas, & Clark, 2013).

One way computational thinking and computer science are brought into K-12 classrooms is through robotics activities. Robotics creates an opportunity for learners to work collaboratively to program a robot and complete various tasks (Yuen et al., 2015). The physical nature of this activity is important. Robotics activity creates an opportunity for learners to make use of various tools and their affordances (e.g., the robot) to connect their abstract thinking with the environment. Thus, learners who engage in robotics activity are likely to experience an embodied experience, where mind, body, and environment are intimately connected (Gibson, 1979) in an effort to set and achieve specific goals with the robot.

However, little is known about how young children's thinking might be embodied during robotics activity. Research on computer science and programming is lacking from K-12 classrooms (Lye & Koh, 2014). Few if any studies of computational thinking examine the process through the lens of embodied cognition (Grover & Pea, 2013). Likewise, there is a need for empirical studies of young children who engage in robotics activity; early success with computer science through robotics has the potential to increase interest in and success with STEM outcomes in future coursework (Sullivan & Bers, 2016).

The purpose of this case study was to explore how young children's thinking was embodied during collaborative robotics activity. Two 5th grade learners were recorded as they worked collaboratively to program and navigate a robot across a 3'x3' grid of obstacles. The video was then analyzed using a grounded theory approach (Glaser & Strauss, 1967) to determine how cognition manifested in an embodied manner (e.g., gesturing, use of conceptual metaphor) through the tools and affordances employed by the learners. This study provides key insight into the ways that learners learn together through the help of digital technologies; examining computational thinking as an embodied phenomenon holds implications for collaborative robotics activities with young learners.

## Overview of ecological embodiment in its implications for collaboration

### Ecological perspective towards embodiment
Gibson's ecological psychology (1979) laid the foundation for our research. An ecological perspective towards embodiment suggests that the body serves as a mediator between perception and action; both perception and action are influenced by the environment (Gibson, 1977, 1979). From this perspective, thought is “a whole-body activity in context than simply an in-the-head process” (p.171). In other words, embodied cognition involves a complex relationship between the mind, body, and the environment. This stands in contrast with our understanding of information processing, where perception and action are treated “separately and sequentially” (Michaels & Palatinus, 2014, p.20). As argued by Richardson, Shockley, Fajen, Riley, and Survey (2009), perception and action are mutual and act together to meet a particular goal. This suggests that thinking is a goal-directed reaction against what is perceived from the environment (Chemero, 2011, Michaels & Palatinus, 2014). In this paper, we draw on the basic premise of ecological psychology to explore children's thinking as a whole-body activity, where the environment guides their decision making.

## Affordances

The concept of affordances is at the forefront of the ecological perspective. Affordances are conceptualized as opportunities or possibilities for action placed in the environment (Rietveld & Kiverstein, 2014). For example, “a flat horizontal surface affords standing and walking, a graspable rigid object affords throw” (Hirose, 2002, p.290). That being the case, the notion of affordance has been gaining prominence in different disciplines such as game design and development, computer science, and learning sciences (Harlow, Dwyer, Hansen, Iveland, & Franklin, 2018; Young, 2013). For instance, Linderoth (2012) used the perception of the affordances in the games and acting on them to understand the gameplay. Hammond (2010) examined the concept of affordance regarding its implications for the use of information and communications technology (ICT) in teaching and learning.

Tools can also serve as an affordance from the perspective of ecological embodiment. Liben (2002) explained how “tools are the functional extension of the environment, it has specific affordances and provides new opportunities for action” (p. 290). Tools can amplify perception to move beyond the physical capabilities of the body. In addition, tools provide opportunities for learners for “active material engagement” (Malafouris, 2013, p. 169). Therefore, the tools become an affordance as the user acts upon them. In this way, the tool and the user of the tool are often considered as a single unit of analysis (see Harlow et al., 2018). That is, the user and the tools become a compound element of a larger system in a cycle of perception and action.

## Embodied collaboration and gestures

Physical gestures are considered strong evidence of embodied cognition (Alibali et al., 2014). Gestures also reveal the features of collaboration in that “gestural communication is based on a common conceptual ground, on the shared knowledge of what ‘we both know together’” (Vasc & Ionescu, 2014, p. 150). Vasc and Ionescu (2014) further established that collaboration in a given medium has two sides: the receiver and the communicator; the receiver knows the meaning that a gesture carries and the communicator uses the gestures with confidence that the receiver would understand since it emerges out of a shared experience. That being the case, learners’ gestures add another layer to understand how collaboration emerges as an embodied phenomenon as pairs of learners mutually act on the affordances they perceive. In this study, we therefore examine the way tools were used as affordances, and how gesturing supported learner’s thinking as they engaged in collaborative problem solving during robotics activity.

## Methods

The video data that was analyzed in this study was collected as part of a larger design research project that focused on developing and testing a robotics unit. The unit was designed as integrative STEM curriculum that supported computational thinking and embodied cognition during collaborative problem solving (see Kopcha et al., 2017). Six 5th grade teachers from a local school setting played an integral role in developing and testing the unit. The unit was enacted over a 2-week period, where learners worked in small groups to navigate a robot through a 3'x3' grid containing various obstacles. As part of the unit, learners explored science and math-related concepts that met specific state standards, including: constructive and destructive forces; operations with fractions; coordinate grid; and algebraic thinking.

Video of Learner activity was recorded in each of the six teacher's classrooms throughout the three class periods in which learners engaged in programming the robot (120 minutes total). Cameras were placed such that small group activity was recorded at both the computer (i.e., when programming) the robot as well as the grid where they executed their program. This provided data for fine-grained analysis of gesturing as well as evidence of the different forms of computational thinking (e.g., problem decomposition, debugging).
To investigate embodied cognition during collaborative robotics activity, we focused on the final 60-minute segment of video in which learners completed the majority of the robotics task. We reviewed the video data from each classroom and ultimately selected a single case to explore our research question. The case was a pair of 5th grade learners (one boy, one girl) who successfully completed the robotics activity. Using a single case allowed for an in-depth and fine-grained analysis of the how the learners used each tool and affordance to successfully complete the robotics task. It also allowed us to better understand the ways that various tools served as an affordance to support successful robotics activity as an embodied collaborative experience.

## Instructional context

The robotics task was designed around the source-path-goal schema (Johnson, 2008). Learners were tasked with helping scientists use a robot to collect three samples from an active volcano. These three samples served as the checkpoints that the robot needed to visit in the overall task. Independent from the what path it followed, Learners were asked to program the robot to stop at each collection then arrive back to its original location. The pair then collaborated to determine an overarching solution path and the decompose the overarching task into smaller subunits.

The active volcano was represented on a 3-by-3-foot grid consisting of 36 six-inch squares. The three collection sites were scattered about the grid, along with a variety of elements associated with an active volcano (e.g., dust plume, flowing lava). Learners were challenged to maneuver the robot around the grid, avoiding the elements of the active volcano and visiting each of the three collection sites. The grid served as a testing site where learners practiced the codes that they wrote.

Computer-supported collaboration took place throughout the activity. Learners were put into pairs and tasked with breaking the task down into sub-goals, programming the robot, and accomplishing as much of the overall task as possible. They repeatedly moved between programming the robot at the computer and testing their programming at the grid (see Figure 2). With every attempt, learners continuously advanced and optimized their calculations to be able to move the robot in the coordinates of the grid precisely.

<figure><img src="image_180.png" /><figcaption>Figure 1. The visual block coding program (left) and the grid representing an active volcano (right).</figcaption></figure>

## Data analysis: A grounded approach

Knowing that gestures carry a semantic value (Alibali, 2005), we transcribed the video and audio data simultaneously for the selected case. Transcripts included the full conversation between learners, paired with short sequences of images from the video to illustrate specific gesturing and learner action. We then followed a grounded theory approach (Glaser & Strauss, 1967), employing an inductive data analysis process to generate our own theoretical understanding of the ways thinking was embodied during robotics activity. A grounded theory approach served the aims of this study because it results in identifying overall patterns that are grounded in data.

Data analysis started with open coding (Johnson & Parry, 2016). During open coding, we tried to be “stay close to the data” (Charmaz, 2006, p.49) by avoiding apriori theoretical presumptions that we had. Our initial codes included: tools/affordances, gestures, and semantics/verbal content. Tools/affordances included the objects in the environment that the learners drew upon as part of their collaborative problem solving, such as the map and the computer and the robot at the grid. Gestures were any hand or body movements made by the learners during collaborative work; this often took the form of imitating the movement of the robot. Semantics/ content included specific aspects of note such as descriptions of specific gestures (e.g., learner uses hands to mimic wheels of robot turning right) or the meaning behind those gestures. These codes were broad enough to begin analysis while leaving room for new codes to emerge.

After the open coding, the researchers came together to identify a core variable from the initial codes that was also related to our inquiry. Our goal was to better understand the tools that were used as affordances by
learners and how their thinking was embodied with and through those tools. Our core variable was therefore the tools that learners drew upon during collaborative problem solving. After identifying our core variable, we returned to the transcripts to engage in selective coding and establish the relationship between the categories surrounding the core-variable (Johnson & Parry, 2016). Using the participants' own words to examine these relationships helped retain the essence of the mutual conversation and understand the affordances and semantic context that framed the gestures (Birks & Mills, 2011). This confirmed that the learners used four primary tools while solving their robotics task (Charmaz, 2006): the robot, the grid, the map and the computer.

Two additional coding categories emerged from this deeper analysis. First, we coded basic robot moves that the learners developed as part of a shared language during their problem solving, such as turning at a right angle and moving one unit square on the grid. Second, we coded specific gestures that also served as a fundamental tool for communicating problem solving. These two codes were strongly related in that the gestures used for problem solving often reflected basic robot moves. This served as an indicator that learners' thinking was distributed onto "external information bearing structures" (p.15) (e.g., hand motions indicating turns or unit movements) to articulate their underlying cognition and improve their problem-solving during computational thinking (Rowlands, 2010).

## Findings

Four primary tools that the learners used while solving their robotics task were: (1) basic robot moves, (2) gestures, (3) the map and the computer, and (4), the grid and the robot. These tools illustrate the ways that thinking was embodied during their collaboration.

### Tool 1: Basic robot moves

The learners began by establishing an overarching solution path. In this case, the overarching solution was to program the robot to visit the three collection sites without hitting the obstacles scattered in the grid (see Figure 2). The learners then formulated that path into two basic robot moves: (1) a unit square and (2) a 90-degree turn. The unit square was a move that traversed one of the squares on the grid. It was created explicitly by the pair to solve the problem; units could be combined to represent larger movements by the robot. In this way, these basic robot moves represented an abstraction of the 6x6 grid in that they represented a group of underlying algebraic thinking - moving one unit meant making the robot move at a specific speed for a specific amount of time. If two units were desired, the learners would double the time of associated with that speed. This increased the precision of their calculations while allowing them to work more effectively without repeated visits to the grid.

<figure><img src="image_181.png" /><figcaption>**Figure 2.** The dotted lines represent the learners’ different attempts at completing the robotics tasks.</figcaption></figure>
The 90-degree turn represented an abstraction of multiple underlying algebraic calculations, in the same way as the unit square functioned. To achieve the 90-degree turn, the pair needed to think about multiple variables at once – the relational speed of the wheels with one another as well as their turning direction and duration of the movement. Negotiating those variables led to the development of a shared language in which each of the basic moves (i.e., unit square and 90-degree turn) made collaborative problem solving more efficient and effective. For example, the learners repeatedly returned to the computer after testing a potential set of movements from the robot. Here, they re-enacted the robot’s movements at the grid and constructed new movements to correct any problems with their solution path. They accomplished this by invoking these basic moves to articulate their thinking (e.g., “We need to go forward one, then turn right.”)

## Tool 2: Gestures

The findings from the data indicated that the emergent physical gestures supported learner's thinking as they engaged in collaborative problem solving during robotics activity. The gestures largely served as a re-enactment of the robot's movement at the grid. The gestures helped the learners gain the perspective of the robot, specifically when debugging the solution and reprogram the robot's movement. In this way, their gestures served as a tool for thinking through and communicating their ideas for combining basic moves into more complex movements (see Figure 3).

The most common combination of basic moves was moving straight and turning; this took place five times over the course of the entire solution path. As the learners collaboratively planned these combinations of basic moves, they employed several gestures to denote the movement of the robot. Figure 3, parts a, b, c, and d, display the gestures associated with this sequence of movements: “We stopped right there and we turned to the left [3a]. And now we need to go forward for one second at speed 5 [3b].” The boy then moves his hands apart to match the distance he wants the robot to traverse after turning (3c). The girl then programs the sequence in the computer as the boy begins to repeat his thinking (3d). The type of gestures reveals how learners built upon their basic moves in an embodied fashion to work more adeptly between their abstract and concrete thinking. Lakoff and Nunez (2000) explained how, as humans, we naturally draw on our basic bodily movements to better understand abstract concepts. From this perspective, the learners’ gestures suggest that their thinking was highly embodied as they planned a solution path for the robot.

<figure><img src="image_182.png" /><figcaption>Figure 3. Gestures communicating basic moves combined into larger sequences of movement and thinking.</figcaption></figure>

## Tool 3: The map and the computer

The learners combined the visual block coding (i.e., computer program) and a small, hand-drawn map to serve as a tool for supporting their collaborative problem-solving. The map and computer together served as a physical representation of their thinking; they would visualize movements from the code itself and then attempt to act those movements out to determine if they would be effective. This is strong evidence of embodiment - learners were offloading cognition onto the map and computer, using the environment to mutually inform their cognition (Wilson, 2002). This is important because it shows how learners dealt with the abstract aspects of computer programming.

Figure 4 displays how the boy used the map (Figure 4a) and computer (4b) to plan a 90-degree turn. The boy explained: “That is where we take the turn (4c). And, that is delay [pointing to the screen, he finds the code that represents the delay] (4d).” The boy’s gestures suggest that he has found two physical representations of his abstract thinking to better communicate with his partner. The learners are looking at both the codes on the screen and the location on the map to predict how the robot will move. They continued:

* B: To take the right turn, we need to set the time.

* G: For how many seconds?

* B: The half, point five,

* G: Point five... Sure [pointing to the screen, both learners look at the monitor and come to a mutual decision].
B: And now we need to set the delay.

This exchange suggests that the tools served as an affordance that helped them come to agreement. The physical representations of both the grid and the movement of the robot helped the learners visualize their thinking and determine whether they agreed with each other.

<figure><img src="image_183.png" /><figcaption>Figure 4. The map and computer together served as a physical representation of abstract thinking.</figcaption></figure>

## Tool 4: The robot at the grid

The robot also served as an important affordance that mediated perception and action. As the learners viewed the robot at the grid, they watched to see whether the robot's movements aligned with their overarching solution path for completing the task. To the extent that those movements did not, the learners returned to their other affordances (e.g., gestures; the map and computer) to debug their program or revise their intended goal in a collaborative fashion. This repeated itself until the task was successfully completed.

Figure 5 displays this process. At the grid, the learners see how the robot did not reach the intended destination (Figure 5a). The girl identified the problem: “We need to go really faster. Not alone faster but ... [pause, waiting for a response from her partner] (5b).” The boy then responds: “Let’s make it the time [that I’ve used] before [waiting for an approval from the partner] (5c).” The girl then offers a guess: “Yeah, make the time...so point 3 more?” The boy then recognizes how they tried this speed setting previously, leading to a new solution in which they retrace their previous attempts at completing the overarching solution path.

<figure><img src="image_184.png" /><figcaption>Figure 5. The grid and the robot serve as a mediator between perception and action.</figcaption></figure>

## Conclusion and implications

This study presents a grounded theory of problem solving during robotics activity from an embodied perspective. The learners engaged in a process in which they were repeatedly evaluating whether their current thinking helped them achieve their overarching solution path. This process was emergent and reactive - sometimes they debugged their programming while other times they revisited their solution path. This dynamic process emphasizes the nature of cognition from the perspective of ecological psychology -- the learners set a goal, and information from the environment confirmed whether their actions helped achieve that goal. This is particularly apparent when looking across the learners' attempts at solving the overall task. Their second attempt took them in an entirely different direction than their first (see Figure 1). They then reacted to what emerged within the environment. Their third attempt is a continuation of their second rather than a re-creation of their first. In this way, their thinking shaped the environment while, at the same time, the environment shaped their thinking. This dynamic interplay between mind, body, and environment is at the heart of embodied cognition and typifies Gibson's (1979) notions of an affordance.

The tools identified in this study, then, illustrate how collaborative robotics activity is an embodied experience. Liben (2008) explained how, as humans, we engage in abstract thinking by manipulating external spatial representations both directly and symbolically. In this study, the robot at the grid provided direct access to the learners' problem solving. The learners they drew on their other tools to create symbolic access in the absence of direct access - they used gestures, the map, and the computer to communicate their thinking and create an effective solution.
These findings hold several implications for the embodiment of computational thinking during collaborative robotics activity in K-12 settings. To begin, learners may need a variety of tools to support their engagement in abstract thinking. Tools that provide symbolic access to their direct experiences with the robot can help them offload cognition into the environment during complex problem-solving. This can support collaborative activity. In this study, creating symbolic access helped the learners make their abstract thinking visible to each other. Tools such as maps, block programming, and basic movements helped them more effectively decompose a task, set goals, and create solutions that are mutually agreed upon. Others have similarly found that these tools can support robotics activity in the classroom (Harlow et al., 2018; Liben, 2012).

Additionally, this study offers insight into the way that collaborative robotics activity is an embodied phenomenon. While a great deal of attention has been spent on robotics and computer science, few if any studies view it from an embodied perspective. Viewing this activity as an embodied phenomenon opens a wealth of unexplored questions and potential for future research. For example, how can we design learning environments from an embodied perspective and be more responsive to learners' needs? How does the design of a robot affect how learners move between symbolic and direct access, and can we reduce the distance between these types of access? Moreover, what is gained by studying how we solve complex problems not as a linear, sequential process but rather as a coupling of perception and action in an emergent, complex setting? This study is an initial attempt at exploring these questions in the context of computer science and improving the design of computer-supported collaborative learning in today's schools.

## References

Alibali, M. W. (2005). Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. *Spatial Cognition and Computation, 5*(4), 307-331.

Alibali, M. W., Nathan, M. J., Wolfgram, M. S., Church, R. B., Jacobs, S. A., Johnson Martinez, C., & Knuth, E. J. (2014). How teachers link ideas in mathematics instruction using speech and gesture: A corpus analysis. *Cognition and instruction, 32*(1), 65-100.

Birks, M., & Mills, J. (2011). Essentials of grounded theory. *Grounded theory: a practical guide, 11-26*.

Charmaz, K. (2006). *Constructing grounded theory: A practical guide through qualitative analysis*: Sage.

Chemero, A. (2011). *Radical embodied cognitive science: MIT press*

Cuny, J., Snyder, L. & Wing, J.M.: Demystifying computational thinking for noncomputer scientists. Work in progress (2010)

Gibson, James J. (1977). “The Theory of Affordances.” In Robert Shaw and John Bransford (eds.), *Perceiving, Acting, and Knowing*. Hillsdale, NJ: Lawrence Erlbaum Associates

Gibson, J. J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin Company.

Glaser, B., & Strauss, A. (1967). Grounded theory: The discovery of grounded theory. *Sociology the journal of the British sociological association, 12*, 27-49.

Grover, S., & Pea, R. (2013). Computational thinking in K-12: A review of the state of the field. *Educational researcher, 42*(1), 38-43.

Hammond, M. (2010). What is an affordance and can it help us understand the use of ICT in education? *Education and Information Technologies, 15*(3), 205-217.

Harlow, D. B., Dwyer, H. A., Hansen, A. K., Iveland, A. O., & Franklin, D. M. (2018). Ecological Design-Based Research for Computer Science Education: Affordances and Effectivities for Elementary School Students. *Cognition and instruction, 1*-23.

Hirose, N. (2011). Affordances, effectivities, and extension of the body. In W. Tschacher & C. Bergomi (Eds.), *The implications of embodiment: Cognition and communication*. (pp. 231–252). Charlottesville, VA: Imprint Academic.

Hirose, N. (2002). An ecological approach to embodiment and cognition. *Cognitive Systems Research, 3*(3), 289-299.

Ionescu, T., & Vasc, D. (2014). Embodied cognition: challenges for psychology and education. *Procedia-Social and Behavioral Sciences, 128*, 275-280.

Johnson, M. (2008). The meaning of the body. In *Developmental perspectives on embodiment and consciousness* (pp. 19-43). Psychology Press.

Johnson, C. W., & Parry, D. C. (2016). *Fostering social justice through qualitative inquiry: A methodological guide*. Routledge.

Kopcha, T. J., McGregor, J., Shin, S., Qian, Y., Choi, J., Hill, R., ... & Choi, I. (2017). Developing an integrative STEM curriculum for robotics education through Educational Design Research. *Journal of Formative Design in Learning, 1*(1), 31-44.
Israel, M., Pearson, J. N., Tapia, T., Wherfel, Q. M., & Reese, G. (2015). Supporting all learners in school-wide computational thinking: A cross-case qualitative analysis. *Computers & Education, 82*, 263-279.

Lakoff, G., & Núñez, R. E. (2000). Where mathematics comes from: How the embodied mind brings mathematics into being. *AMC, 10*, 12.

Liben, L. (2008). Representational development and the embodied mind's eye. *Body in mind, mind in body: Developmental perspectives on embodiment and consciousness*, 191-224.

Liben, L. S. (2012). Embodiment and children's understanding of the real and represented world *Developmental perspectives on embodiment and consciousness* (pp. 207-240): Psychology Press.

Linderoth, J. (2012). Why gamers don't learn more: An ecological approach to games as learning environments. *Journal of Gaming & Virtual Worlds, 4*(1), 45-62.

Lye, S. Y., & Koh, J. H. L. (2014). Review on teaching and learning of computational thinking through programming: What is next for K-12?. Computers in Human Behavior, 41, 51-61.

Malafouris, L. (2013). *How things shape the mind*: MIT Press.

Michaels, C. F., & Palatinus, Z. (2014). A ten commandments for ecological psychology. *Routledge handbook of embodied cognition*, 19-28.

Richardson, M. J., Shockley, K., Fajen, B. R., Riley, M. A., & Survey, M. T. (2009). Ecological psychology: Six principles for an embodied-embedded approach to behavior. In *Handbook of Cognitive Science* (pp. 159-187).

Rietveld, E., & Kiverstein, J. (2014). A rich landscape of affordances. *Ecological Psychology, 26*(4), 325-352.

Rowlands, M. (2010). *The new science of the mind: From extended mind to embodied phenomenology*. MIT Press.

Sengupta, P., Kinnebrew, J. S., Basu, S., Biswas, G., & Clark, D. (2013). Integrating computational thinking with K-12 science education using agent-based computation: A theoretical framework. *Education and Information Technologies, 18*(2), 351-380.

Sullivan, A., & Bers, M. U. (2016). Robotics in the early childhood classroom: learning outcomes from an 8-week robotics curriculum in pre-kindergarten through second grade. *International Journal of Technology and Design Education, 26*(1), 3-20.

Wilson, M. (2002). Six views of embodied cognition. *Psychonomic bulletin & review, 9*(4), 625-636.

Wing, J. M. (2006). Computational thinking. *Communications of the ACM, 49*(3), 33-35

Yadav, A., Mayfield, C., Zhou, N., Hambrusch, S., & Korb, J. T. (2014). Computational thinking in elementary and secondary teacher education. *ACM Transactions on Computing Education (TOCE)*, 14(1), 5.

Young, M. (2013). An ecological psychology of instructional design: Learning and thinking by perceiving-acting systems *Handbook of research on educational communications and technology* (pp. 180-188): Routledge.

Yuen, T., Stone, J., Davis, D., Gomez, A., Guillen, A., Tiger, E. P., & Boecking, M. (2015). A model of how children construct knowledge and understanding of engineering design within robotics focused contexts. *International Journal of Research Studies in Educational Technology, 5*(1).
# Bugs as a Nexus for Emergent Peer Collaborations: Contextual and Classroom Supports for Solving Problems in Electronic Textiles

Deborah A. Fields, Utah State University, deborahfields@usu.edu

Gayithri Jayathirtha, University of Pennsylvania, gayithri@gse.upenn.edu

Yasmin B. Kafai, University of Pennsylvania, kafai@upenn.edu

**Abstract:** Few studies have examined the role of failure in more open-ended situations where problems develop as a consequence of designing projects and where collaborations can emerge as an outgrowth of debugging said problems. In this paper, we explore the peer-to-peer collaborations that emerge spontaneously in the context of coding, crafting and design bugs within open-ended design activities, specifically an electronic textiles unit for secondary students taught over 10-12 weeks in introductory computer science classes. Examining observations from three introductory computer science classrooms, we address the following research questions: (1) How and what kinds of peer-to-peer collaborations emerged in unstructured ways, especially around bugs in open-ended projects? and (2) What curricular, spatial, social, and teacher supports allowed these interactions to emerge and flourish? In the discussion, we consider implications for supporting similar types of emergent collaborative learning in open-ended computational making designs.

## Introduction

One could argue that failure in its various forms has more often been seen as an impediment rather than a complement to successful collaboration (Barron, 2003). Indeed, failure can lead to stable patterns of discourse in a classroom community with troubling consequences for students’ ideas about themselves (DeLiema, 2017). However, the recent success of “productive failure” (Kapur, 2008) has put failure into a different light: it highlights the counterintuitive notion that failure can precede later success in learning. Yet the research on productive failure has generally emphasized tight structures both in collaborative groups and in the sequence of planned ill- and well-structured problems (e.g., Kapur & Kinzer, 2009, Kapur, 2008). Missing in these studies is an opportunity to examine the role of failure in more open-ended situations where problems develop as a consequence of designing projects and where peer collaborations can emerge as an outgrowth of debugging said problems.

Problems that stem from designing open-ended projects, such as those common in most STEM-oriented maker activities (Peppler, Halverson & Kafai, 2016), occur frequently in many situations where groups are not pre-designated. In open-ended maker activities, temporary failures or unexpected bugs are not just hindrances but also opportunities for learning, when and if students reach outside of their immediate work to a wider group of people and resources (e.g., Sheridan et al., 2014). Reaching out can generate emergent collaborations, that occur quite frequently in makerspaces as students help each other, invent uses for technology, and catch ideas from each other (Halverson, Litts & Gravel, 2018). This raises questions about how peers help each other and what attributes of the broader environment facilitate such improvised collaborations, for even spontaneous collaboration does not take place in a vacuum.

In this paper, we explore the peer-to-peer collaborations that emerge spontaneously in the context of bugs (or problems) within open-ended design activities, specifically an electronic textiles unit for secondary students taught over 10-12 weeks in introductory computer science classes (Fields et al., 2018). Electronic textiles (e-textiles) involve programmable circuits hand-sewn onto soft objects like clothing and stuffed animals, with conductive thread, LEDs, digital sensors, and seawable microcontrollers, providing a space for creating personally relevant computational artifacts (Buechley, Peppler, Eisenberg & Kafai, 2013). Making an e-textile artifact involves learning not only about crafting, circuitry and code but also about identifying, isolating, and fixing bugs at the intersection of these domains (Fields, Searle, & Kafai, 2016; Jayathirtha, Fields & Kafai, 2018). Whereas the study of collaboration has often focused on structural supports that teachers or computer environments provide to ensure success, this study focuses on how peers help each other progress through failure and the qualities of the broader learning environment that support these emergent collaborations. The particular context of our study includes three introductory computer science high school classrooms with 69 students where different teachers at separate public secondary schools led an e-textiles unit in which students created a series of four e-textile projects (Fields et al., 2018). In this paper we explore 1) How and what kinds of peer-to-peer collaborations emerged in unstructured ways (i.e., outside of assigned collaborative groupings), especially around bugs in open-ended projects? 2) What curricular, spatial, social, and teacher supports allowed these interactions to emerge and
flourish? In the discussion we consider implications for supporting similar types of emergent collaborative
learning in open-ended computational making.

## Background

Debugging has long been noted as a productive, if not particularly collaborative, site of learning (e.g., Papert, 1980). As a site of problem solving, debugging is recognized as a key computational thinking practice in engineering and computing that is essential but often overlooked in K-12 classrooms (College Board, 2016; McCauley et al., 2008). Debugging computational issues requires a deep and systematic understanding of the program along with the programming language and environment (i.e., McCauley et al., 2008). The ability to debug or troubleshoot may be especially challenging for novice programmers who lack experience in seeing programs as a whole and systematically identifying, testing, and solving problems (Vessey, 1985). Yet while there is significant research around tools and programming environments designed to support learners through the process of debugging (e.g., Ko & Myers, 2004; Sorva, Lönnberg, & Malmi, 2013), there is relatively little research about other kinds of pedagogical, social, and environmental structures that can support learners dealing with various debugging challenges. Understanding learning environments that support debugging is especially important since the complexity associated with debugging demands not just programming skills but also other skills such as decision-making, emotional intelligence, and perseverance (e.g., Patil & Codner, 2007).

Within the field of computer science education, collaboration is generally understudied even if it is
recognized as helpful. Most prominent is the research on pair programming where a team of two learners
collaborate on one project, designing, coding, testing and debugging on a single machine with students
periodically changing roles as “driver” and “navigator” (Williams & Kessler, 2000). While this model illuminates
the social aspect of problem solving in programming contexts, this more formal, structured model of collaboration
does not include other possible modes of collaboration when more than a pair of learners attempt to support each
other. Further, these formal collaboration arrangements become more difficult to maintain when learners are
programming in hybrid computing environments such as robotics and e-textiles. For example, the studies about
collaboration in making e-textiles or similarly crafted computational artifacts have shown students splitting up
their learning in inequitable ways rather than learning productively from each other (e.g., Litts et al., 2017).
Moreover, a focus on only formal ways of pairing up students misses opportunities to explore other more open-
ended forms of collaboration and emergent social supports where students can draw help from peers other than
their immediate partners.

In this study then, we focus on the nature and contexts of emergent peer learning while debugging in a
classroom environment, how it emerged (because of the teacher, peers, or through other means), and how it
mattered to individual students. This type of peer-to-peer collaboration has received much less attention in
research because most studies have focused on organized small groups, with students taking on various roles or
with some students designated as more experienced experts or less experienced novices in situations intended to
generate “peer pedagogy” where peers educate each other (e.g., Ching & Kafai, 2008). Yet teachers are already
developing environments where more emergent peer pedagogy, not structured in specific groups, can take place.
For instance, a prior study on the e-textiles curricular unit for introductory computing classes documented
productive teacher practices that supported equity, namely by legitimizing peer expertise and supporting iterative
learning (Fields et al., 2018). The researchers observed that teachers modeled their own and their students’
mistakes to the whole classroom, lowering the risk associated with failure and allowing students to provide ideas
to solve problems in the projects. Sometimes, especially in situations where the teacher could not give
individualized help to each student, the teacher would train one student in a task (e.g., a crafting or computing
technique) with the expectation that that student would then share that knowledge with others. These and other
practices, structures, and environmental features may support productive peer pedagogy.

The many peer-to-peer learning collaborations that emerged beyond formal partnerships in our study
appeared to be highly significant to students’ continued perseverance in the e-textile unit. So, we sought to
understand how these specific collaborations emerged, what relevance they had to individual students, and what
aspects of the larger classroom environment facilitated these types of interactions.

## Methods

### Context and participants

Over the course of two years, we developed an e-textiles unit for the *Exploring Computer Science* (ECS) curriculum, a year-long introductory computing course with equity-focused and inquiry-based teaching (Goode, Chapman, & Margolis, 2012). This paper focuses on the second year of this e-textiles unit implementation, where three teachers led the unit separately in each of their respective schools. The e-textiles unit took place over 10-12