# Collaboration on a Massive Scale – Conceptual Implications of the
# Crowd

Norma Reichelt, Christine Bussian, Christoph Richter, Heidrun Allert, and Lars-Arne Raffel
reichelt@paedagogik.uni-kiel.de, bussian@paedagogik.uni-kiel.de, richter@paedagogik.uni-kiel.de,
allert@paedagogik.uni-kiel.de, lars.raffel@email.uni-kiel.de
Christian-Albrechts-Universität zu Kiel, Institute of Educational Science

**Abstract:** The question of how to engage large user groups in web-based learning arrangements is consistently raised in CSCL discourse. Recently the interest in crowd phenomena of joint knowledge creation and inquiry as well as their adaptability within institutionalized learning is brought together under the term of mass collaboration. This paper proposes a conceptualization of the crowd since learner involvement, sociality, collaboration and knowledge construction, coordination and regulation efforts, as well as assessment must be fundamentally thought anew, when we take a step away from a small group setting. Thus, we put forward a design framework that provides perspectives on CSCL environments on a massive scale and thereby challenges our understanding of collaborative learning.

## Introduction

In the last years, the masses have played a decisive role in raising social, economic, political and environmental issues and mass movements like Occupy, the Arab Spring or #blacklivesmatter have drastically established unprecedented ways of public discourse and engagement. In science and education, large accumulations of learners, researchers or civic participants are addressed in open education initiatives like Massive Open Online Courses (MOOCs) or Open Educational Resources (OERs) and get involved in open science or citizen science projects. Further, it is also relevant to mention phenomena of joint development and creation (for example open source software development or the global online encyclopedia Wikipedia) as well as various possibilities for distributed financing, fundraising or forms of investigative journalism (for example the investigation platform Bellingcat). With these new forms of participation we notice new forms of collaboration and knowledge practices that are inseparably linked with technological developments especially in the fields of social software and web2.0/3.0 applications like social network sites and microblogging services. Herein lie promising challenges and opportunities for CSCL research and development to further evolve in the light of technological advancement and novel forms of interaction. The relevance of web 2.0 dynamics and tools has already been extensively discussed within the CSCL community, particularly in the discourses on collaborative knowledge building (Lu et al. 2010; Scardamalia et al. 2012). Also, the quality of massiveness has been introduced under the term of mass collaboration as one of the future-oriented topics for the design of and the research on collaborative learning environments (Cress et al. 2016; Fischer 2016; Jeong et al. 2017).

This paper contributes to the ongoing discourse. Firstly, we try to conceptualize the socio-technical construct behind the above-mentioned phenomena, that we further refer to as the crowd. We distinguish the crowd from a community and outline it as a distinct entity with specific qualities. In this way, the crowd serves as a productive anchor point to envision innovative learning scenarios and learning environments that are designed to transcend conventional classroom or distant learning settings towards participatory, engaging and fast paced forms of interaction. Secondly, we put forward a design framework for CSCL scenarios that allows to discuss implications of the crowd for the development of a learning environment that supports large groups of learners to establish collaborative knowledge practices. Building on four forms of mediation (Paavola et al. 2012), we introduce five design areas - learner involvement, coordination and regulation, sociality, collaboration and knowledge construction, assessment and feedback - under which we deepen our understanding of the crowd for learning contexts. Finally, we discuss in what way and to what extent the crowd as an organic entity and as an institutionalized and formalized educational setting - as most CSCL scenarios suggest - can be integrated with one another productively. Additionally, forms of crowd inquiry and crowd insight must be specified, especially in comparison to conventional scientific knowledge practices. Thus, this paper provides clarification regarding the current and highly relevant social phenomenon of crowds and systematically opens it up for further theorization and the development of computer supported collaborative learning environments.

## Conceptualizing the Crowd for a CSCL context

In the following section, we discuss the crowd from two complementary angles. We ask how the crowd can be understood in its procedural nature - mainly in its formation, its internal coherence and its actions - and how we
can describe it in its fundamental structural and socio-technical properties. To ensure integrability within the current CSCL discourse, we structure our argumentation by building on the state of the art of the mass collaboration approach and by deriving it closely from widespread and acknowledged concepts, namely the Community of Practice (CoP) (Lave & Wenger 1991) and the Community of Interest (CoI) (Fischer 2001; Fischer 2007). As stated above, mass phenomena are quite different in nature. Not only explicit cases of education or, in a broader sense, knowledge creation come to mind but also instances of civic engagement, socio-political participation and activism. While in the field of institutionalized education massiveness remains an ongoing challenge, at the same time we can observe the emergence of informal large-scale accumulations of respective reach, vigor, creative power, productivity and voice. Following, the latter serve as a starting point to conceptualize the crowd and discuss it from a CSCL point of view. Therefore, with the crowd, we are facing a complex social entity, characterized by a highly dynamic nature and intricate internal structure, making use of diverse sets of socio-technical tools and infrastructures and bridging between online landscapes and physical arenas.

From an educational perspective, these current mass phenomena lack a theoretical framework which approaches the crowd as a distinct type of socio-technical entity. Large-scale interactions, in an admittedly expansive understanding, are mainly discussed in connection to the CSCL concept of mass collaboration (Cress et al. 2016; Fischer 2016; Jeong et al. 2017). Despite the marginal conversation about the crowd as a theoretical construct and the use of the term itself, the discussion on how participants come together and engage in large-scale collaborative efforts offers a valuable starting point to establish an understanding of the crowd. Under the term of mass collaboration we are able to distinguish the collaboration in massive scale environments from interactions in small groups or teams. Mass collaboration, in the broadest sense, is described as a large gathering of people in order to work or learn together (Fischer 2016). These social entities are understood as knowledge communities which Jeong et al. (2017) use as an umbrella term to describe a union of people in which knowledge is shared, picked up by others and in some cases newly constructed (Cress et al. 2015). For such knowledge communities, Jeong et al. (2017) list three distinct characteristics in terms of their interaction. Firstly, when such knowledge communities come together at arguably large scale, this form of interaction results in the collaboration among participants from various backgrounds and different geographical locations (Jeong et al. 2017; Fischer 2016). Secondly, despite the differences in knowledge background and geographical location, the participants of knowledge communities manage to organize their work in order to collaborate on artifacts (Jeong et al. 2017). Lastly, the product of mass collaboration usually entails knowledge creation based on collective work efforts (Jeong et al. 2017). Although this definition includes an underlying notion of a crowd, by taking a closer look, it becomes clear that the crowd and the community are not considered separately. Yet, we have reason to believe that the crowd bears its own distinct features which we discuss in more detail in the following.

Knowledge communities understood in the context of massiveness are defined by bearing a shared “mission” (Jeong et al. 2017) in order to achieve a productive way of working together. We argue that when considering the crowd, this shared “mission” has to be viewed from two complementary perspectives. A shared concern on the one hand is what incites masses of participants to form and maintain a crowd while joint efforts on the other hand are their unified actions. To illustrate this distinction: For the social justice crowd phenomenon #blacklivesmatter the unified rage about police brutality against citizens of african-american decent can be regarded as a shared concern and the various activities within the movement (for example using the hashtag across the whole social media landscape or participating in rallies and demonstrations) represent the joint efforts of the crowd. Shared concerns in our understanding of crowds are able to initially and virally vitalize exceptionally large amounts of people and keep their unified momentum going. Whether or not crowds eventually dissolve without a shared concern or with a concern losing its force of attraction over time - just like CoIs dissolve, after a project has ended (Fischer 2007). Another possible scenario could be the gradual stabilization or institutionalization of certain domains of a crowd in more sustainable communities. Also, communities may not only be seen as a remnant of crowd activity but as a relevant factor when it comes to virally spreading a concern or collectively joining efforts. In this reasoning we assume that crowd and community aspects interplay in a certain way for productive large-scale collaboration, the complex synergy of those socio-technical entities has yet to be empirically explored in more detail.

To further distinguish the crowd from communities, we consult the conceptions CoP and CoI. For one thing, a CoP is characterized by a shared social practice as a constituting element. It implies specific properties such as long-lasting, ongoing participation and holding together the members through the attribution of certain roles (e.g., newcomer/oldtimer). The internal structure of a CoP has an underlying notion of being potentially open to anyone while practically offering restricted access for people that can constitute themselves as a member by enacting and upholding the communities’ shared practices or by becoming part of it by progressively adapting (Lave & Wenger 1991). This notion of community is typically used to describe social unions of people from similar (knowledge) backgrounds which are defined by a form of biased communication as the members only
communicate amongst each other, building up on a common background (Fischer et al. 2007, p. 14). Since the crowd as pointed out is mainly held together by its shared concern and joint efforts, the diversity among its participating individuals may reach far beyond that of a CoP. Although CoIs (primarily described as focusing on a specific mutual interest) include the notion of diverse members as well (Fischer 2001), it is important to point out that CoIs fail to pick up the argument of viral temporality and the inherent momentum that we would ascribe crowds. CoIs per definition present a “community of representatives of communities” (Fischer et al. 2007, p. 13) for example in interdisciplinary project teams. In contrast to these conceptions of community, a crowd does not exhibit the procedural characteristics of moving from peripheral to full and core membership (as conceptualized by Lave & Wenger 1991), for example engaging in an ongoing process of enculturation by upholding a shared practice. Rather, in a crowd, various perspectives on shared issues are brought together and expressed side by side, not in a way of successively becoming a part of something by adapting to a set of enacted practices but in a way of being momentarily involved. Certainly shared practices play an important part in connecting the various members of a crowd and securing their mutual understandings to some extent. Nevertheless we would argue that there is a declining significance of inherent processes of establishing convergence for the social entities at hand. Within the concept of CoPs there is a strong focus on reenacting a given internal structure. This is also discussed for CoIs with the difference that a diversity in participants calls for processes of negotiating and establishing common ground to a greater extent (Fischer 2007). For massive-scale interactions however it can be put into question if common ground and a widely accepted consensus provide an underlying mechanism of agreement at all or if antagonisms and notions of dissent can coexist in a unified crowd without endangering the overall efforts.

To this point we have established the crowd by building on a community perspective. To add to this we have to further take into account the crowd not only as a social entity but as a socio-technical one. Crowds are inextricably connected with modern informational and communicational technology. Especially Web 2.0 and 3.0 applications play a significant role when it comes to the aspect of virality, various forms of contributions and the modes of communication within a crowd. In this, crowds are not bound to one service alone but act across a diverse range of social media platforms in particular. Additionally, crowd efforts are not limited to virtual environments but also cover joint activities within offline spaces.

Given the fact that we establish a conceptualization of the crowd against a pedagogical background, naturally the aspect of learning must not be neglected. Therefore, we concern ourselves with the question what an approach to learning in crowd settings could look like. One specific model, that we assume to be adaptable in a crowd context, is the rhizomatic learning approach (Cormier & Stewart 2010). We see a great chance in further refining the concept of rhizomatic learning - which in its core means the distribution of knowledge in digital environments without hierarchical forms of dispersion - in close correlation with our notion of the crowd as a specific case of application.

Concluding, this attempt to provide a first heuristic discussion of the crowd, we can summarize the following procedural and structural qualities: The crowd in our understanding is a mass phenomenon that virally emerges around a concern that touches all participants in their own way and initiates them to contribute to joint actions. The crowd's members are diverse. They face the shared concern from potentially very different perspectives while simultaneously being connected and activated by their smallest intersection, their mutual affectedness. Therefore, the crowd's temporality is solely bound to its own momentum. The crowd is potentially open for everyone to join. Participation involves solely an act of contribution and no process of growing into a shared practice. In this sense, the content of a contribution may differ broadly in regard to the individuals behind them but they are not endless in their variation. Rather, forms of meaningful participation are connected to overstretching practices that are reproduced or emerge within the crowd's socio-material manifestation (e.g., posting on a social network platform), social events (e.g., taking part in a demonstration) and the crowd's operational area (e.g., writing an article for an online encyclopedia). In this, the crowd proposes a promising new take on education in a digital society.

## Collaboration on a massive scale - a design framework

Thus far we have proposed a first conceptualization of the crowd that ties in the current CSCL discourse. Now, the next question is how to approach this insight from a developmental perspective to envision innovative CSCL scenarios on a massive scale. More precisely, the question is to which extent learning activities in the crowd raise design issues that resemble or go beyond those, pertinent to other CSCL scenarios and environments.

To identify crucial themes of interest for computer supported collaboration and learning in general, we conducted an exploratory literature review into notable research discourses that bring together notions of collaboration and knowledge creation, namely CSCL research and research on open education. To pinpoint the consequences of massiveness, we then enriched our findings with insights from additional domains that address scenarios of large-scale participation and creation, like citizen science and open source software development.
Within these domains we focused on (1) collaborative scenarios in general while keeping in mind the transferability to, as well as consequences for the crowd context and (2) particular scenarios that involve arguably massive participation.

<table><caption>Table 1: Design Framework for CSCL Scenarios</caption><thead><tr><th>Formational Mediation:<br>Learner Involvement</th><th>Pragmatic Mediation:<br>Coordination &amp; Regulation</th><th>Social Mediation:<br>Sociality</th><th>Epistemic Mediation:<br>Collaboration &amp; Knowledge Construction</th><th>Reflective Mediation:<br>Assessment &amp; Feedback</th></tr></thead><tbody><tr><td>Recruiting Participation Engagement Motivation Sustainability</td><td>Dynamic Static Intrinsic Extrinsic Process awareness Decision making Didactics Technology Negotiation Established practice Hierarchy</td><td>Communality Social awareness Communication Authorship Accountability Responsibility</td><td>Common ground Resources Knowledge objects Knowledge artifacts Shared insights Currents of inquiry Epistemic frames Subjects Social permeability</td><td>Many-to-many Peer-to-peer Instructor-based Creation of value Institutionalization Formalization</td></tr></tbody></table>

To be able to map and compare critical elements of crowd-based learning scenarios in relation to other CSCL settings, we build on the multidimensional model of mediation, suggested by Béguin & Rabardel (2000) and adapted to the field of CSCL by Paavola et al. (2012). According to Paavola et al. (2012), CSCL scenarios can be described and analyzed along four dimension or types of mediation. These types of mediation include (a) epistemic mediation, the procedural means through which knowledge is created collaboratively, (b) pragmatic mediation, the procedural means through which the collective efforts are coordinated and regulated, (c) social (or collaborative) mediation, the procedural means through which social relations and networks are (re-produced), as well as (d) reflective mediation, the procedural means through which the collective efforts are assessed, evaluated and advanced. The literature review however pointed to a set of design issues, which goes beyond the types of mediation suggested by Paavola et al. (2012). These issues relate to what Jeong et al. (2017) have discussed as different “forms of joint interaction that occur in large-scale community settings” (p. 134). We take up the relevance of categorizing such phenomena to describe how people work together and interact with each other by abstracting it to a more nuanced idea of learner involvement which is concerned with those processes and means by which actors are recruited and participation is motivated and ensured. We subsume these issues under the notion of (e) formational mediation, the procedural means through which involvement and participation are ensured. Table 1 provides an overview of the design issues we have identified with regard to these five types of mediation. The table can be understood as generic framework for the description, analysis, design and evaluation of computer supported collaborative learning scenarios and environments. In the following, we will use this framework as a vantage point to further discuss the particularities of crowd-based learning scenarios.

## Formational mediation – Learner involvement

The first question for the setup of collaborative learning is that of how to address and involve participants and who those participants are. The recruiting of learners, so to say. Thinking along the lines of a crowd setting that transcends a given and formalized union of a classroom, recruiting implies that a critical mass of individuals has to be attracted, for crowd dynamics to develop at all. The actual process of rallying a crowd can only be understood in regard to the party responsible for the recruitment - for example a research team in a citizen science context (Chu et al. 2012), a MOOC provider (Haywood 2016, p. 75) or the crowd itself in its own momentum. Following this argument, the internal structure of a crowd comes into view, since different activities call for different levels of diverse or shared perspectives or practices among the involved individuals (Bonney et al. 2016) - regardless of the externally given or internally established origin of mutual objectives. Therefore, the participants and their individual qualities can be of high importance as well, because for being successfully involved in the joint effort they might need to have certain knowledge and competences at their disposal (Kobori et al. 2016) or their contributions need to achieve a specific standard to be considered of value or compatible at all (Ghosh & McAfee 2011). Adding to the consideration of recruitment, a distinction between participation and engagement seems reasonable. Within the frame of participation we can ask in what ways participants are able to take part in a meaningful way, which forms of contribution are compatible and if members face some kind of participational
threshold. Engagement on the other hand addresses the ongoing and active involvement of participants and is therefore a vital topic within the open education and open science discourse (Walji et al. 2016; Sprinks et al. 2017). Considering the social entity of a crowd we must challenge the idea that ongoing and active involvement constitutes a desirable participant. On the contrary, we should be mindful of punctual - and maybe minimal - acts of contribution and dissemination in consideration of the crowd's shared efforts and concern.

Following this contemplation, the view on motivation changes as well. Participant motivation is a leading topic for all kinds of educational and participatory scenarios (Rogat et al. 2013; Eveleigh et al. 2014) while the explicit implications for crowd-based learning have yet to be discussed. In the light of heterogeneous participants, diverse forms of participation and various modes of engagement as well as shared concerns and efforts as a constituting element of a crowd, motivation has to be conceptualized between these contrasting poles of enormous variability of being part of a crowd and presumably more common reasons to be involved at all.

Finally, we have to question the live-cycle of a crowd, for example its sustainability. With the constitutive factor of a shared concern and a joint effort, a crowd may dissolve when those inherent impulses are no longer given or may partially evolve in more stable forms of communality like a community or an organization. Besides this, processes of institutionalization of crowd phenomena, the compatibility of temporal arbitrariness of crowds and often temporally ridged educational structures have to be considered when imagining learning in a crowd-based environment.

## Pragmatic mediation – Coordination and regulation

The various ways in which collaborative processes are coordinated and regulated are an ongoing issue within CSCL discourse (Ludvigsen et al. 2018). Assuming an inseparable relation between didactic prompts and interventions and technical affordances, we have to ask how participants are able to interact with the environment and with each other in a meaningful way. Especially when designing for a crowd the question arises how didactics and technical guidelines can support the joint efforts without restricting or preventing the crowd's constitutive momentum. In short, how can an emergent property like virality be facilitated at all? It is also relevant to consider the coordinating and regulating aspects of a learning scenario between the poles of static and dynamic structures and intrinsic and extrinsic determinations. Is a structure for example pre-established or does the environment enable the participants themselves or an external instructor to question, adapt and change their ground rules of collaboration? With the quantity and the diversity of crowd participants in mind, we further have to ask if equal negotiation and unanimous decision making is possible or even necessary for the crowd to be able to act and for people to join. Rather it seems essential to establish some kind of process awareness to help participants to find ways for at least minimal or perhaps more elaborate forms of contribution. Process awareness here serves as an ideal concept, because within the crowd's complexity and its emergent evolution it is an impossible task to concurrently describe it entirely. Finding the fine line to foster a crowd's dynamic without crippling it with a tight pedagogical frame and ensuring access for learners despite its complexity seems to be one of the main challenges when designing crowd scenarios for educational context.

## Social mediation – Sociality

We have outlined the crowd as a socio-technical entity that is on the one hand diverse, complex and to an extent unbound and on the other hand temporally constituted under a joint concern and with a mutual cause. By accommodating such qualities, it can be clearly distinguished from other forms of communality that are addressed in CSCL scenarios as we argued in the previous paragraph.

Another concept that relates to sociality within the fields of computer supported collaboration is that of social awareness (Dourish & Bellotti 1992). It addresses possible ways for users in a virtual environment to perceive themselves as part of a social unit and consequently take note of each other. In this regard we raise questions at three levels. How is an individual able to become aware of all the other participants in the crowd entity, given the fact that so many people are involved in different ways at different levels (Jeong et al. 2017)? How does a notion of being part of a mass-movement emerge and what shapes this experience? And lastly, how can social awareness as a concept address the crowd as an entity that may not be attached to one specific environment but stretches over multiple virtual spaces and spreads into the offline world as well?

When members of a crowd communicate with each other, it can reasonably be concluded that communication exceeds dialogical forms. Such exceeding communication modes - which we assume to be highly relevant for crowd settings - can be observed in social networking or microblogging applications (e.g., in follower-followee relations, under shared hashtags or through @mention functionalities). With those mass-environments in mind, communication modes like one-to-many broadcasting (Page 2012) or pushing of pre-existent content without adding to it (Kaplan & Haenlein 2011) have to be considered in their consequence for the development of means of communication within learning scenarios for the crowd.
Another important factor that has to be thought anew in crowd settings is that of authorship which also implies the question of accountability and responsibility. We assume that in part, participants maintain visibility as authors as such. Additional scenarios though are anonymous forms of participation as well as rather content-less acts of meaningful participation (e.g., liking or reblogging/retweeting) that are immanent in crowds. In this case the question arises who can be held responsible for contents, statements and actions. If we consider a joint effort of a crowd as manifestation of its work, is it even possible to ascribe authorship, responsibility or accountability to a single person? Or must a crowd as entity be held accountable or responsible somehow?

## Reflective mediation – Assessment and feedback

Assessment and feedback structures are among the most recurring and discussed issues in the CSCL community, not just as final evaluation method, but especially as an advancing act within the process of collaborative learning. In this regard the concept of peer-to-peer interaction is of particular importance and investigated in its own qualities (Reinholz 2016) as well as in its differentiation from instructor-based feedback and assessment mechanics (Harney et al. 2017). Additionally, many-to-many assessment scenarios have been explored within the field of open education and MOOC development, with the expectation of bringing together different levels of expertise, experience and prospects to evaluate contributions (Clougherty & Popova 2013). For crowd-based collaboration we have to ask how hierarchy-driven, peer-driven and in particular crowd-driven feedback and assessment dynamics take shape and in which way they can be fostered or are emerging organically during massive-scale interactions. In the open, complex and diverse socio-technical structure of the crowd, the ways in which the ascription of value unfolds in regard to the joint concern and efforts can therefore only be anticipated as an issue of future investigation. We assume that in addition to evaluating and reflecting functions of assessment and feedback, we will observe aspects of admission, such as generally including contributions as meaningful or disregarding them as irrelevant. In comparison to institutionalized educational settings, we assume that there are no explicit and formalized forms of valuation (e.g., examinations, grades or standards of education) existent in the crowd. Nevertheless, we ask ourselves if processes of formalization - beyond the shared practices of participation - play a role in crowd environments and how they can be explored further.

## Epistemic mediation – Collaboration and knowledge construction

Lastly, we have to consider the crowd from an epistemic standpoint and discuss the circumstances of collaborative knowledge practices. To understand such processes, we have to determine the crowd's resources which ultimately create the foundation for knowledge construction. These resources include a certain diversity in perspectives, different approaches to problem solving and by sheer mass of participants, a major workforce.
Within a heterogenous crowd we can assume participants from various epistemic backgrounds to bring in their disciplinary practices and standpoints or, as Shaffer (2006) suggests, their epistemic frames. For a crowd setting it has to be revised, if an epistemic diversity only presents itself as a resource for multifaceted knowledge creation or if it may interfere as a participatory obstacle or even a gatekeeper. Diversity may call for the negotiation of common ground. But, as we have already discussed that common ground in a crowd context is sufficiently ensured by the shared concern and that beyond that, integrability of contributions only arises by the participatory act of contributing itself. Thus, different perspectives and approaches can coexist under a shared concern and are realized through joint efforts without necessarily coming into conflict with one another.

Framing crowd activities as an epistemic endeavor the question arises what can be considered as the outcome of a collaborative knowledge work. What shapes do crowd insights might take and are they for example of a more collective or a more individual nature. Clarification is needed on how such insights can become visible at all. How can a multitude of contributions together form a coherent current of inquiry? To further elaborate on the shared concern from an epistemic perspective, we see great potential in adapting Knorr Cetina's (2001) concept of Knowledge Objects and partial artefacts. Especially the notion of partiality we see as a starting point to examine the role of scattered contributions in relation to the collective epistemic process of a crowd.

Finally, recalling the observation that crowd phenomena are socially motivated, the matter of permeability comes into focus. We have to ask in what way integrability beyond the educational context of crowd insights can be established not only for scientific societies but for civic discourses. This can be understood in two different ways: For one, the crowd can be enriched and motivated by sources beyond the specific educational context, but also, its insights can be regarded in their societal accessibility.

## Implications and discussion

With this paper we have put forward two contributions to the consideration of massiveness within CSCL contexts. Building on the discourse around mass collaboration and the concepts of CoPs and CoIs, we have conceptualized the crowd as a first attempt to establish a productive connection between current mass phenomena and educational
design. Within a framework of five areas of mediation we then took this conception of the crowd as a vantage
point to raise issues of collaborative mass scenarios in a more detailed way. Drawing conclusions from this
approach, we see the crowd as an opportunity to evolve computer supported collaborative learning alongside
current technical and societal processes of transformation. Nevertheless, or possibly for exactly this reason, the
crowd presents itself as a challenging design target.

We have outlined the crowd as a highly complex entity with an emergent nature and an inner momentum
that is primarily fueled by the mutual affectedness of its participating individuals. The open question remains how
an educational design can address all those issues in a way that fosters a crowd's formation and productive
interactions within the scope of pedagogical expectations and constraints without inadvertently suppressing the
dynamics. How can we achieve to cultivate an organic entity inside institutionalized education? Or do we rather
have to envision a new form, a guided crowd, as a formalized variety alongside informal crowd phenomena?

In the light of the crowd, the aspects of learning, of collaboration and of inquiry require a reconsideration
as well. We can ask where learning processes arise while taking part in crowd activities and which understanding
of learning seems suitable to describe and facilitate those instances (e.g., rhizomatic learning). Further, we may
have to widen our understanding about subjects of learning all together and consider not only the individual learner
but the crowd as a collective learning entity. It is also worthy of discussion whether or not present conceptions of
collaboration are suitable to describe joint efforts where interpersonal activity is not a constituting circumstance,
or if crowd phenomena can lead to an extended theorization of collaboration all together. Another fundamental
challenge lies within the CSCL objective to foster forms of inquiry derived from scientific practice. An idea of
crowd-based inquiry may help to broaden the perspective of meaningful collective knowledge creation, especially
in close relation to societal and cultural issues.

Finally, to entirely grasp the crowd, we have to acknowledge it not only as a socio-technical entity but
in its performativity. The crowd as such only exists as long as individuals engage and contribute in a certain way.
Only from this joint action the crowd emerges and is produced and reproduced in its particular features and only
by contribution, participants can perceive themselves as being a part of it. Thus, understanding the crowd as a
performative act is marked by an inherent contingency that proposes highly relevant implications not only for the
design of educational crowd scenarios but for their scientific exploration as well.

## References

Béguin, P. & Rabardel, P. (2000) Designing for instrument-mediated activity. Scandinavian Journal of Information Systems, 12, 173-190.

Bonney, R., Phillips, T.B., Ballard, H.L., & Enck, J.W. (2016b). Can citizen science enhance public understanding of science? *Public Understanding of Science*, 25(1), pp. 2-16.

Chu, M., Leonard, P. & Stevenson, F. (2012). Growing the base for citizen science. In: Dickinson, J.L. & Bonney, R. (Eds.), Citizen science: public participation in environmental research. Cornell University Press, Ithaca, pp. 69-81.

Clougherty, R. J. & Popova, V. (2013). Crowdsourcing to Assess MOOCs: A Position Paper. MOOCs FORUM, 1(P), pp. 10–13.

Cormier, D., & Stewart, B. (2010). Life in the open: 21st Century learning & teaching. Atlantic Universities' Teaching Showcase, 24, pp. 24-31.

Cress, U., Stahl, G., Ludvigsen, S. & Law, N. (2015). The core features of CSCL: Social situation, collaborative knowledge processes and their design. *International Journal of Computer-Supported Collaborative Learning*, 10(2), pp. 109-116.

Cress, U., Jeong, H. & Moskaliuk, J. (2016). Mass Collaboration as an Emerging Paradigm for Education? *Theories, Cases, and Research Methods*. In U. Cress, J. Moskaliuk & H. Jeong (Eds.), *Mass Collaboration and Education* (pp. 3-27). Cham: Springer International Publishing.

Dourish, P.; Bellotti, V. (1992): Awareness and coordination in shared workspaces. In: M. Mantel und R. Baecker (Hg.): CSCW '92 Proceedings of the 1992 ACM conference on Computer-supported cooperative work. New York, pp. 107-114.

Eveleigh, A., Jennett, C., Blandford, A., Brohan, P. & Cox, A.L. (2014). Designing for Dabblers and Deterring Drop-outs in Citizen Science. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, CHI '14. ACM, New York, NY, USA, pp. 2985-2994.

Fischer, G. (2001). Communities of Interest: Learning through the interaction of multiple knowledge systems. In: S. Bjornestad, R. Moe, A. Morch, & A. Opdahl (Eds.), *Proceedings of the 24th IRIS Conference*, pp. 1-14. Bergen: Department of Information Science.
Fischer, G., Rohde, M., & Wulf, V. (2007). Community-based learning: The core competency of residential, research-based universities. *International Journal of Computer-Supported Collaborative Learning*, 2(1), pp. 9-40.

Fischer, G. (2016). Exploring, understanding, and designing innovative socio-technical environments for fostering and supporting mass collaboration. In: Cress, U., Moskaliuk, J. & Jeong, H. (Eds.), Mass collaboration and education, pp. 43-63. Cham: Springer International.

Ghosh, A. & McAfee, P. (2011). Incentivizing high-quality user-generated content. In: Proceedings of the 20th International conference on World Wide Web. ACM, pp. 137-146.

Harney, O. M., Hogan, M. J., & Quinn, S. (2017). Investigating the effects of peer to peer prompts on collaborative argumentation, consensus and perceived efficacy in collaborative learning. *International Journal of Computer-Supported Collaborative Learning*, 12(3), pp. 307-336.

Haywood, J. (2016). Learning from MOOCs: lessons for the future. Wenner Gren Symposia, 88(6), pp. 69-79, Portland Press.

Jeong, H., Cress, U., Moskaliuk, J. & Kimmerle, J. (2017). Joint interactions in large online knowledge communities: The A3C framework. *International Journal of Computer-Supported Collaborative Learning*, 12(2), pp. 133-151.

Kaplan, A. M.; Haenlein, M. (2011): The early bird catches the news: Nine things you should know about micro-blogging. In: *Business Horizons* 54 (2), S. 105–113.

Knorr Cetina, K. (2001): Objectual practice. In: T. R. Schatzki, K. Knorr Cetina und E. & von Savigny (Hg.): *The Practice Turn in Contemporary Theory*. New York: Routledge, S. 184–197.

Kobori, H., Dickinson, J.L., Washitani, I., Sakurai, R., Amano, T., Komatsu, N., Kitamura, W., Takagawa, S., Koyama, K., Ogawara, T. & Miller Rushing, A.J. (2016). Citizen science: a new approach to advance ecology, education, and conservation. *Ecological Research*, 31(1), pp. 1-19.

Lave, J. & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. New York: Cambridge University Press.

Lu, J.; Lai, M.; Law, N. (2010): Knowledge Building in Society 2.0: Challenges and Opportunities. In: Khine M. S. und Saleh I. M. (Hg.): *New Science of Learning. Cognition, Computers and Collaboration in Education*. New York: Springer, pp. 553-568.

Paavola S., Engeström R., Hakkarainen K. (2012). The Trialogical Approach as a New form of Mediation. In: Moen A., Mørch A.I., Paavola S. (Eds.) Collaborative Knowledge Creation. Technology Enhanced Learning, vol 7, pp. 1-14. SensePublishers, Rotterdam.

Page, R. (2012): The linguistics of self-branding and micro-celebrity in Twitter: The role of hashtags. In: *Discourse & Communication* 6 (2), S. 181–201.

Reinholz, D. (2016). The assessment cycle: a model for learning through peer assessment. Assessment & Evaluation in Higher Education, 41(2), pp. 301-315.

Rogat, T. K., Linnenbrink-Garcia, L., & DiDonato, N. (2013). Motivation in collaborative groups. *International handbook of collaborative learning*, 250-267.

Ross, J., Sinclair, C., Knox, J., Bayne, S. & Macleod, H. (2014). Teacher experiences and academic identity: The missing components of MOOC pedagogy [MOOC Special issue]. *MERLOT Journal of Online Learning* and Teaching, 10(1), pp. 57-69.

Scardamalia, M.; Bransford, J.; Kozma, B.; Quellmalz, E. (2012): New Assessments and Environments for Knowledge Building. In: P. Griffin, B. McGaw und E. Care (Hg.): *Assessment and Teaching of 21st Century Skills*. New York: Springer, pp. 231-300.

Shaffer, D.W. (2006). Epistemic frames for epistemic games. *Computers & Education*, 46(3), 223–234.

Sprinks, J., Wardlaw, J., Houghton, R., Bamford, S., & Morley, J. (2017). Task Workflow Design and its impact on performance and volunteers’ subjective preference in Virtual Citizen Science. *International Journal of Human-Computer Studies*, 104, pp. 50-63.

Walji, S., Deacon, A., Small, J., & Czerniewicz, L. (2016). Learning through engagement: MOOCs as an emergent form of provision. *Distance Education*, 37(2), pp. 208-223.

## Acknowledgments

The research leading to these results has received funding from the Federal Ministry of Education and Research (BMBF) under grant agreement n° 16DHB2119 and was carried out as part of the project: »SCoRe – Student Crowd Research«. The content of this article does not necessarily reflect the opinion of the Federal Ministry of Education and Research.
# Adaptive Support for Collaboration on Tabletop Computers

Abigail C. Evans, Khoury College of Computer Sciences, Northeastern University, ab evans@northeastern.edu
Katie Davis, Information School, University of Washington, kdavis78@uw.edu
Jacob O. Wobbrock, Information School, University of Washington, wobbrock@uw.edu

**Abstract:** We present the design, implementation, and evaluation of a system providing adaptive support for collaborative learning at a tabletop computer. Adapting to support collaboration involves tackling several sub-problems: detecting when a group is struggling, determining when the system should provide support, what sort of support it should provide, and for how long. A classroom evaluation of our system showed it was effective at detecting collaboration breakdowns and provided preliminary evidence that just-in-time adaptive support for collaboration might help to deter disruptive behavior in small group settings.

## Introduction

Computer-supported collaborative learning (CSCL) environments are becoming increasingly sophisticated, enabling new ways for students to work together. However, prior research has demonstrated that students do not always know how to collaborate effectively, which can inhibit the success of small group learning, e.g., (Dillenbourg & Jermann, 2007; Järvelä & Hadwin, 2013; Rogat & Linnenbrink-Garcia, 2011). The findings of this prior work suggest that collaboration itself is a skill that needs to be developed in the classroom.

In this work, we focus on face-to-face, small group collaborative learning at tabletop computers because their large, shared touch screen makes them well-suited for collaboration. However, the mere presence of a large touch screen cannot make up for a lack of collaboration skills—students who do not know how to collaborate effectively in traditional face-to-face settings will continue to struggle at a tabletop computer. Our long-term goal has been to enable tabletop computers to make up for poor collaboration skills. Existing work has addressed the first step in realizing this goal, namely detecting collaboration breakdowns (Evans, Wobbrock, & Davis, 2016; Evans & Wobbrock, 2014). In this paper, we present our work toward the next step—designing ways for the tabletop interface to adapt when breakdowns are detected in order to encourage more effective collaboration. We describe the design and implementation of our system, which was guided by learning theory and educational research, and a classroom evaluation. Our analysis focuses on how the configuration of the adaptations appeared to influence group dynamics over time and through repeated exposure for one group of students particularly prone to collaboration problems. Our key findings are that the adaptations deterred disruptive behavior and halved the length of periods of low-quality collaboration for this group.

## Measuring collaborative learning

To determine what qualifies as *effective* collaborative learning, we use the concept of social regulation, which refers to “the social processes groups use to regulate their joint work on a task” (Rogat & Linnenbrink-Garcia, 2011). Social regulation is an extension of the concept of self-regulation in individual learning to groups of learners in a collaborative setting (Järvelä & Hadwin, 2013; Panadero & Järvelä, 2015; Rogat & Linnenbrink-Garcia, 2011; Volet, Vauras, & Salonen, 2009). Social regulation typically refers to metacognitive processes independent of domain knowledge, although this relationship needs further research.

Rogat and Linnenbrink-Garcia (2011) identify three dimensions of social regulation: *planning* the group's approach to a task, *monitoring* of understanding and progress, and *behavioral engagement*—efforts to get group members to engage with the task. A group's use of each dimension can vary in quality, with high-quality social regulation processes leading to socially-shared regulation, in which all group members maintain joint attention on the learning activity, regularly checking in on the goals, plans, and progress. Low-quality social regulation processes, such as when groups fail to come up with an appropriate plan, can lead to poor learning outcomes. We chose to follow Rogat & Linnenbrink-Garcia's framework when coding for social regulation. Their approach was selected above others because of their detailed descriptions of what high-and low-quality social regulation processes look like in practice; thus, it could easily be applied to our own work.

## Adaptive support for collaborative learning

There has been extensive prior work on adaptive support for collaboration in text-based CSCL environments. For example, guiding systems coach students through a collaboration using adaptive feedback, e.g., (Hadwin, Oshige, Gress, & Winne, 2010; Kumar & Rosé, 2011; Wang, Rosé, & Chang, 2011), and group awareness tools provide feedback to students, often in the form of visualizations, on how they are collaborating, e.g., (Järvelä et
al., 2014; Malmberg, Järvelä, Järvenoja, & Panadero, 2015; Trausan-Matu, Dascalu, & Rebedea, 2014).

Many of the group awareness tools described in the literature make use of the closed nature of text-
based collaborative environments, in which all group interactions can be captured and mined for patterns
associated with the quality of collaboration. In face-to-face group work at a tabletop, however, the computer can
only access direct human-tabletop interaction—the verbal and gestural interactions that learners have with each
other cannot be captured without the use of external sensors. Martinez-Maldonado et al. used external sensors to
capture learners’ verbal and physical interactions during tabletop collaboration in order to visualize each group
member’s level of participation (Martinez-maldonado, Kay, Yacef, Edbauer, & Dimitriadis, 2013; Martinez-
Maldonado, Yacef, Kay, Al-Qaraghuli, & Kharrufa, 2011; Martinez, Collins, Kay, & Yacef, 2011). Although
these visualizations were aimed at teachers, they could also be presented to students as group awareness tools.
At the time of this writing, the work by Martinez-Maldonado et al. remains the only other research providing
real-time adaptive support for collaborative learning at tabletop computers.

It is important to consider how adaptive support should be presented to students and situated in the
larger context of a learning activity or course. Wise (2014) notes it is not enough to simply present students with
interventions—if students are to benefit from an intervention, they need to understand its intent and how to
engage with it. An intervention should be an “integral part of course activity tied to goals and expectations”
(Wise, 2014, p. 206). Interventions should also encourage students to have agency in their own learning.

System design

Providing real-time adaptive support for collaboration involves two major components: (1) detection of
behavior that will trigger adaptations, and (2) the software adaptations themselves. A third consideration is
when and how adaptations are triggered, which we refer to as the configuration.

Detection

We used Evans et al.'s approach to detecting the quality of collaboration at a tabletop computer using a group's touch-interaction patterns (Evans et al., 2016). Evans et al. identified two touch patterns that serve as reliable indicators of a group's collaboration quality: *unrelated touches* (UT), and *overlapping unrelated sequences* (OUS). A *sequence* is a series of touches carried out by an individual that represents a complete action. UT measures the proportion of touches in a sequence that involve objects that are unrelated in the context of a learning activity, with a greater proportion of touches to unrelated objects corresponding to lower quality collaboration—specifically, off task behavior or failure to settle on a suitable plan to complete the activity. OUS is a measure of whether a single student or multiple students are interacting with the tabletop, and when multiple students are interacting, whether they are working with objects that are related to each other in the context of the activity. Evans et al. found that, during periods of high-quality collaboration, students either took turns to interact with the computer or multiple students worked together with related objects. Multiple students interacting with unrelated objects was a sign that students were either working independently or off-task.

Evans et al.'s detection approach was built in to our learning software following the implementation
details described therein (Evans et al., 2016)—two-minute intervals of touch data are checked against the two
touch patterns, UT and OUS, each of which returns a label estimating the quality of collaboration occurring in
that two-minute period as either high-, medium-, or low-quality. It is important to note that the touch patterns
need to be able to distinguish the touches of individual students from those of other group members, which most
tabletop computers are unable to do. We also therefore implemented Evans et al.'s *Group Touch* method of
distinguishing among tabletop users to meet this requirement (Evans, Davis, Fogarty, & Wobbrock, 2017).

Adaptations

We aimed to create adaptations that would be application-independent in principle. Therefore, we developed the
adaptations without a specific application in mind. Initial brainstorming resulted in 11 adaptation ideas, of
which four were implemented and pilot tested. See Figure 1, below, for screen shots of these adaptations:

The *group awareness icon* draws on prior work on group awareness tools e.g., (Järvelä et al., 2014;
Malmberg, Järvelä, Järvenoja, & Panadero, 2015; Trausan-Matu, Dascalu, & Rebedea, 2014). An icon remains
visible on screen for the duration of the activity and changes color according to the detected collaboration
quality, with green representing high-quality collaboration, amber representing medium-quality collaboration,
and red representing low-quality collaboration.

*Control lockout* blocks or disables select controls, such as buttons or menu items, to reduce the number of active application elements. The rationale for this adaptation comes from prior work that found that when group members jump around between many different controls, they are typically either off task or struggling to identify or stick to a task plan (Evans et al., 2016). The aim is to encourage sustained whole-group focus on a
specific piece of the assigned activity.

*Voting* prevents actions that affect the global state of the application, such as deleting work or marking an activity complete and moving on, from being carried out unless the majority of group members vote to proceed. The goal is to encourage discussion and prevent individual students from dominating the group.

*Escalate to authority* sends an alert to the teacher's mobile device requesting human intervention.

<figure><img src="image_88.png" /><figcaption>Figure 1. The adaptations in the study applications. Top left: the group awareness icon, positioned in the corners of the screen, showing amber (medium-quality collaboration). Top right: the prompt, added after the pilot. Bottom left: control lockout has disabled a control that allows students to switch the task. Bottom right: voting blocks actions from being carried out until a majority of the group members vote for it.</figcaption></figure>

## Configuration

The adaptations aim to scaffold effective collaborative interactions by making it difficult for students to interact with the screen in ways deemed to be ineffective. However, no single adaptation covers the full range of desirable collaborative behaviors, all of which need to be adopted by the group if they are to work together effectively. Therefore, we decided that the adaptations should be triggered in a sequence, where each new adaptation is added to those already in use but is only triggered if the collaboration quality does not improve. Layering adaptations can make it easier for students to ground themselves in the activity's expectations, because together the adaptations give a more complete picture of how to be effective than any isolated adaptation. When the collaboration does improve, all active adaptations are removed at once—an example of “fading,” which, in the scaffolding literature, refers to the process of removing supports that are no longer needed (Pea, 2004).

Removing adaptations when collaboration improves also supports Wise's (2014) principle of agency. After a group has experienced a sequence of adaptations once, they will know what is coming next if the sequence begins again. Because the students can anticipate the next step in the sequence, they also have the opportunity to preemptively adopt the behaviors that the next adaptation will enforce, removing the need for that adaptation and thereby preventing it from being triggered. For example, if students know that the next adaptation to be triggered if their collaboration does not improve will be *control lockout*, they can aim to improve by interacting only with objects directly relevant to the aspect of the task they are working on.

The sequence begins with the *group awareness icon*, which remains visible on-screen for the whole activity, changing color according to the detected collaboration quality. The second adaptation, *control lockout*, is triggered when the icon turns red, representing poor collaboration quality, and no other adaptations are active. *Control lockout* was chosen as the first restrictive adaptation based on observations from prior work on tabletop collaboration that groups often jumped into an activity without taking time to plan or check the instructions (Evans et al., 2016). If collaboration does not improve, the third adaptation, *voting*, is added. *Voting* can slow down progress and become frustrating, so it is only triggered after students have had the opportunity to reflect on and improve their collaboration. Finally, if collaboration still does not improve after a period of voting, the application will *escalate to authority* and call the teacher to intervene. There are many other ways to combine,
order, and layer the selected adaptations in a sequence. This particular sequence was chosen initially as the adaptations progress in order from least to most restrictive or interventionist, giving students time to reflect on and change their collaboration behavior before increasing the level of restrictions imposed on the group.

In addition to deciding which adaptations to build and the order in which they should be triggered, we had to decide exactly when an adaptation should be triggered, how it should be presented and explained to students, how long each adaptation should remain active, and when adaptations should be removed. Each of these choices could impact the outcome of the intervention. The criteria we chose for triggering adaptations were derived from Evans et al.'s (2016) analysis of the reliability of using the touch patterns (UT and OUS) to detect collaboration quality. Adaptations were triggered immediately if both touch patterns labeled a single interval low quality, or if two consecutive intervals were labeled low or medium quality by UT. Adaptations remained active until both touch patterns produced high quality labels for four consecutive minutes.

## Classroom evaluation

The prototype adaptations were evaluated in a classroom setting. Our research questions were: (1) Are the adaptations triggered appropriately? (2) How do students respond when an adaptation is triggered?

## Participants and apparatus

Eleven 10th and 11th graders (9 male, 2 female) participated in the study. The students were enrolled in a six-week beginner's course on video game development offered as part of a college preparation program serving low income students. The course used in this study was one of the program's elective options. The course focus was educational games, and students were asked to design a game to raise awareness of an environmental issue: snow leopard conservation. In the first part of the course, the students learned about snow leopards and ongoing efforts to protect the species. In the second part of the course, the students learned the basics of video game development using Unity (a 3-D game development platform: see https://unity3d.com/). All but one of the students were new to writing code and all were new to video game development and Unity. In the third and final part of the course, the students worked on their game projects.

The students were randomly assigned to three small groups—two groups of four and one group of three. The students stayed in the same groups for all collaborative activities. The groups used three custom-built learning applications across four sessions at a Microsoft Surface Hub, which features a 55" multitouch screen. The applications addressed the learning objectives for the class sessions in which they were to be used and were designed to be used alongside other activities.

*Snow Leopards 101* is an introduction to snow leopards and their ecosystem adapted from an existing non-tabletop curriculum (Facing the Future & Snow Leopard Trust, 2009). In *Help a Scientist*, students take on the role of scientists studying wild snow leopards in Mongolia, gathering and analyzing photographic data. *Game Challenge* was used during the third part of the course. This application featured a partially created game world that students could build upon and adjust to create different outcomes while learning important and complex concepts used in Unity, such as how to work with its built-in physics engine.

The Surface Hub is primarily intended to be used in a vertical orientation but for this study it was placed flat on a table in a small breakout room across a hallway from the main classroom. A wide-angle video camera was mounted on a wall so that students could freely move around the table. The camera was angled toward the screen so that it could capture every touch and the interactions among the group members. Ten group sessions at the tabletop were video recorded for this study and the computer logged every touch.

## Study design

The study was split into three phases: (1) baseline data collection; (2) a pilot test of the initial implementation of the adaptations, which resulted in a revised implementation; and (3) an evaluation of the final implementation. The *Snow Leopards 101* application was used for baseline data collection, *Help a Scientist* was used for the pilot test, and *Game Challenge* was used for the final evaluation taking place two and a half weeks after the pilot. Students used each application for 25 minutes per session. *Game Challenge* was used over two sessions.

The adaptations were piloted with Group 1 only. The final adaptations were available for Group 1 and Group 3 during the final evaluation—if poor quality collaboration were detected, the adaptations would trigger. Group 2 served as a control so the adaptations were not available for them during phase 3, although the collaboration quality labels output by the touch patterns were still recorded. We wanted to look for differences in how groups dealt with collaboration problems with or without the intervention of adaptations. Students were randomly assigned to groups and groups were randomly assigned to conditions in an attempt to reduce differences between groups. However, given that it was unlikely the groups would be truly equal in a study of this scale, the goal of the baseline data collection was to understand the differences between groups.
The first time a group used the tabletop with adaptations enabled, a researcher explained to them that the computer was tracking how they interacted with it in order to help them work together effectively. They were told that the color of the *group awareness icon* updated every minute to give them feedback on how they were collaborating and that if it turned red or stayed amber for several minutes, the computer would block some controls or ask them to vote before carrying out certain actions. They were also told that the adaptations would go away if the icons stayed green for four minutes, and that they could keep the icons green by maintaining a shared focus on the assigned task, discussing the content, and listening to each other's ideas.

## Pilot test

The adaptations were piloted with Group 1, a group of four boys randomly assigned to the pilot, as they used the *Help a Scientist* application. The *group awareness icon* (Figure 1, top left) was displayed in two of the corners of the screen so that it would be visible to all group members without obstructing the work area. *Control lockout* (Figure 1, bottom left) was configured to trigger if the icon turned red when no other adaptations were active. Before activation, a warning popped up on screen stating, "Some buttons and controls will be temporarily disabled to help you to stay focused on the task." The warning remained on screen for 30 seconds, blocking all interaction for the time that it was on screen. *Voting* (Figure 1, bottom right) was also preceded by a 30-second warning message: "You will temporarily be asked to vote in order to carry out certain actions such as changing activity or closing windows." If a group was asked to vote on an action, a message was shown describing what the group members were voting on. Finally, *escalate to authority* would send an alert to the teacher that the group might be struggling. Unlike *control lockout* and *voting, escalate to authority* was invisible to the students.

To evaluate the piloted adaptations, the first author reviewed the video of the pilot session and the computer's log files. Given that the purpose of the pilot was to get feedback on a number of design choices quickly enough to make changes in time for a formal evaluation two and a half weeks later, it was not possible to do a full in-depth analysis of the video at this point. Instead, observations were made of what the group was doing in the run up to an adaptation being triggered and how group members responded. This informal analysis was confirmed by a formal analysis of the pilot once the study was complete.

The sequence of adaptations triggered twice during the pilot: at 8 minutes into the session and again at 23 minutes. In both cases, the sequence progressed from *control lockout* to *voting* and was canceled before *escalate to authority*. This means that low-quality collaboration was still detected after at least one interval of *control lockout*, causing *voting* to be triggered. The group was able to sustain high-quality collaboration for at least three intervals while *voting* was active, so the adaptations were removed without messaging the teacher.

After reviewing the verbal and physical interactions among the students, we determined that the adaptations triggered appropriately and that they did appear to get the group to collaborate more effectively. However, the positive behavior change that occurred seemed to be a result of coercion—the adaptations proved so annoying that they forced the students to improve their working style without encouraging reflection. The main lesson learned was that it was too difficult to get the adaptations removed. Students did initially improve their collaboration but it degraded after a couple of minutes without feedback that they were on the right track. Additionally, the students read the warnings that appeared with each adaptation, but they complained that they felt they were already doing what was asked of them. When *voting* triggered a second time, they were quickly able to get back on track, maintain a shared focus but with little discussion or deep engagement with the content.

As a result of these observations from the pilot study, we reduced the length of time a group had to sustain high-quality collaboration in order to remove the adaptations from four minutes to two minutes. We also added an additional adaptation, *prompt* (Figure 1, top right), before *control lockout*. This adaptation simply provides students with a reminder of what is expected of them and gives them the opportunity to self-correct before the restrictive adaptations are triggered. The time that warning messages remained on-screen before activating *control lockout* and *voting* was reduced to 20 seconds as the video showed that to be enough time for the students to read the message. The rest of the implementation details remained the same.

## Data analysis

Formal analysis of all study sessions began by coding the videos for social regulation using the same codes that were used to develop the collaboration quality detection approach (Evans et al., 2016). The bulk of the coding was carried out by a doctoral student who was unfamiliar with the adaptations being evaluated. To establish inter-rater reliability, the student and the first author independently coded a session from the baseline data collection phase of the study. Codes were applied to episodes (Chi, 1997), and each episode could contain multiple codes. The majority of codes had a Cohen's kappa (κ) above 0.61, typically considered "substantial" agreement, with several codes above 0.81, or "almost perfect" agreement (Landis & Koch, 1977).

To determine if the adaptations were triggered appropriately, we looked at the video codes in the
intervals leading up to the triggering of each adaptation. If the video codes that occurred in that same interval
were primarily negative, the adaptations were considered appropriate. We also looked for video intervals that
showed improvement while adaptations were active to determine if the improvements were detected. Finally,
we looked for video intervals that showed collaboration problems that were undetected by the computer.

To understand how students responded to the adaptations, we first looked at the video codes in
intervals immediately following the activation of an adaptation. For an adaptation to be considered successful,
the video codes should show improved collaboration. Due to the frustration observed in the pilot, we also
reviewed the videos from the evaluation sessions to understand students’ emotional responses to the adaptations.

Results

The results from the summative evaluation sessions suggest that the revised approach to triggering adaptations was more effective. Collaboration improved, along some dimensions, immediately following the first interventionist adaptation in the sequence (*prompt*) every time it was triggered, meaning that later adaptations were never triggered. This is a positive result because the *prompt* appeared to lead to more effective collaboration very quickly and consistently, but a side effect is that most of the adaptations were not therefore tested in this phase of the study. Additionally, the adaptations could not address all collaboration problems—disengaged students who showed no inclination to participate in the group work remained disengaged whether or not adaptations were present, and off-task interactions taking place away from the tabletop computer could not be detected. The biggest positive impacts of the adaptations were reduced disruption caused by individual students and less time spent engaged in low-quality collaboration.

Each group used the tabletop computer twice during the final evaluation phase of the study. Of the two
groups that used the tabletop with adaptations available (Groups 1 and 3), only Group 1 triggered the
adaptations. In Group 1's first session in the evaluation phase, they triggered the *prompt* once, at 4 minutes into
the session. The video showed that the prompt was triggered at an appropriate time, after several minutes of the
group being off task and pressing buttons on-screen without any explicit coordination. The prompt included a
reminder that students should make sure they understood the task goal and that they were working toward it.
When the prompt appeared, the group did revisit the instructions. The sequence of adaptations did not progress
to the next stage because, immediately after reading the instructions, the group began to engage in on-task work.
However, only some students in the group were engaged during most subsequent episodes. The disengaged, off-
task students typically refrained from touching the computer after the prompt appeared in this session and were
therefore undetectable. This behavior was noticeably different from Group 1's baseline and pilot sessions, in
which these students would attempt to interact with the screen without fully engaging in the activity, disrupting
students who were engaged.

In Group 1's second session in the evaluation phase, they triggered the *prompt* twice, at 9 and 17 minutes. The sequence did not progress beyond the *prompt* in either case. The video analysis showed that, in the intervals leading up to the first prompt, the whole group was engaging in primarily low-quality collaboration for around a minute, followed by a period of high-quality collaboration between two students with the other two students completely off task. After the prompt was dismissed, the off-task students continued to be off-task but refrained from touching the screen while the other two students worked collaboratively. In the intervals leading to the second instance of the *prompt*, one of the off-task students became interested in the screen, trying to take control of a particular object by repeatedly hammering on it. This interaction was highly disruptive to the engaged students, who were close to completing the task. When the prompt appeared for the second time, the engaged students appeared annoyed by it but they were able to dismiss it quickly and it deterred the other student from hammering on the screen. He sat back from the computer but made some verbal contributions to the collaboration—encouraging his teammates as they solved the assigned task.

We also compared the length of Group 1's periods of sustained low-quality collaboration taking place
at the computer in three types of intervals: (1) "pre-adaptation"—intervals that cause an adaptation to trigger;
(2) "adaptation active"—intervals in which an adaptation is active; and (3) "no adaptation"—intervals during
which no adaptations are present. "Periods of sustained low-quality collaboration" means periods of time with
one or more continuous episodes of low-quality collaboration.

Table 1 shows that the adaptations appeared to reduce the length of periods of low-quality
collaboration involving the computer. The median length of periods of low-quality collaboration was
consistently longer during intervals that caused an adaptation to trigger than during other intervals in all sessions
where adaptations were present. In the baseline session, when the adaptations were not in use, the median length
of sustained low-quality collaboration was at least twice that of the sessions where adaptations were available.
In both final evaluation sessions, there were considerably more occurrences of low-quality collaboration when
no adaptations were present than in other intervals. This effect occurred because, when there were no
adaptations present, occurrences of low-quality collaboration were brief and punctuated by high-quality collaboration, causing the number of occurrences to increase and the length of the occurrences to decrease.

<table><caption>Table 1: The median length (in seconds) of sustained periods of low-quality collaboration in Group 1's sessions</caption><thead><tr><th rowspan="2">Interval Type</th><th colspan="2">Baseline</th><th colspan="2">Pilot</th><th colspan="2">Evaluation 1</th><th colspan="2">Evaluation 2</th></tr><tr><th>Median length (s)</th><th># of occurrences</th><th>Median length (s)</th><th># of occurrences</th><th>Median length (s)</th><th># of occurrences</th><th>Median length (s)</th><th># of occurrences</th></tr></thead><tbody><tr><td>Pre-adaptation</td><td></td><td></td><td>16</td><td>7</td><td>106</td><td>1</td><td>15</td><td>5</td></tr><tr><td>Adaptation active</td><td></td><td></td><td>9.5</td><td>19</td><td>0</td><td>0</td><td>11</td><td>2</td></tr><tr><td>No adaptation</td><td>20</td><td>11</td><td>9</td><td>6</td><td>8</td><td>19</td><td>10</td><td>15</td></tr></tbody></table>

Although Group 3 did not trigger the adaptations, the video analysis showed a serious collaboration problem—only one student engaged with the task while the other two group members sat back from the tabletop computer, chatting and using their phones. This highlights a known limitation of the detection approach used in our system, namely that it is only able to detect interactions with the screen (Evans et al., 2016).

## Discussion and conclusion

The results show that our system was able to detect certain collaboration problems—primarily disruption caused by individual students and poor coordination among group members. Adaptations were triggered appropriately in these instances. How students in Group 1 received the adaptations differed by how motivated they were to engage with the activity—motivated students adjusted their behavior in a positive direction but already disengaged students were put off completely. Although this effect led to positive outcomes for the motivated students, who were able to make progress where they had previously been blocked by disruptive students, it was problematic for the disengaged students.

Although Group 3 never triggered the sequence of adaptations, both Group 1 and 3 saw the *group awareness icon* change color in response to the collaboration quality detected by the computer. However, with the exception of a single utterance in the pilot, there was no evidence that either group made use of the icon.

The *prompt* appeared to be effective at encouraging Group 1 to think about how they were interacting and to make sure they were working on the task as assigned. The first time the prompt appeared, the students took time to read it and follow its advice. The second and third time it appeared, the students were quicker to dismiss it, possibly due to familiarity, but both times, it caused an off-task student to stop disruptive behavior.

We consider the fact that, once *prompt* was added to the sequence of adaptations, no further adaptations were triggered, to be a positive outcome for this work. In all cases, the prompt was followed by sustained periods of high-quality collaboration, albeit only for those students who were engaged in the task. Beyond this study, the ideal outcome of using these adaptations over a longer period of time would be that they render themselves unnecessary—with such an outcome it would be possible to conclude that the adaptations successfully scaffold effective collaboration, fading once a group has adopted the principles that the adaptations support (Pea, 2004).

In this paper, we have described the design, implementation, and evaluation of a set of tabletop software adaptations to encourage effective collaboration when problems are detected. Presented as a sequence that grows increasingly restrictive if collaboration does not improve, our adaptations showed promise as supports for groups that struggle with disruptive behavior. Due to the small scale of our classroom field evaluation, further study is needed to determine the extent of our approach's effectiveness, but overall, our work demonstrates that tabletop applications that can detect and adapt to poor-quality collaboration can encourage more effective group work by deterring disruptive behavior.

## References

* Chi, M. T. H. (1997). Quantifying Qualitative Analyses of Verbal Data: A Practical Guide. *Journal of the Learning Sciences*, 6(3), 271–315.

* Dillenbourg, P., & Jermann, P. (2007). Designing Integrative Scripts. In F. Fischer, I. Kollar, H. Mandl, & J. M. Haake (Eds.), *Scripting Computer-Supported Collaborative Learning: Cognitive, Computational and Educational Perspectives*. New York: Springer.

* Evans, A. C., Davis, K., Fogarty, J., & Wobbrock, J. O. (2017). Group Touch: Distinguishing Tabletop Users in Group Settings via Statistical Modeling of Touch Pairs. In *Proceedings of CHI 2017*, 35–47. New
York: ACM Press.

Evans, A. C., Wobbrock, J. O., & Davis, K. (2016). Modelling collaboration patterns on an interactive tabletop in a classroom setting. In *Proceedings of CSCW 2016*, 860–871. New York: ACM Press.

Evans, A., & Wobbrock, J. O. (2014). Filling in the gaps: capturing social regulation in an interactive tabletop learning environment. In *Proceedings of ICLS*, 1157–1161. Boulder, Colorado: ISLS.

Facing the Future, & Snow Leopard Trust. (2009). *Engaging Students in Conservation: Protecting the Endangered Snow Leopard*. Seattle, WA. Retrieved from http://www.snowleopard.org/learn/resources-for-educators

Hadwin, A. F., Oshige, M., Gress, C. L. Z., & Winne, P. H. (2010). Innovative ways for using gStudy to orchestrate and research social aspects of self-regulated learning. *Computers in Human Behavior*, 26(5), 794–805.

Järvelä, S., & Hadwin, A. F. (2013). New Frontiers: Regulating Learning in CSCL. *Educational Psychologist*, 48(1), 25–39.

Järvelä, S., Kirschner, P. A., Panadero, E., Malmberg, J., Phielix, C., Jaspers, J., Koivuniemi, M., & Järvenoja, H. (2014). Enhancing socially shared regulation in collaborative learning groups: designing for CSCL regulation tools. *Educational Technology Research and Development*, 63(1), 125–142.

Kumar, R., & Rosé, C. P. (2011). Architecture for Building Conversational Agents that Support Collaborative Learning. *IEEE Transactions on Learning Technologies*, 4(1), 21–34.

Landis, J., & Koch, G. (1977). The measurement of observer agreement for categorical data. *Biometrics*, 33(1).

Malmberg, J., Järvelä, S., Järvenoja, H., & Panadero, E. (2015). Computers in Human Behavior Promoting socially shared regulation of learning in CSCL: Progress of socially shared regulation among high- and low-performing groups. *Computers in Human Behavior*, 52, 562–572.

Martinez-maldonado, R., Kay, J., Yacef, K., Edbauer, M. T., & Dimitriadis, Y. (2013). MTClassroom and MTDashboard: supporting analysis of teacher attention in an orchestrated multi-tabletop classroom. In *Proceedings of CSCL*, 320–327. Madison, WI: ISLS.

Martinez-Maldonado, R., Yacef, K., Kay, J., Al-Qaraghuli, A., & Kharrufa, A. (2011). Analysing frequent sequential patterns of collaborative learning activity around an interactive tabletop. In *Proceedings of EDM*, 111–120. International Educational Data Mining Society.

Martinez, R., Collins, A., Kay, J., & Yacef, K. (2011). Who did what? Who said that?: Collaid: an environment for capturing traces of collaborative learning at the tabletop. In *Proceedings of ITS*, 172–181. New York: ACM Press.

Panadero, E., & Järvelä, S. (2015). Socially Shared Regulation of Learning: A Review. *European Psychologist*, 20(3), 190–203.

Pea, R. D. (2004). The social and technological dimensions of scaffolding and related theoretical concepts for learning, education, and human activity. *Journal of the Learning Sciences*, 13(3), 423–451.

Rogat, T. K. & Linnenbrink-Garcia, L. (2011). Socially Shared Regulation in Collaborative Groups: An Analysis of the Interplay between Quality of Social Regulation and Group Processes. *Cognition and Instruction*, 29(4), 375–415.

Trausan-Matu, S., Dascalu, M., & Rebedea, T. (2014). PolyCAFe-automatic support for the polyphonic analysis of CSCL chats. *International Journal of Computer-Supported Collaborative Learning*, 9(2), 127–156.

Volet, S., Vauras, M., & Salonen, P. (2009). Self- and social regulation in learning contexts: An integrative perspective. *Educational Psychologist*, 44(4), 215–226.

Wang, H. C., Rosé, C. P., & Chang, C. Y. (2011). Agent-based dynamic support for learning from collaborative brainstorming in scientific inquiry. *International Journal of Computer-Supported Collaborative Learning*, 6(3), 371–395.

Wise, A. F. (2014). Designing pedagogical interventions to support student use of learning analytics. In *Proceedings of LAK*, 203–211.

## Acknowledgments

We thank Cindy Hmelo-Silver for her guidance on this project. This work was supported in part by the National Science Foundation under awards IIS-0952786 and IIS 1053868. Any opinions, findings, conclusions or recommendations expressed in this work are those of the authors and do not necessarily reflect those of the National Science Foundation.
# Integrative Visualization:
Exploring Data Collected in Collaborative Learning Contexts

Ben Rydal Shapiro, Georgia Institute of Technology, ben@benrydal.com

**Abstract:** The development and use of computational approaches to make sense of data collected in collaborative learning contexts is expanding rapidly in the Computer-Supported Collaborative Learning (CSCL) and Learning Sciences (LS) communities. However, developing and using these approaches in ways that maintain a commitment to theory and situational context is a significant challenge. This paper proposes *integrative visualization*, a computationally assisted, human-centered process to visually explore data collected in collaborative learning contexts in a way that maintains a commitment to theory and situational context. To do so, this paper summarizes how integrative visualization supported the development and use of interaction geography, an approach to describing, representing, and interpreting collaborative interaction across the physical environment. This paper concludes by emphasizing the need to further develop integrative visualization in the CSCL and LS communities through stronger connections with the information visualization community.

## Introduction

The development and use of computational approaches to make sense of data collected in collaborative learning contexts is expanding rapidly in the Computer-Supported Collaborative Learning (CSCL) and Learning Sciences (LS) communities. In particular, computational approaches are providing ways to extend traditional data analysis of phenomena such as language, gesture, or gaze important to studying collaborative learning (Rosé et al., 2008; Schneider & Pea, 2014). Alternatively, these approaches offer resources to reason about larger data sets and data scales including multimodal data collected across classroom settings (Knight et al., 2017). However, developing and using these approaches in ways that maintain a commitment to theory and situational context is a significant challenge (Wise & Schwarz, 2017; also see Law et al., 2018). This challenge also reflects broader efforts in and outside of education to develop “orientations towards and frameworks for data science that are not reducible to mere efficiency” (Zegura, DiSalvo & Meng, 2018; also see Kahn, 2017; Berland et al., 2017).

This paper proposes *integrative visualization*, a computationally assisted, human-centered process to visually explore data collected in collaborative learning contexts in a way that maintains a commitment to theory and situational context. Integrative visualization extends visual methods of exploratory data analysis (Tukey, 1977) to iteratively transcribe data collected in context in order to organize data and develop codes, categories, units of analysis, and questions that support the development of grounded theory (Glaser & Strauss, 1967) and new computational tools. In particular, integrative visualization extends the practice of “integrative diagramming” (Strauss, 1987) through techniques of exploratory data analysis and is summarized as follows:

data context → transcription amplified by visualization → grounded theory & computational tools

This paper illustrates this process of integrative visualization by describing how it supported the development and use of interaction geography, an approach to describing, representing, and interpreting collaborative interaction across the physical environment (Shapiro, Hall & Owens, 2017; Shapiro & Hall, 2018). This paper concludes by emphasizing the need to further develop integrative visualization in the CSCL and LS communities through stronger connections with the information visualization community.

## An example of integrative visualization

### Data context

The example described in this paper draws from data collected during a three-year project in collaboration with a nationally renowned museum located in the mid-South region of the United States. This project sought to understand how visitors to this museum cultivated interests in and learned about the diverse historical and cultural heritage of American Roots music during and after their visit. Two initial research questions guided this work. First, this work aimed to understand the organization of visitors' activity not only at single museum exhibits as was typically the case in existing research but also as visitors moved across gallery spaces over their complete museum visit. Second, this work sought to understand how visitors furthered their own interest-driven engagement and learning through their activity across gallery spaces.
To answer these questions, a purposive sample of complete museum visits across 22 visitor group cases (2–5 visitors per group) was collected over a period of six weeks. This sample was collected in close collaboration with museum partners, many generous families and visitors participating in this research, and followed institutional review board (IRB) protocols. Data from these 22 cases included continuous, multi-perspective video and audio records (72 hrs total) of each visiting groups’ movement, interaction, and social media use collected through small, unobtrusive cameras that each visitor in a group wore as necklaces for the duration of their visit with no researchers present (visits ranged from 30 min to 4 hrs). This data provided detailed records of visitors’ interaction (e.g., conversation, movement, use of cell phones and cameras).

This data presented two novel research challenges. First, existing research was only beginning to make sense of complex, multi-perspective audio and video data in ways necessary to answer the questions informing this work (see Marin, 2013; Steier, 2014; Taylor & Hall, 2013 for early work). Second, apart from a few novel and inspirational examples (see Fouse et al., 2011), computational tools to support this work did not exist. For example, contemporary transcription software (e.g., InqScribe, NVivo) and video editors (e.g., Final Cut Pro, Adobe Premier) were not designed to account for the spatial dimension of collaborative interaction as people moved. Likewise, information visualization and visual analytics software used to study visitor activity in settings such as museums did not operate at a fine enough grain size to answer the research questions informing this work and also typically focused exclusively on visitors’ movement ignoring their conversation.

In summary, this particular cultural heritage museum along with these initial questions, goals, types of data collected, and unique challenges framed the data context of this work. This data context included a theoretical commitment to studying visitors’ interest-driven engagement and learning as they moved across gallery spaces through continuous, multi-perspective video records of visitor activity. The following section uses a set of figures and analysis to illustrate an iterative process of transcribing this data amplified by visualization in a way that maintains this theoretical commitment and is driven by this data context.

## Transcription amplified by visualization

Figure 1 is a hand drawn sketch that represents a first attempt to make sense of this data and specifically, a five-member family’s activity within a particular museum gallery space. Namely, the sketch tries to make sense of the family’s experiences across a gallery space from watching five separate video records.

<figure><img src="image_89.png" /><figcaption>Figure 1. Manual sketch of a family’s activity in a museum gallery transcribed from multi-perspective video.</figcaption></figure>

Importantly, the sketch draws from a set of famous architectural drawings known as the “Manhattan Transcripts” developed by architect Bernard Tschumi: The Manhattan Transcripts aimed to expand the theorization and design of architecture beyond space and form to also include the relations between space, movement, and event or what happens in space (Tschumi, 1994). The sketch uses the Manhattan Transcripts to organize aspects of this family’s activity in this gallery space across dimensions of object/space, movement, and event. For example, the 2nd column describes one family member’s (named Adhir) activity at an exhibit. The first row of this column describes the exhibit/object of focus (a musician named Hank Williams); the second row characterizes Adhir’s movement as frozen or not moving; the third row characterizes events/things Adhir does at the exhibit (stands in reverence, talks, takes a photograph). Altogether, the figure shows a commitment to
understanding visitor groups as a mobile unit (e.g., multiple participants are placed in columns across different times of their visit) and questions concerning how and what happens as each family member moves.

Figure 2 is a computationally generated sketch produced 4 months after the previous sketch. It illustrates an early attempt to transcribe the same five-member family’s conversation as they move across part of the museum gallery space described previously. Put differently, this sketch illustrates an initial attempt to address a fundamental problem regarding how to make sense of people’s conversation as they move.

<figure><img src="image_90.png" /><figcaption>Figure 2. Extending the space-time cube to transcribe a family’s conversation across a museum gallery space.</figcaption></figure>

The sketch shows a floor plan of part of the gallery space (i.e., looking down on the space). This floor plan is tilted in 3D isometric perspective. A set of images is shown on top of the floor plan. These images correspond to six museum exhibits that line a semicircle drawn on the floor plan. A timeline in minutes and seconds along with grid lines that separate 6 regions of the floor plan (corresponding with the 6 exhibits along the semicircle) extend downward/below the floor plan. Turns of transcribed talk from each family member are placed along this timeline and in regions of this gridded space (i.e., placed in time and space). Turns of talk are also grouped into topically related conversations (typically spatially related conversations about exhibit content). Thus, the position of turns of talk/conversations vertically on the timeline and horizontally across the floor plan indicate when (over approximately 5 minutes) and where (along the semicircle of six exhibits) each turn of talk/conversation occurs. In addition, turns of talk in the first conversation (top of the timeline) are overlaid by colored lines that indicate speaker (i.e., color indicates which family member speaks that conversation turn).

Altogether, this sketch illustrates an initial effort to extend a geographical perspective called time geography and a visualization system known as the space-time cube (Hagerstrand, 1970) to begin to develop codes and categories to explore this family’s conversation over space and time in new ways. Doing so necessitated categorizing and grouping conversation in ways that foregrounded the spatial and mobile dimensions of visitors’ conversation (i.e., reflective of the theoretical commitments and data context of this work).
Figure 3, produced approximately three months after the previous figure visualizes different aspects of the same family's interaction over the total time they spent in the same part of the gallery space (approximately 8 minutes) in a small multiple format (Tufte, 1990). The top left visualization in the figure titled "overlaid talk" shows the turns of talk from the family colored by speaker. The visualization below this titled "conversation structures" isolates visual boxes that group turns of talk into topically related conversations while the visualization adjacent titled "transcribed talk" shows all transcribed talk for the family. In comparison to Figure 2, this includes about 3 more minutes of talk. The visualization above titled "mobility paths" shows the family's movement across this space and over time as lines or paths (color again indicates family member). The four visualizations to the right isolate each family member's individual movement and turns of talk: the mother named Mae is shown in purple, her son and daughter named Jeans and Lily are shown together in green and yellow respectively as their movement is nearly identical, their 6-year old brother named Blake is shown in blue, and Lily's fiancé Adhir is shown in orange. By visualizing each family member's movement and conversation as layers over space and time comparisons can be drawn across family members to simultaneously study who speaks, what is said, and where each family member goes in this gallery space. For example, the figure supports asking new questions such as how parents' conversation across gallery spaces structures and responds to children's movement and conversation.

<figure><img src="image_91.png" /><figcaption>**Figure 3.** Small multiple of a family's movement and conversation over space and time in a museum gallery.</figcaption></figure>

Figure 4 presents a more refined computationally generated visualization produced four months later of two members of this family in this gallery space, Blake and Adhir (shown once again in blue and orange respectively). On the left, the floor plan is now shown in 2D (looking directly down on the space) and depicts the full gallery space. The semicircle set of six exhibits is now clearly shown and one exhibit is marked with "Williams" to indicate it features content about a famous musician named Hank Williams. Blake and Adhir's movement is shown across the floor plan or in "floor plan view" indicating where they travel while visiting this gallery space. A timeline extends to the right of the floor plan as opposed to below/under the floor plan. In comparison to Figures 2 and 3, this subtle rotation of the timeline (suggested by a collaborator, Lara Heiberger, in the context where this data was collected) aids in interpreting complex space-time visualizations. In this case, similar to the previous figures but more clearly illustrated here, the "space-time view" (Hagerstrand, 1970) extends
Blake and Adhir's movement on the floor plan horizontally over time. This view shows how they interact with exhibits and one another over time. For example, the space-time view shows that after entering the gallery space (top left of the floor plan view and beginning of the space-time view), Adhir and Blake walk together toward the Hank Williams exhibit. Subsequently, Adhir stands for almost 5 minutes at the Hank Williams exhibit, as indicated by his horizontal orange path in the space-time view that extends from approximately minutes 0–5 and corresponds to the vertical position of the Hank Williams exhibit in the floor plan view. In the meantime, while Adhir is standing, Blake is moving quickly (apparently running) back and forth across the gallery space (i.e., across the semi-circle of exhibits on the floor plan) in multiple attempts to draw Adhir away from the Hank Williams exhibit. After four failed attempts, Blake finally succeeds in leading Adhir on what is described as a tour of other exhibits in the gallery, indicated by their intertwined paths from approximately minutes 5-6. The change in line pattern in Blake's path distinguishes between three different horizontal areas of space on the floor plan providing some description of horizontal movement on the floor plan in the space-time view.

In comparison to previous figures, this figure begins to provide more detailed ways to interpret collaborative interaction across a physical environment (i.e., reflective of the original commitments of this work). For instance, the figure characterizes new, path-based units of collaborative interaction such as Blake's tour that support asking and answering new types of questions including how young children use their movement to manage their families as resources for their own interest-driven engagement and learning.

<figure><img src="image_92.png" /><figcaption>Figure 4. Adhir (orange) and Blake's (blue) movement over space and space-time in a museum gallery.</figcaption></figure>

## Grounded theory and computational tools

Figure 5 was produced over a year after Figure 4 and represents a significant amount of theoretical and computational development. The figure is a screenshot from a dynamic visualization tool called the Interaction Geography Slicer (IGS), which allows for new forms of interaction and multimodal analysis. The figure also illustrates a refined method to transcribe movement and conversation, in this case, of all five members of the previous family (including Blake and Adhir) in the previously described gallery space. This method called Mondrian Transcription draws inspiration from the Manhattan Transcripts and the Modernist artist, Piet Mondrian (1872–1944), particularly his use of lines in relation to forms, which resemble how movement and conversation are represented, coded, and categorized in Mondrian Transcription. The top half of the figure shows the family's movement and the bottom half shows their conversation in relation to their movement (i.e., the family's movement is shown in gray beneath their conversation to link the two halves of the figure). Conversation is transcribed and organized in ways introduced previously. First, each turn at talk is shown as a colored line to indicate which family
member speaks that conversation turn (indentations indicate overlapping speech). Second, colored lines of talk are gathered into boxes that group topically related sequences of conversation turns and movement (e.g., usually related to artifacts/musicians in this setting). In the space-time view, each box marks the start, duration, and end of a sequence. In the floor plan view, conversation turns and separate (in time) sequences accumulate within regions of gridded space--the box thickness in the floor plan view increases with each repeated sequence within a region of space (resembling a "heat map" of talk in place). For example, the region of space around the Hank Williams exhibit has the largest number of conversation turns (indicated by the many colored lines of talk) and is enclosed by a dense box that reflects five separate (in time) sequences occurring at the Hank Williams exhibit.

<figure><img src="image_93.png" /><figcaption>**Figure 5.** Screenshot from the Interaction Geography Slicer (IGS) of family's movement and conversation.</figcaption></figure>

In the figure, the highlighted sequence (i.e., readable conversation) in the space-time view expands the conversation turns of one particular sequence. In other words, the highlighted sequence illustrates an analytic “operation” possible within the IGS on data collected in this work. As the figure shows, one can use the IGS to select, magnify, visualize, and read conversation turns. Not shown is the additional ability to use the IGS to watch and listen to video/audio from the perspective of each family member gathered as part of this work. The IGS syncs
multi-perspective audio and video to visualizations such as Figure 5. As a result, anywhere a user clicks on the visualization activates audio or video from the perspective of an individual.

The figure shows how these refined tools/methods provide ways to engage in exploratory data analysis (Tukey, 1977) that pays careful, contextual attention to how people engage and learn across museum gallery spaces. For example, the highlighted conversation or sequence in the figure from approximately minutes 4–5 in the space-time view encompasses a complex mesh of activity around the Hank Williams exhibit. Reading this sequence of activity in relation to the rest of the figure shows how: 1) Lily (yellow) soothes the emotions of Adhir (her fiancé) by hugging and consoling him as he compares the Hank Williams exhibit to a “grave” (in line 8); 2) Jeans (green) gives Lily and Adhir privacy by leading a frustrated Blake away from the Hank Williams exhibit (the extension of their movement paths upwards in the floor plan and space-time views indicating their movement away from the exhibit); 3) Blake and Jeans rejoin Lily and Adhir as Adhir continues to share his own account of Hank Williams’s painful life; 4) Mae (Mom in purple), who has been standing near Adhir and Lily and observing her family’s interaction, helps Blake lead Adhir on a tour of other exhibits by saying to Adhir, “but you gotta.. you gotta go see Bill Monroe’s mandolin” (in lines 22–23); and 5) Evidently fully aware of Blake’s ongoing project to lead a tour, Adhir whispers to Blake, “ok let’s go” and they move forward together to the next exhibit along the semicircle of exhibits (at the end of the highlighted conversation; see Shapiro, Hall & Owens, 2017).

Altogether, the ability to read this sequence of activity in relation to the rest of the figure by exploring data within the IGS reveal phenomena and relations between phenomena such as Blake’s tour, Adhir’s persistent engagement with the Hank Williams exhibit, a mother’s efforts to support her children’s engagement, and particularly important units of engagement or “peak engagement contours” such as the previously described sequence of activity during this family’s visit to this gallery space. In other words, the figure communicates concepts and methods of interaction geography, a new approach to describing, representing, and interpreting people’s collaborative interaction as they move across the physical environment, and how to use interaction geography to study visitors’ interest-driven engagement and learning across a museum gallery space.

## Discussion and conclusion

In summary, the development and use of interaction geography was interleaved with and furthered a line of exploratory data analysis that led to new questions, units of analysis, and computational tools to describe how visitors pursued their own interest-driven engagement and learning in a museum. Put differently, the example characterizes integrative visualization, a computationally assisted, human-centered process to visually explore data in a way that maintains a commitment to theory and situational context.

Though illustrated through a single example in this paper, integrative visualization is proposed as a generalizable process researchers or practitioners can leverage to make sense of data collected in collaborative learning contexts. Integrative visualization begins with a data context, understood as an initial set of questions, goals, challenges, and particular types of data collected in a specific context. In the example, this data context included continuous, multi-perspective audio/video data collected in a cultural heritage museum and a theoretical commitment to understanding visitors’ interest-driven engagement and learning as they moved across gallery spaces. Subsequently, integrative visualization extends visual methods of exploratory data analysis to iteratively transcribe data. Like any process of transcription, transcription of data amplified by visualization is theory laden and selective because it aims to organize data and develop codes, categories, questions, and units of analysis central to the data context (see Ochs, 1979; Hall, 2000). For instance, the example illustrated the use of the space-time cube to categorize conversation over space and time in ways that foregrounded mobility. Finally, integrative visualization supports the development of grounded theory and new computational tools. On one hand, the example characterized concepts and methods of interaction geography, units of analysis, and questions about visitors’ interest driven engagement and learning as potentially generalizable to other types of settings where people move to engage and learn (e.g., other museums, natural or urban environments). On the other hand, the example demonstrated how integrative visualization produced computational tools that support new forms of interaction and multimodal analysis. Importantly, developing these tools required (and requires) shifting from using traditional transcription tools (e.g., ranging from Microsoft Word to InqScribe to NVivo) to composing in more dynamic graphical layout tools (e.g., Adobe products) to developing visualizations and software in programming languages used in exploratory data analysis such as Processing and p5.js (see Fry, 2004). Though fully describing this shift is beyond the scope of this paper, it is critical to integrative visualization.

Integrative visualization may be particularly well suited to addressing some of the unique challenges of making sense of unstructured data (e.g., audio and video) about people’s interaction collected in and important to studying collaborative learning contexts. However, such work entails stronger connections between the CSCL and LS communities and the information visualization community. These connections are challenging to develop but essential to integrating technical skills of parsing, mining, representing, and interacting with data (see Stasko
et al., 2008; Fry, 2004) with non-technical ways of working with data that foreground theoretical and contextual dimensions (see Wise & Schwarz, 2017; Kahn, 2017; Zegura, DiSalvo & Meng, 2018; DiSalvo, 2016).

## References

Berland, M., Halverson, E., Polman, J. & Wilkerson, M. (2017). Expressive construction: Enabling learners to represent powerful ideas. In J. Roschelle, W. Martin, J. Ahn, & P. Schank (Eds.), *Cyberlearning Community Report: The State of Cyberlearning and the Future of Learning with Technology*.

DiSalvo. B. 2016. Participatory Design through a Learning Science Lens. In *Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems*. ACM, New York, NY, USA, 4459-4463.

Fouse, A., Weibel, N., Hutchins, E., and Hollan, J.D. (2011). ChronoViz: a system for supporting navigation of time-coded data. In *CHI '11 Extended Abstracts on Human Factors in Computing Systems* (CHI EA '11). ACM, New York, NY, USA, 299-304.

Fry, B. J. (2004). Computational information design. Ph.D. Dissertation. Cambridge: MIT.

Glaser, B.G. & Straus, A.L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL: Aldine.

Hagerstrand, T. (1970). What about people in regional science? *Papers in regional science*, 24(1), 7-24.

Hall, R. (2000). Video recording as theory. In D. Lesh & A. Kelley (Eds.) *Handbook of Research Design in Mathematics and Science Education* (pp. 647-664). Mahweh, NJ: Lawrence Erlbaum.

Kahn, J. B. (2017). At the intersection of self and society: Learning, storytelling, and modeling with big data. Ph.D. Dissertation. Vanderbilt University.

Knight, S., Wise, A., Ochoa, X. & Hershkovitz, A. (2017). Learning analytics: Looking to the future. *Journal of Learning Analytics, 4*(2), 1-5.

Law, N., Rose, C.P., Cress, U. and Ludvigsen, S. (2018). Different technologies, methodologies and epistemologies—is CSCL a community or communities? *International Journal of Computer-Supported Collaborative Learning*, 13(2), 131–136.

Marin, Ananda M. (2013). *Learning to Attend and Observe: Parent-child Meaning Making in the Natural World*. Ph.D. Dissertation. Northwestern University.

Ochs, E. (1979). Transcription as theory. In E. Ochs & B. Schieffelin (Eds.), *Developmental pragmatics* (pp. 43-72). New York: Academic Press.

Rosé, C., et al. (2008). Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. *International Journal of Computer-Supported Collaborative Learning*, 3(3), 237–271.

Schneider, B., & Pea, R. (2014). Toward collaboration sensing. *International Journal of Computer-Supported Collaborative Learning*, 9(4), 371–395.

Shapiro, B.R., Hall, R. & Owens, D. (2017). Developing & Using Interaction Geography in a Museum. *International Journal of Computer-Supported Collaborative Learning*, 12(4), 377-399.

Shapiro, B.R. and Hall, R. (2018). Personal Curation in a Museum. In *Proceedings of the ACM in Human-Computer Interaction*, Vol. 2, CSCW, Article 158 (November 2018). ACM, New York, NY.

Stasko, J., Görg, C., & Liu, Z. (2008). Jigsaw: Supporting Investigative Analysis through Interactive Visualization. *Information Visualization*, 7(2), 118–132.

Steier, R. (2014). Posing the question: Visitor posing as embodied interpretation in an art museum. *Mind, Culture, and Activity*, 21(2), 148-170.

Straus, A.L. (1987). Integrative diagrams and integrative sessions. In A.L. Strauss, (Ed.), *Qualitative Analysis for Social Scientists* (pp. 170–182). Cambridge University Press.

Taylor, K. H., & Hall, R. (2013). Counter-mapping the neighborhood on bicycles: Mobilizing youth to reimagine the city. *Technology, Knowledge and Learning*, 18(1–2), 65–93.

Tschumi, B. (1994). The Manhattan Transcripts. Wiley.

Tukey, John W (1977). *Exploratory Data Analysis*. Addison-Wesley.

Tufte, E. R. (1990). Envisioning Information. Chesire, Connecticut: Graphic Press.

Wise, A., & Schwarz, B. (2017). Visions of CSCL: Eight provocations for the future of the field. *International Journal of Computer-Supported Collaborative Learning*, 12, 423–467.

Zegura, E., C. DiSalvo, & A. Meng. (2018). Care and the Practice of Data Science for Social Good. In *Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies*. ACM, New York, NY.

## Acknowledgements

Amanda Meng, Jennifer Kahn, and Rogers Hall provided critically important feedback that contributed to the development of ideas in this paper. Figures copyright © Ben Rydal Shapiro. Reprinted with Permission.
# An Empirical Study of Educational Robotics as Tools for Group Metacognition and Collaborative Knowledge Construction

Chrysanthos Socratous, Cyprus Interaction Lab, Cyprus University of Technology,
chrysanthos@cyprusinteractionlab.com

Andri Ioannou, Cyprus Interaction Lab, Cyprus University of Technology, Research Center on Interactive Media, Smart Systems and Emerging Technologies (RISE), andri@cyprusinteractionlab.com

**Abstract:** The affordances of Educational Robotics (ER) for advancing teaching and learning has become a widely researched topic. This study aims to identify the major components of collaborative knowledge construction in an ER learning environment and to investigate the mediating role of ER as mindtools to support group metacognition. Data analysis involved a micro-level examination of students' discourse, interaction with the technology, peers and the facilitator, using fine-grained analysis of video and audio recordings. The results made evident that metacognition, along with questioning and answering, were prevalent elements of collaborative knowledge construction around ER. We support that ER can be used as a learning tool and can be effective in supporting group metacognition through immediate feedback, openly accessible programmability and students' embodied interaction with the physical robot. Beyond the instrumental role of ER for supporting metacognitive processes in CSCL settings, the study provides initial evidence for a temporal relation of metacognitive talk to collaborative talk in group problem-solving.

## Introduction

The use of Educational robotics (ER) in educational contexts to support teaching and learning has become an extensively researched topic. ER is constructible and programmable high-tech devices which can be employed in education as constructivist learning tools to support teaching and learning through hands-on activities. ER too early earned an influential role as a research field, motivating the attention of many schools, and universities, both from an instructive and a research point of view. Jonassen (2000) first introduced the theoretical background and the motivation for the integration of robotic technologies as cognitive tools which can improve and enrich the educational process. According to Gaudiello and Zibetti (2013), two features of ER are linked to their high educational potential; “transparency” and “interactivity.” “Transparency” refers to the openly accessible programmability of the robot whilst, “interactivity” refers to the immediacy of the feedback given by the robot when a student programs and executes the commands (Gaudiello & Zibetti, 2013).

Many studies have focused on exploring the affordances of ER in promoting several transversal skills such as problem-solving (e.g., Atmatzidou, Demetriadis & Nika, 2018), collaboration (e.g., Ardito, Mosley & Scollins, 2014), and computational thinking (e.g., Bers, Flannery, Kazakoff & Sullivan, 2014; Constantinou & Ioannou, 2018). Still, ER as metacognitive tools have been considered only recently (e.g., La Paglia, Caci, La Barbera & Cardaci, 2010; Gaudiello & Zibetti, 2013) and the research evidence is inconsistent. Further investigation in the area is needed to fully understand the potential of ER in supporting students' metacognitive processes and especially socially-mediated metacognitive processes in CSCL settings.

We present an empirical investigation of ER in a CSCL experience aiming at engaging students in collaboration and co-construction of shared understandings in the mathematics domain. In this work, the whole experience around using the robot is seen as a metacognitive experience that assists students to become more aware of their process of thinking and learning. We aim to unfold the elements of collaborative knowledge construction, identify details of the metacognitive processes during students' interaction with the robot and their peers, and document the educational potential of ER as tools for supporting group metacognition. Specifically, the research questions of this study are the following:

1. What are the elements of collaborative knowledge construction in an ER learning environment?

2. How does ER help to activate group metacognitive processes?

3. What is the relationship between collaborative talk and metacognitive talk in ER learning settings?

In the following lines, we present the theoretical framework of the study, findings from previous empirical studies, methodology, and findings from the present investigation along with discussion of the implications of this work.

## Theoretical framing
## Metacognition

Whilst, over the past years various theoretical models of metacognition have evolved, researchers agree that metacognition consists of three or at least two fundamental processes. According to Schraw and Moshman (1995), metacognition can be divided into a knowledge component (knowledge of cognition) and a skill component (regulation of cognition). Other researchers have expanded this model suggesting a three-tier model of metacognition namely, metacognitive knowledge, metacognitive judgments and monitoring, and self-regulation and control (Pintrich, Wolters, & Baxter, 2000). More recently Efklides (2011) proposed a complex interplay between the task level, the person level (where metacognitive knowledge and skills are located) and the interaction level (where metacognitive experiences take place).

There are already some studies examining the use of ER to promote metacognition at the individual level. For example, a study by Keren and Fridin (2014), examined how ER can assist the teaching of geometric thinking and promote children's metacognitive development. Findings from the study showed that students' performance on geometric thinking and metacognitive tasks were improved because of their participation in ER activities. Also, to investigate the process of constructing and programming robots as metacognitive tools, La Paglia et al. (2010) found that ER may be conceptualized as a novel metacognitive setting that motivates learners to monitor and control their own learning actions. Gaudiello and Zibetti (2013) tried to identify and classify the heuristics that are applied by elementary school students while they interact with and control ER technologies. The results demonstrated three main types of heuristics: (a) procedural-oriented, (b) declarative-oriented, and (c) metacognitive-oriented. Atmatzidou, Demetriadis, and Nika (2018) investigated the development of students' metacognitive skills in ER activities when the facilitators performed different levels of guidance (low and high) in different age groups. The results suggested that strong guidance had a positive impact on students' development of metacognitive thinking skills independently of their age and gender.

## Group metacognition

There is a recent shift in the literature towards the study of group metacognition rather than metacognition as an individual endeavor. Yet, research on metacognition in group (e.g., CSCL) situations is not well developed, despite group learning being commonplace in schools and other learning environments (Smith & Mancy, 2018). Despite the limited research on group metacognition, some findings suggest that metacognition is mediated and socially shared among group members in collaborative activities (Goos, Gailbraith & Renshaw, 2002) and that group metacognition can be considered as an extension of individual metacognition into group interactions. Also, researchers agree that metacognition in group situations consist of students' monitoring, reflecting and controlling of one-another's knowledge and actions. In CSCL research, the potential role of CSCL tools for supporting group metacognition has not been examined to date. As Järvelä and Hadwin (2013) explained, the potential role of CSCL tools for supporting the planning, monitoring, and regulation of collaborative learning processes has been virtually ignored. In this work, we see ER as CSCL tools that can promote students' metacognitive thinking. Most of ER activities are collaborative learning activities, yet, there are virtually no studies in the CSCL literature that examine the impact of ER on the development of group metacognition as an essential part of group work.

## Methods

### Participants

The participants were 14 students (6 male and 8 female) in Grades 4, 5 and 6 (aged 9-11 years old) in a public primary school in Cyprus. The students worked in 4 groups of 3-4 students each. Each group was formed with different genders and abilities (i.e., mathematical, technological and problem-solving abilities) to allow different discourses and problem-solving approaches to develop. The participating students had no previous experience in robotics.

### Procedures

There were two weeks of introductory activities to help students get familiar with the EV3 kit. These activities were followed by three 80-minutes sessions of STEM-related problem-solving tasks. Students should program a robot using a tablet, which was connected to the robot via Bluetooth. Each group was tasked with the following programming problems.

* Program a robot to move from its starting position, through a maze, to the finish position

* Program a robot to move along the outside of the flags without touching them

* Program a robot to draw a hexagon

Students in groups could adopt any approach they wanted to come to a solution. The teacher acted as a facilitator assisting the whole procedure e.g., assessing progress, examining understandings, monitoring group
work, and suggesting attention to data. When the groups completed their tasks, a debriefing phase took place. In this case, the groups demonstrated their approach in addressing the problem and answered questions asked by the facilitator and the students of other groups.

## Data collection and analysis

Verbal contributions were recorded via audio recorders next to each group. A camera was also placed in the room to record the overall student interaction and technology use.

To answer RQ1, the audio data were transcribed verbatim and analyzed using a fine-grained analysis. The unit of analysis was the individual participant and the discourse was coded on a turn-by-turn basis. A new turn was considered to start when the speaker changed. When the speaker shifted the theme of the discussion or when a different kind of discourse appeared, these were parsed into extra coded units. Generally, a conversational turn had more than one coding units. For instance, when a student asked a question but also added one or more statements, this was coded as two or more different coding units. Two independent raters coded 35% of the data to verify the reliability of coding. Reliability was acceptable (agreement 75%), and therefore, the first researcher completed coding the complete dataset. We used the coding scheme reported in Hmelo-Silver (2003), which conceptualizes the thinking processes and the general cognitive, metacognitive and social characteristics involved in collaborative knowledge construction.

To answer RQ2 and RQ3, a group was selected for further examination with a chronological investigation of within-group interaction. We used the CORDTRA technique initially presented by Hmelo-Silver, Jordan, Liu, and Chernobilsky (2011) and later applied in varied CSCL settings by Ioannou (2011), Ioannou, Brown and Artino (2015), and Socratous and Ioannou (2018). CORDTRA was examined in combination with excerpts of students' discourse to identify details of metacognitive and collaborative processes and the role of the technology.

## Findings

### What are the elements of collaborative knowledge construction? (RQ1)

#### Knowledge

As presented in Table 1, the students rarely referred to prior conceptual knowledge or experience of knowledge (5.2%). Not surprisingly, the students made comparisons and links referring to observations of previous actions in the same task.

#### Metacognition

The students used a larger amount of metacognitive utterances (24.1%). The majority were monitoring statements. Planning contributions occupied the second largest percentage of metacognitive utterances and were almost always in response to data derived from the results of previous trials. However, students did not mediate their planning with, prior knowledge, experience, or existing theories.

#### Interpretation

Students dedicated some effort in interpreting data derived from the robot or the tablet display (6.8%). Interpreting data was an opportunity to reconsider, test and refine their solutions.

#### Collaboration

Collaboration category included three subcategories: conflict, questioning and facilitator's input. Conflicts within groups were few and appeared mostly at early stages of the task. When conflicts appeared, they were more often related to the robot's failure to perform the expected outcomes; conflicts were rarely over a concept. Student groups generated many questions (22.7%), most of which referred to teammates rather than the facilitator. Most of these questions were planning-related questions as well as software- and robot-related questions. As shown by the relatively large number of statements related to agreement with peers (12.8%), students' consensus-seeking behavior was frequent. Responses by the students (24.3%) revealed the degree of consensus within the group. Students constructed simple explanations and brief answers more often than they elaborated explanations. Facilitator questioning was mainly concerned with software and robot use. The main operation of the facilitator's input was coded as monitoring (7.4%).

<table><caption>Table 1: Major categories and subcategories frequencies</caption><thead><tr><th>Coding categories</th><th>N (%)</th><th>Coding categories</th><th>N (%)</th></tr></thead><tbody><tr><td><strong>Knowledge</strong></td><td>29 (5.2%)</td><td><strong>Collaboration</strong></td><td>355 (63.9%)</td></tr><tr><td>Conceptual knowledge</td><td>5 (0.9%)</td><td>Conflict</td><td>26 (4.8%)</td></tr></tbody></table>
<table><tr><td>Prior experiences</td><td>4 (0.7%)</td><td>Conceptual</td><td>3 (0.6%)</td></tr><tr><td>Analogies</td><td>20 (3.6%)</td><td>Task-specific</td><td>23 (4.1%)</td></tr><tr><td><strong>Metacognition</strong></td><td><strong>134 (24.1%)</strong></td><td>Questioning</td><td>126 (22.7%)</td></tr><tr><td>Monitoring</td><td>74 (13.3%)</td><td>Clarifications</td><td>30 (5.4%)</td></tr><tr><td>Evaluation</td><td>4 (0.7%)</td><td>Plan-related</td><td>44 (7.9%)</td></tr><tr><td>Reflection</td><td>19 (3.4%)</td><td>Software-related</td><td>22 (4%)</td></tr><tr><td>Total planning</td><td>37 (6.7%)</td><td>Self-answered</td><td>5 (0.9%)</td></tr><tr><td>Theory-driven Planning</td><td>2 (0.4%)</td><td>General</td><td>2 (0.4%)</td></tr><tr><td>Data-driven Planning</td><td>33 (5.9%)</td><td>Facilitator</td><td>23 (4.1%)</td></tr><tr><td>Unjustified</td><td>2 (0.4%)</td><td>Responses</td><td>135 (24.3%)</td></tr><tr><td><strong>Interpretation</strong></td><td><strong>38 (6.8%)</strong></td><td>Agreement with facilitator</td><td>20 (3.6%)</td></tr><tr><td>High-level</td><td>7 (1.3%)</td><td>Agreement with peer</td><td>71 (12.8%)</td></tr><tr><td>Low-level</td><td>31 (5.5%)</td><td>Brief answers</td><td>24 (4.3%)</td></tr><tr><td></td><td></td><td>Simple explanations</td><td>16 (2.9%)</td></tr><tr><td></td><td></td><td>Elaborate explanations</td><td>4 (0.7%)</td></tr><tr><td></td><td></td><td>Facilitator's input</td><td>68 (12.1%)</td></tr><tr><td></td><td></td><td>Monitoring</td><td>41 (7.4%)</td></tr><tr><td></td><td></td><td>Explaining concepts</td><td>3 (0.4%)</td></tr><tr><td></td><td></td><td>Explaining Software</td><td>24 (4.3%)</td></tr></table>

## How does ER help to activate group metacognitive processes? (RQ2)

To answer RQ2, we examined an integrated view of an episode, using the CORDTRA diagram of Fig. 1. On CORDTRA diagram the numbers on the x-axis represent the chronological order of the coded units, whilst the y-axis represents the coded categories (records 6 to 29) and the speakers (records 1 to 5). The diagram reveals the nature of student's talk, including metacognitive talk, and its temporal relation to the use of the robot when students tried to solve the "Draw a hexagon" challenge. Combining the diagram with discourse excerpts helped to understand student's interactions across time. We zoomed into an episode in which the students work on solving the "draw a hexagon" challenge (lines 90-190).

### ER activating group metacognitive processes through embodied interaction

For the activity, students should combine mathematical knowledge, experience with ER (i.e., introductory lessons), and programming skills to solve the problem. First, students started to discuss how they could solve the problem without having many ideas. A student stated that they should use the gyro sensor while another student added that they should place a pen holder on the robot. A detailed discussion about where they could set the pen holder took place in lines 91-99. Here, questioning discourse appeared as an essential aspect of collaborative knowledge construction. The students' questioning about where to put the gyro sensor and the kinds of turns the robot should make, triggered the dialogue for the next steps. The students started to research the question using the robot as a mean for experimentation by adjusting the pen holder in different places on the robot. Students seemed to recognize the significance of where they should adjust the pen holder; this important discussion moved students' thinking forward. The overall experimentation involved their bodies as students held the robot in their hands and were trying to simulate (with their bodies) possible movements of the robot and thinking of possible pen footprints on the paper. Students involved their bodies in understanding the difference between swing and point turn (lines 100-114). Students' embodied interaction with the physical robot triggered further social interaction and stimulated group metacognitive processes. The students tested and modified their new ideas, against existing knowledge and new data. Thus, it appears that ER, through embodied interaction, served as a tool for experimentation, activating group metacognitive processes and collaborative knowledge construction.

* Student 1 We must draw a hexagon (*laughing*). Any ideas?

* Student 3 We must use the gyro sensor to turn exactly as degrees as we program it (*for accurate angle measure*).

* Student 2 Yes, the robot must turn exactly as degrees as we program it.

* Student 4 We also need to adjust a marker to draw the hexagon as the robot moves and turns.

* Student 2 What kind of turns?

* Student 3 Turns.

* Student 1 Pivot turns. The robot must turn very sharp and make pivot turns to draw an angle.
Student 1 Ok! Where can we apply the marker?

Student 2 If we put it here (holds marker and robot and try to make turns to draw a random angle).

These tests continued until students manage to draw angles formed by two rays rather than curved lines. Students placed the marker on different parts of the robot and tried to draw random angles. They tried to put the marker in different places (between the wheels, next to the right wheel and on the back of the robot) to understand where it would be more efficient to place the marker.

## ER activating students' group metacognitive processes through interactivity and transparency

In lines 160-176 of the CORDTRA diagram, students went through an exploration in which they used their conceptual knowledge of mathematics and programming in a real-world situation. Students were concerned about how many degrees their robot should turn and, with the teacher's assistance, they managed to connect their mathematical knowledge and programming skills with a real-world condition. A student influenced by the introductory robotics lessons used a flowchart describing the required moves of the robot to draw a hexagon (line 168). Then, they decided to program the robot to turn 120°, as much as the internal angle and observed their robot turning much more than they expected. Immediate feedback from the robot's moves (i.e., observing the robot turn more than they expected) made the students think and monitor their thoughts (line 169). Robot's failure to produce the expected outcome seems to have triggered the group's metacognitive thinking. Thinking of what they were doing wrong, checking various aspects (lines 169-172) and building on each-other's thoughts, they excluded various possibilities and proposed a solution to the problem. After that, student 4 contributes a more advanced thinking to the discussion, suggesting that they should put a smaller value for the turning angle because with 120° the robot was turning too much. Student 4 proposed to represent the problem on a paper to calculate the turning angle. Students acknowledged this idea and began to model the problem on a paper. By representing the problem on paper students managed to find the correct value for the turning angle. Then, Student 1 made his thinking visible showing on the paper the correct angle (line 174). Student 2 built on the previous thought proposing the solution to the problem (line 175). Therefore, the process of socially-shared metacognition emerged in this group when Student 4 provided a metacognitive regulation statement (i.e., "If the robot turns 120° left, it will get into the hexagon. Let's draw the hexagon on a paper to find the angle"). The other group members acknowledged this contribution and developed a solution to the problem.

The transparency features of ER helped the students think, apply and check their ideas to overcome the problem. Easy changes to the software and hardware, at no cost, helped the students to avoid frustration, and through the open and accessible programmability of the robot, they managed to overcome the obstacles. The robot's programming, the expected results, and the actual results of its actions served as a metacognitive tool and as a data reference that students could use to negotiate their developing solution. The students identified gaps in their knowledge and collectively discussed, elaborated, and improved their solution. Regulatory statements that were produced due to the interactivity and transparency features of the robot promoted group metacognition and facilitated collaborative knowledge construction.

Student 3 Now, we will program the robot to move forward, then make a turn for some degrees then again forward and then turn, etc.

Student 2 Ok, we have to think about how many turns and how many degrees.

Student 1 Six turns and six forward. I do not know how many degrees.

Teacher What do we know about the total internal angles of polygons?

Student 4 It depends on how many different triangles are formed into the hexagon that does not overlap each other.

Student 2 How many different triangles does a hexagon have?

Student 3 I will draw a hexagon to find how many triangles are formed.

Student 3 4 different triangles. So, multiplies by 180° each equal 720°

Student 1 Divide by 6 angles of a hexagon (thinking). Equals 120°. So, we will program the robot to move forward and then turn 120° for 6 times.

(The team programmed the robot and is going to test the program).

Student 2 No, it is turning too much. Perhaps we calculate the angles wrongly. Let's check it.

Student 3 Or, the sensor is not working

Student 2 Gyro sensor looks ok!
Student 1 (*They are doing the calculations*) The angle is correct 120°. Must be something else.

Student 4 Yeah, but I think we just have to take a smaller angle. 120° are all the internal angles of the hexagon. The robot moves on one of the sides of the hexagon. If the robot turn 120° left, it will get into the hexagon. Let's draw the hexagon on a paper to find the angle.

*(They draw a hexagon with a robot, representing it with a dot, on one of its angles)*

Student 1 The robot is this dot and must turn here (showing with his finger). So the turning angle is this one, we must find this one (showing on the paper).

Student 2 This angle is supplementary of the internal angle. So its 180-120 = 60.

Student 4 Yes, that is. The robot must turn as much as the supplementary of the internal angle, only 60° not 120°.

<figure><img src="image_94.png" /><figcaption>Figure 1. CORDTRA diagram of students' contributions.</figcaption></figure>

## What is the relationship between collaborative talk and metacognitive talk? (RQ3)

Both in the previous excerpt and the one below, the students made their metacognitive thinking visible mainly in mutual interaction with their teammates. Student 1 tried to explain their failure to solve the challenge proposing that the flags were small, so the sensor could not detect them. This contribution triggered the thinking of Student 3, leading him to suggest a new idea that is, the use of two ultrasonic sensors instead of one. Student 1 pointed out his disagreement over the proposed idea and documented his position using the experience of a previous failing effort outside the current activity. Then, Student 1 contributed a metacognitive statement to justify his position proposing that they do not know well how to handle an ultrasonic sensor and so, he proposed a trial and error plan. Student 3 ignored Student's 1 plan highlighting that with two sensors, it would be easier for the robot to detect the flags. When they failed, Student 3 accepted to use the alternative plan but he first proposed to measure the distances among the flags so that they did not use a trial and error plan. In the excerpt below, the students compared their thinking with the thinking of their peers and this involved the use of collaborative talk in parallel with metacognitive talk. Also, as shown in the CORTDRA (Fig. 1), collaborative and metacognitive talk seemed to have a temporal relationship between them. For example, contributions that were coded as collaborative talk were usually followed by one or more metacognitive contributions vice versa.

Student 2 We will use the ultrasonic sensor to avoid the flags.

Student 3 Ok then. Put the ultrasonic sensor. (*They executed their plan, but they failed*).

Student 1 The flags are small, so the sensor cannot detect them.

Student 3 We can use two ultrasonic sensors. What do you think?

Student 1 No, we tried to use the ultrasonic sensor once, and we failed. We do not know how to handle it. Let's program the robot to move, and then we can adjust the values.

Student 3 If we put two sensors, it will be easier for the robot to detect the flags.
Student 2 Ok! Let's try with two sensors.

Student 1 Ok then. (They executed their plan using two sensors, but they failed).

Student 1 I told you, we do not need the sensors.

Student 3 One more trial with two sensors and then, if we fail, we can move with your plan. (They changed the position of the two sensors and tried again, but they failed).

Student 3 Ok. Let's do what you said, but first, we can measure the distance between the flags to calculate the value of rotations.

## Discussion

The study presents evidence that CSCL activities using ER can engage students in collaborative knowledge construction with prevalent elements of metacognitive processes, questioning, and answering. Indeed, students' discourse demonstrated logical reasoning coupled with metacognitive statements enabling the students to predict and to plan the flow of actions required to solve the problem. Monitoring elements of metacognition seem to be activated in an ER learning environment, engaging students in the process of exploration for the acquisition of knowledge. The large volume of monitoring elements of metacognition can be explained as the ER's value in encouraging procedural knowledge rather than declarative knowledge i.e., student learning by doing and understanding strategies of problem-solving rather than concepts.

During the ER activity, intensive collaboration was enacted in the form of questioning and answering while metacognition was enacted in the form of monitoring and planning. Many researchers have identified questioning (e.g., Hmelo-Silver & Barrows, 2008) and reflective thinking (e.g., Baker & Lund, 1997) as important kinds of discourse in knowledge building situations. Contributions of prior knowledge were limited, although this might not be replicated in a setting where learners have prior experiences with ER. Our findings confirm previous evidence about ER promoting collaborative knowledge construction (Chambers, Carbonaro, Rex & Grove, 2007; Socratous & Ioannou, 2018). This work contributes further in that it presents a fine-grained analysis of the phenomenon to strengthen the scientific evidence in the area. While previous studies rely heavily on the study of metacognition as an individual endeavor, using self-reported data (e.g., Atmatzidou et al., 2018), this study documents metacognition as a result of group work, while it occurred in-situ.

Metacognitive elements, coded as monitoring, evaluation, reflection, and planning, are activated in ER activities through embodied interaction with the physical robot. Indeed, when a robot is being used in the activity, it enables students' physical action and simulation of the robot's expected actions. Such activities seem to encourage expression and personal involvement in the learning process, whilst supporting teamwork which is important for the metacognitive process. Moreover, the transparent software design and the direct interactivity (feedback) coming from the robot's moves in response to students' programming, seem to facilitate the group's metacognitive thinking. In fact, when the robot failed to perform the expected outcomes, monitoring and planning elements of metacognition were documented on our chronological diagrams. Metacognition was necessary for students to understand how the tasks were performed and to be able to identify problems, negotiate modifications and operating changes to solve the problems. Embodied interaction with the physical robot, combined with feedback coming from the robot, acted as an extension of students' mind, scaffolding knowledge construction by re-evaluating their solutions. From this perspective, ER can be considered as "scaffolding embedded technological tools" (Chambers et al., 2007).

Our research has provided some initial evidence for a temporal relationship between collaborative talk and metacognitive talk in a problem-solving ER environment. The study further presents an instrumental role of ER technology in supporting metacognitive processes in CSCL settings. Metacognitive and collaborative talk appear to mediate each-other in this CSCL, ER setting. We understand that this evidence is not clear yet. Further development of our understanding of ER as metacognitive tools, will help us develop strategies to fully maximize their effectiveness in group problem-solving CSCL tasks.

## Conclusions

Coding and plotting student's discourse around an ER experience in CSCL settings can shed light on the value to the technology for collaborative knowledge construction and group metacognition. In this work, fine-grained analysis of student's discourse made evident that metacognition, along with questioning and answering, are prevalent elements of collaborative knowledge construction in ER activities. What is more, the role of the technology seems to be instrumental; namely, the embodied interaction, direct feedback and openly accessible programmability enabled by the robot were tightly coupled with group metacognitive processing and overall collaborative knowledge construction. In conclusion, this work extends the evidence on the value of ER integration in learning environments and CSCL activities. The study contributes in that it presents a fine-grained
analysis of the phenomenon to strengthen the scientific evidence in the area. While previous studies rely heavily on the study of metacognition as an individual endeavor using self-reported data (e.g., Atmatzidou et al., 2018), this study documents metacognition as a result of group work, while it occurred in-situ. Future work should extend on the nature of the problem, the teacher's scaffolding, the students' roles and the characteristics of the technology which might further endorse collaborative knowledge construction and metacognition in CSCL settings.

## References

Ardito, G., Mosley, P., & Scollins, L. (2014). We, robot: Using robotics to promote collaborative and mathematics learning in a middle school classroom. *Middle Grades Research Journal, 9*(3), 73.

Atmatzidou, S., Demetriadis, S., & Nika, P. (2018). How Does the Degree of Guidance Support Students' Metacognitive and Problem Solving Skills in Educational Robotics? *Journal of Science Education and Technology, 27*(1), 70-85.

Baker, M., & Lund, K. (1997). Promoting reflective interactions in a CSCL environment. *Journal of computer assisted learning, 13*(3), 175-193.

Bers, M. U., Flannery, L., Kazakoff, E. R., & Sullivan, A. (2014). Computational thinking and tinkering: Exploration of an early childhood robotics curriculum. *Computers & Education, 72*, 145-157.

Chambers, J. M., Carbonaro, M., Rex, M., & Grove, S. (2007). Scaffolding knowledge construction through robotic technology: A middle school case study. *Electronic Jrl for the Integration of Technology in Education, 6*, 55-70.

Constantinou, V., & Ioannou A. (2018). Development of Computational Thinking Skills through Educational Robotics. In European Conference on Technology Enhanced Learning - ECTEL. Retrieved December 2018 from http://ceur-ws.org/Vol-2193/paper9.pdf

Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. *Educational psychologist, 46*(1), 6-25.

Gaudiello, I., & Zibetti, E. (2013). Using control heuristics as a means to explore the educational potential of robotics kits. *Themes in Science and Technology Education, 6*(1), 15-28.

Goos, M., Galbraith, P., & Renshaw, P. (2002). Socially mediated metacognition: Creating collaborative zones of proximal development in small group problem solving. *Educational studies in Mathematics, 49*(2), 193-223.

Harel, I., & Papert, S. (Eds.). (1991). *Constructionism*. Westport, CT, US: Ablex Publishing.

Hmelo-Silver, C. E. (2003). Analyzing collaborative knowledge construction: Multiple methods for integrated understanding. *Computers & Education, 41*(4), 397-420.

Hmelo-Silver, C. E., & Barrows, H. S. (2008). Facilitating collaborative knowledge building. *Cognition and instruction, 26*(1), 48-94.

Hmelo-Silver, C. E., Jordan, R., Liu, L., & Chernobilsky, E. (2011). Representational tools for understanding complex computer-supported collaborative learning environments. In *Analyzing interactions in CSCL* (pp. 83-106). Springer US.

Ioannou, A. (2011). Online collaborative learning: The promise of wikis. *International Journal of Instructional Media, 38*(3), 213-223.

Ioannou, A., Brown, S.W., & Artino, A.R. (2015). Wikis and forums for collaborative problem-based activity: A systematic comparison of learners' interactions. *The Internet and Higher Education, 24*, 35-45.

Järvelä, S., & Hadwin, A. F. (2013). New frontiers: Regulating learning in CSCL. *Educational Psychologist, 48*(1), 25-39.

Jonassen, D. H. (2000). *Computers as mindtools for schools: Engaging critical thinking*. Prentice Hall.

Keren, G., & Fridin, M. (2014). Kindergarten Social Assistive Robot (KindSAR) for children's geometric thinking and metacognitive development in preschool education: A pilot study. *Computers in Human Behavior, 35*, 400-412.

La Paglia, F., Caci, B., La Barbera, D., & Cardaci, M. (2010). Using robotics construction kits as metacognitive tools: A research in an Italian primary school. *Studies in Health Technology and Informatics, 154*, 110-114.

Pintrich, P. R., Wolters, C. A., & Baxter, G. P. (2000). 2. assessing metacognition and self-regulated learning.

Schraw, G., & Moshman, D. (1995). Metacognitive theories. *Educational psychology review, 7*(4), 351-371.

Smith, J. M., & Mancy, R. (2018). Exploring the relationship between metacognitive and collaborative talk during group mathematical problem-solving-what do we mean by collaborative metacognition?. *Research in Mathematics Education, 20*(1), 14-36.

Socratous, C., & Ioannou, A. (2018). A Study of Collaborative Knowledge Construction in STEM via Educational Robotics. In *J. Kay & R. Luckin (Eds.), Rethinking Learning in the Digital Age: Making the Learning Sciences Count, 13th International Conference of the Learning Sciences (ICLS) 2018* (Vol. 1, pp. 496-503). London, UK: ISLS.

## Acknowledgments

This work is part of a project that has received funding from the European Horizon 2020 research and innovation programme under grant agreement N°739578 and the government of the Republic of Cyprus through the Directorate General for European Programmes, Coordination and Development.
# Disciplinary Task Models for Designing Classroom Orchestration:
The Case of Data Visualization for Historical Inquiry

Josh Radinsky, University of Illinois Chicago, joshuar@uic.edu
Iris Tabak, Ben-Gurion University of the Negev, itabak@bgu.ac.il
Monika Moore, University of Illinois Chicago, mmoore49@uic.edu

**Abstract:** Incorporating data visualizations into college classrooms can enhance disciplinary learning, when accompanied by associated meaning-making practices. However, tools not specifically designed for CSCL may lack supports for collaboration and disciplinary reasoning. Therefore, it is necessary to design supports into classroom orchestration. In this paper we examine the design of such orchestration to scaffold historical reasoning with geospatial data visualizations. The orchestration of a sequence of small-group tasks scaffolded progressively more complex reasoning about historical waves of immigration, which the instructor leveraged to support students’ reasoning at multiple levels in whole-class discussion. Our analyses point to these kinds of orchestrated scaffolds as a valuable direction for CSCL research and design.

In proposing productive research directions for CSCL (Ludvigsen, Cress, Law, Rosé, & Stahl, 2016; Wise & Schwarz, 2017), a number of scholars converge on broadening the purview to tools beyond the products of CSCL design research. Specifically, Reimann (in Ludvigsen et al., 2016), foresees greater attention to sophisticated representational systems used for everyday and professional goals. Ludvigsen (2016) further calls for research to identify practices for arriving at shared meanings through interactions with these representational systems, and studies that highlight when and why collaboration might be useful (Wise & Schwarz, 2017).

In this paper we explore ways to incorporate such systems and their accompanying knowledge-building practices into the post-secondary classroom. College classrooms are environments in which learners can be introduced to disciplinary inquiry for the first time. This includes being exposed to a discipline's *professional vision* (Goodwin, 1994) – being apprenticed by an instructor to see information (texts, data) through a particular lens that enables discipline-relevant insights and reasoning. This is a process that often proves challenging for college instructors (PCAST, 2012). Merely using representational tools in instruction does not guarantee that learners will see data through the eyes of the profession, especially when the tools lack support for collaboration and disciplinary meaning making (e.g., Fischer et al., 2013; Raes, Schellens, De Wever, & Vanderhoven, 2012). Incorporating these tools poses a number of design challenges: how to create coherence between existing curricula and the forms of thought introduced by the tools; how to help learners see and interpret representations through disciplinary eyes; and how to manage the complexity inherent to these tools. Meeting these challenges requires design work *around* these tools themselves.

We propose that integrating disciplinary models with *classroom orchestration* is a productive way to incorporate existing representational systems into instructional settings. We examine this in a college history education class using a data visualization tool. The study addresses the questions: How can data visualization tools be incorporated into the college classroom to support historical inquiry? How can classroom orchestration facilitate learners’ ability to engage in geo-spatial reasoning for historical inquiry?

## Theoretical framework

### Classroom orchestration and distributed scaffolding

Research in CSCL and the Learning Sciences often argues that the meanings, purposes, and learning value of tools are mediated by the activity structures in which those tools are used. *Classroom orchestration* (Dillenbourg, Prieto, & Olsen, 2018) is a process by which teachers or other instructional designers plan, deploy and regulate multiple resources in the classroom to facilitate learning and achieve a productive workflow (Prieto, Holenko Dlab, Gutiérrez, Abdulwahed, & Balid, 2011). A central role for teachers in classroom orchestration is to assess progress and achievement, and make changes and adaptations on the fly to better align interactions with learning goals (Kaendler, Wiedmann, Rummel, & Spada, 2015). This alignment of multiple resources and interactions to support learning has been described as *distributed scaffolding* (Puntambekar & Kolodner, 2005; Tabak, 2004).

Classroom orchestration and distributed scaffolding can support learners in specific reasoning processes, such as learning to interpret data through a particular disciplinary lens, or *professional vision* (Goodwin, 1994). Members of a professional community are likely to converge on similar interpretations, while members of different communities may focus on different aspects and arrive at different meanings, even when observing the
same representation (Bowen, Roth, & McGinn, 1999). Learners may be able to approach novel representational
systems and derive meaning from them, but they may struggle to produce the types of explanations that typify a
particular field or discipline. Therefore, a pivotal role for distributed scaffolding and orchestration is in helping
learners to focus on particular attributes, and to interpret representations in particular ways, such as by teachers
interpreting representations in disciplinary terms alongside learners (Tabak, 2004; Tabak & Reiser, 2008).

Moderating discussion through levels of visualization

Classroom orchestration can also involve regulating classroom discourse. Discussions around modeling or visualization tools can occur at three levels (Radinsky, Milz, Zellner, Pudlock, Witek, Hoch & Lyons, 2017). At the *interface level*, discussion centers on how to manipulate the tool and extract information and visualizations. At the *modeling level*, discussion centers on the signs, symbols, images and quantities that are represented in the tool and their relationship to the phenomena being represented. Discussions at this level can be about relationships among variables and the interpretation of patterns. At the *represented-world level*, discussion centers on the phenomena in the world that are modeled through the visualization. Discussions at the represented-world level arise from the representations, but speak in the language of the modeled phenomena rather than the images, symbols or quantities that are displayed. The goal is to push discussion beyond the interface level, to concentrate on the modeling level to ensure that new understandings are constructed, and then to the represented-world level so that the knowledge that is constructed is about the phenomena of interest. Although the instructional aim is to move from the language of representation to the language of historical phenomena, it is critical that classroom guidance regulate the connection between discussion at both levels, so that the reasoning stays anchored in data.

Data visualizations and historical reasoning

Historical reasoning involves the construction of narratives of the past that are grounded in observations from historical records (Seixas, 2017). This requires skills such as chronological reasoning, contextualizing historical data, corroborating observations across multiple sources of information, and reasoning about texts as sources (Seixas, 2017). Using a data visualization tool for historical inquiry involves more than extracting and interpreting information; it requires viewing the data as historical records of people whose experiences can be only partially and imperfectly inferred from observations of these data. Geographic information systems (GIS) are a class of visualization tools that has been part of a fundamental spatial shift in many of the social and natural sciences, but has held a more problematic place in the discipline of history. Though some historians find an ontological or epistemological mismatch between the quantified and spatialized representations of GIS and the tentative and contingent nature of historians’ professional vision (Owens, 2007), there is a small but growing group of historians who embrace GIS tools because of their affordances for observing phenomena at spatial and temporal scales beyond what many traditional historical sources allow (Knowles, 2008). The present study explores a design for apprenticing students in historical reasoning that takes advantage of commonly-available GIS tools.

Using disciplinary task models to design orchestration with a “webmap”

The visualization tool used in this study, *Immigration Explorer*, is what Baker (2005) describes as “webmaps”: online, public-use GIS providing access to geospatial historical data with minimal functionality for manipulating the representation, instead emphasizing ease of access and simplicity of use. This was a free resource available on a public-access URL created in 2009 by the *New York Times* for browsing historical census data for the years 1880-2000. The webmap displayed the total population of each county in each census decade; the number of people who were identified as foreign-born; and the number of foreign-born people from each of 23 countries-of-origin. The data were visualized as spheres of varying size, representing the number of county residents born in each country-of-origin. Figure 1 shows an example map of the census count of people born in Mexico living in each U. S. county in 1970, 1980 and 1990.

<figure><img src="image_95.png" /><figcaption>Figure 1. *Immigration Explorer* maps showing Mexican-born population by U. S. county, 1970-1990.</figcaption></figure>
# The design: Orchestration of sequential collaborative data interpretation activities

The simplicity and accessibility of this tool allowed for ease of integration into the classroom but required any distributed scaffolding for historical reasoning to be designed into the orchestration of the classroom activity, since no such supports were built into the tool. The design of these “wrap-around” classroom supports, to foster reasoning that is both historical and geo-spatial, is the focus of the present analysis.

Learners required considerable support in order to engage in meaningful historical inquiry while developing nascent models of multiple population patterns shifting over time. Therefore, the tacit conceptual moves involved in this reasoning process were mapped onto group activity structures to design classroom orchestration. This provided learners with a series of manageable tasks that combined to form richer, more complex interpretations of historical data. The final products of the group work could subsequently be used by the instructor in a whole class discussion to piece together, through a series of prompts, queries, and restatements, the initial building blocks of an historical account. This laid the conceptual groundwork for their subsequent interpretation of a series of historical case studies of migration (beyond the scope of the present analysis).

Five phases of activity with different group configurations guided students toward using the webmap to construct historical narratives about patterns of migration from different countries-of-origin (hereafter, CoO):

1. **Whole-class introduction of the mapping tool:** whole-class discussion showing how to use *Immigration Explorer* (interface level), introducing historical census data as a resource for historical reasoning (modeling and represented-world levels).

2. **Small-group investigation of one CoO in one region:** in groups of three, students select one CoO group and make observations of that group's changing population pattern in one region over time. For example, they might examine the changing Chinese-born population in northern California from 1880 to 1930.

3. **Same-group comparing changing patterns for the same CoO in different regions:** in the same groups of three, students compare the patterns of population change of the same CoO in one region with another region, e.g., comparing Chinese-born population patterns in northern California to the very different patterns of Chinese immigration in the Midwest, around Chicago.

4. **New "jigsaw" groups compare "waves of migration" for different CoO:** students form new groups of three that each include an "expert" on a different CoO population. In these "jigsaw" groups (Johnson & Johnson, 1982), students take turns teaching the patterns they observed for their population, and then consider similarities and differences across these CoO's patterns of migration.

5. **Whole-class sharing and discussion of group findings:** groups present their findings and with the instructor's help articulate connections and interpretations towards constructing historical narratives. Instructor scaffolds bridge from modeling level to represented-world reasoning.

At each of the small-group phases (see Figure 2) there are challenges for reasoning at all three levels (interface, modeling, and represented-world). Phase 2 allows a group to focus on only one region and one CoO as they practice with the tools, attempting to describe change over time. Phase 3 adds the challenge of doing this for two regions of the USA, but with the same CoO map. Phase 4 requires the use of three different CoO maps, and pushes them to reason about different spatio-temporal patterns for different immigrant groups, potentially over centuries.

Whole-class discussion led by the instructor (Phase 1) provides modeling of the language of careful and specific observations (e.g., "the population of people born in Mexico grew quickly in this area from 1970 to 1990"), prior to students practicing these kinds of observations in small groups (Phase 2), and prior to repeating this same skill while attending to multiple regions of the map (Phase 3). The decade slider provides support for noticing population changes without having to change maps, and both regions they are comparing are visible in the same map. Students then take responsibility for sharing those observations with a new group, and making similar comparisons across different maps displaying different CoO's (Phase 4). Finally, the instructor-mediated whole-class discussion (Phase 5) allows for a more formal sharing of the cross-group observations and comparisons, allowing communal scrutiny of their findings. This affords scaffolding, modeling, and shaping of students' language by the instructor, referring back to the worked example from Phase 1.

Though fairly straightforward as a "jigsaw" lesson, the rationale for this sequence embodies a set of propositions about learning that have value for CSCL researchers. As Vogel, Wecker, Kollar and Fischer (2017) note, "CSCL scripts are particularly effective for domain-specific learning when they prompt transactive activities (i.e., activities in which a learner's reasoning builds on the contribution of a learning partner) and when they are combined with additional content-specific scaffolding (worked examples, concept maps, etc.)" (p. 477). The phases of activity reflect the conceptual steps that a historian might take in constructing an inquiry comparing waves of migration for different populations. For a single learner, conducting these steps would be taxing. By reducing the complexity of the task within the original group (comparing patterns within a single map display)
and then pooling resources by forming new comparison groupings, students are able to consider multiple, multi-level comparisons, without having to produce such complex comparisons from scratch. The whole-class, teacher-mediated sharing affords joint attention to reasoning across multiple regions, populations, and decades, modeling data not simply as numbers, but as representations of historical movements of people through space and time.

<figure><img src="image_96.png" /><figcaption>**Figure 2.** Orchestration “jigsaw” design constructs multiple, multi-level comparisons of historical migrations.</figcaption></figure>

# Methods

## Participants and setting

Participants were 15 Masters students (12 female, 3 male) in a teaching licensure program. The lesson was conducted in a course taught by the first author, as an action research project, documented by a research assistant, and following IRB-approved procedures for consenting, data gathering and analysis. The course was a pre-service methods class for history and social studies teachers, and the focal lesson was part of a module on historical inquiry and migration. The larger instructional unit built conceptual understandings of migration, including specific cases of migration drawn from students’ families and a variety of texts, leading to an analysis of case studies from a historical text (not described here). The focus of this lesson was on different geographic and temporal patterns of immigration to the United States.

## Data collection and analysis

We report on a subset of the data collected for the larger research project: a video recording of the focal lesson (65 minutes). The classroom video was transcribed for speech and gesture by one research assistant, second-passed by another. All student names are pseudonyms. In addition, a pre-instruction reflection interview with the instructor (first author) was documented, articulating the logic of the sequence of group activities in the lesson. This was used to articulate the design rationale and intended trajectory of classroom talk.

The transcript of the final presentations and discussions was coded for descriptive and comparative discursive moves at the *interface, modeling, and represented-world* levels, following conventions described in (Radinsky et al, 2017), and a grounded-theory open-coding approach was used to identify the range of historical and geospatial descriptions, comparisons and explanations that emerged. This produced a set of 35 distinct codes, which are being developed into a coding scheme for ongoing analysis of project data. For the analysis presented here, qualitative descriptions of the unfolding spatial and historical observations, comparisons, and explanations were used to examine the ways the orchestration design supported instructional opportunities to scaffold disciplinary learning with the *Immigration Explorer* webmap.

## Findings

Due to space limitation, we present only details from the discussions that occurred in the whole-class segment following the jigsaw activity (Phase 5). We report on interactive presentations from two jigsaw groups: Lisa, Penny and Maritza, who studied immigrations from Vietnam, Sweden and Mexico, respectively; and Tina, Nancy and Cory, who studied immigrations from Sweden, Mexico and India, respectively.
Lisa's group: Comparing quantities and rates (*modeling* level)

Lisa, Penny and Maritza presented each CoO group's pattern in order, starting with Vietnam. Lisa described how "we noticed the biggest county in 1980 was Orange County ... we looked at Orange County and Chicago for comparisons." Using these two counties in different regions of the country to construct their comparison, they noticed a pattern of the Vietnamese-born population doubling: "So from 1980 to 1990 it doubled in Orange County [California] and Cook County [Illinois, where Chicago is located]. And then from 1990 to 2000 it doubled also. Umm that was pretty much all from Vietnam."

When they moved on to present the data for Sweden, Penny followed Lisa's lead at the modeling level in presenting data for a single county (Douglas County, Minnesota), and then constructing a rate of change for that population in that county: "we went back to 1880 and we found um, a population of over 2,000 people ... And every 20, every 20 years ... every 20 years it decreased." Penny moves from naming quantities to giving a qualitative description of the changing map representation as she changes the time: "1940, we only have a little over 800, um, people in Douglas County and 20 years after by 1960 its in the 300s. And the bubble just continues to disappear and I don't find it, by the time we get to 2000, I don't find it. I don't find it anymore." Lisa then connected the patterns for the two groups: "So as you're – as the Swedish population in Minnesota was decreasing by half every 20 years the Vietnamese population was doubling every 10 years. So that was a cool comparison we thought of in our group."

Maritza's presentation of the pattern for people born in Mexico picks up on a different part of Lisa's initial description: the absence of data for some decades. She narrates their inquiry process as a series of discoveries of the limitations of the data, then shifts to add their observations about Hawaii: "we looked it up in Wikipedia that Hawaii didn't become a state until 1959. So that explains why there's no data here."

Three things are notable here: (1) their construction of the task of *making observations* as reporting single-county data values (rather than qualitative descriptions of regional patterns); (2) their use of data values to construct and compare rates of population change over time (doubling each decade, halving every 20 years); and (3) the emphasis on incomplete data. The instructor engages the exploration of some of these opportunities during the presentation, grounding the conversation in the *modeling level* (comparing quantitative patterns) that the group has highlighted. For example, when Maritza says "Once again, it [the data] disappears from Texas," the instructor adds: "And from the whole South. You see that? The whole South disappears," gesturing to the map to highlight the missing data Maritza has mentioned. This move takes up Maritza's language ("disappears"), adds to the observation in a way that better reflects the scale of the pattern ("the whole South"), and invites intersubjective attention to the data ("You see that?", gesturing).

A more active engagement occurs when Lisa presents the group's "cool comparison" of rates of change (doubling versus halving). The instructor endorses their positive self-evaluation ("Very cool"), and then asks the rest of the class to re-represent that comparison of rates-of-change:

Instr: Very cool. Imagine of what ways we can visualize that, other ways we can show that change. Can anybody show that with your hands? What Lisa just said? Like if you were going to describe that change what would it look like?

Penny: [points up with one hand and down with the other]

Instr: So Penny went like this. [mimics Penny's hand motion, eliciting general laughter] What does this mean? [repeats gesture]

Edita: Increase, decrease.

Instr: Edita what are you going to show us again?

Edita: [two flat hands, raising one and lowering the other] One decreased –

Instr: Anyone got another one?

Erica: Like this? [opens one hand and closes the other, eliciting general laughter]

Instr: This is great. A way to visualize these numbers changing helps everybody.

This collaboratively-constructed sequence of re-representations of the group's reported data patterns (doubling versus halving) is explicitly endorsed as something that is "great" and "helps everybody." This presents an opportunity to reflect on multiple ways to communicate these phenomena, encouraging students to think beyond direct quantitative comparisons, in ways that support reflection on the phenomena they are modeling, and their own ability to model it in different ways.
## Tina's group: generating migration concepts (represented-world level)

Tina begins with a concise statement that bundles together a number of historical observations:

**Tina:** Our group we have Mexico, India and Sweden - Sweden. And then we said that - that Mexico has the most um, population compare to India and Sweden. And then in Sweden people actually stayed in the middle first but then um, India they come from California and New York, those port| areas first.

In contrast to the previous group, this statement foregoes a process of recounting each CoO's data separately and avoids reading out specific numbers for individual counties. Instead, it presents summary comparisons that include multiple populations: Mexico had more population than either India or Sweden; Swedish and Indian immigrants came to different geographic areas "first" ("the middle" for Sweden, versus "those ||port|| areas" for India). This begins to bridge the modeling level with the represented-world level - using the patterns to begin to imagine the experiences of actual people in historical places. The instructor, sensing that there are multiple moves worth highlighting here, invites careful attention to Tina's observations:

**Instr:** I want everyone to get what Tina is saying. I want everyone to get this. Can someone repeat what she said? [no one volunteers immediately] [to Tina] OK, will you come up please and show us? I want everyone to get it. I don't want to let her off the hook until we all understand. She is comparing patterns now and she is not just using the numbers. So listen to how she describes it. So I've got Sweden right here. You want to do this? [motioning her to come up to the projecting computer]

**Tina:** [coming to computer] Alright, OK. So Sweden [selects Sweden, 1880] I think in the beginning is all in the middle [circles data pattern on display with mouse] but then if we compare to India [changes menu to India] there's no data over here until 1970 [changes year to 1970]. And then they all in the coastal areas - is that how you say it? Like New York [circling northeast data pattern] and California [circling southwest data pattern] compared to uh, Sweden again [selects Sweden, changes year back to 1880].

**Tina:** In the middle, in the middle [circling data pattern in central Midwest]

**Instr:** And you said "at the beginning"

**Tina:** Yes

**Instr:** So you compare when they *start* to show up, even though that's in different years.

Having brought Tina's group's complex spatial observations into the shared space of the projected display, the instructor again invites other students to re-represent it:

**Instr:** Can someone please restate what was the comparison that Tina made ... - the immigration patterns from people from Sweden, India and Mexico. How did she compare them? ...

**Nate:** So she compared them by uh, genesis point I guess. So when their populations first started to show up on the map, the sort of concentrations.

**Instr:** Can someone explain what Nate means by genesis point? He just gave us an awesome phrase that we can use. What is a genesis point for immigration? He's going to copyright it quickly so let's make sure we know. What does he mean by genesis point? Do you guys get this? If not please ask Nate to explain himself.

**Beth:** Like where they moved? Where they moved to? Entry point

**Instr:** Beth, say that again

**Beth:** Their entry point

**Instr:** Their entry point. Anyone else have another way of saying it?

**Marie:** The beginning of the census for uh, that specific region group um, on the map

**Instr:** The beginning of that census group on the map. You guys got it? Ok, so now ... so you said they each had a different genesis point.

**Nate:** Yeah, even though that was separated by years their uh, they had starting populations in areas and then spread out from there.
Several classmates are encouraged here to develop multiple ways of describing the phenomenon Tina's group introduced. The instructor's moves push the students beyond simply describing the patterns as midwestern and coastal, which locate more within the modeling context. By emphasizing her words "at the beginning," he brings out an important concept for historical reasoning about migrations: "So you compare when they *start* to show up, even though that's in different years." This moves the discussion from the modeling level to the represented-world level by pushing the comparison toward language that is relevant to phenomena of migration, leading to the development and uptake of Nate's concept of "genesis point" for an immigrant population, which is what enables Tina's group to make a geospatial comparison of the Swedish and Indian immigration patterns, despite their being separated in time by a century. Nate explains, "even though that was separated by years ... they had starting populations in areas and then spread out from there." As the class continues to examine this idea, Erica shifts to the represented world, connecting these patterns and her prior knowledge to a possible, partial explanation: "It seems like India's might be connected to hub airports because you have like Chicago, and is that Detroit?"

## Discussion

The learning opportunities created in this lesson proceed from the seeding of historical and spatial reasoning practices embedded in the progression of small-group activities. Like any CSCL design, the space of learning opportunities is mediated by the real-time decisions of the teacher, but the design of the learning environment provides distributed scaffolds and other resources that shape opportunities for disciplinary learning.

## Scaffolding multi-level disciplinary reasoning in group activity structures

All groups were able to bring back to the whole-class discussion descriptions of different CoO groups' changing population patterns. As the analysis demonstrates, there were notable differences between the two focal groups' representational practices in their presentations. Lisa's group remained mostly at the *modeling* level of reporting numbers and comparing quantitative changes for individual counties – valuable visualization practices that were taken up by the instructor and used to scaffold multiple representational moves in the classroom. Tina's group's comparisons of data patterns moved from the *modeling* level to the *represented-world*, enabling the instructor to engage the class in generating concepts for comparing regional migration patterns, even at different points in time.

Neither of these learning opportunities would have been likely in a lesson using this kind of webmap without the progression of distributed scaffolds orchestrated in the lesson. Although not all groups were able to provide descriptions that could be readily used to construct a historical narrative, enough groups were able to do so in order to provide the instructor with fodder to demonstrate how to use their observations in taking this next step. The instructor drew on the building blocks generated by the jigsaw groups to reposition their observations from discussions at the modeling level to discussions at the represented-world level. The progression from group presentations to the seeds of historical narratives in the culminating discussion suggests that the orchestration design, mapping multiple, multi-level comparisons into a sequence of group tasks, was effective in managing complexity and enabling novices to co-construct historical interpretations of complex visual data.

## Re-representation of patterns for disciplinary reasoning beyond the tool

Our analysis points to a promising strategy for fostering deep learning when using visualization tools. The aim of teaching with visualization tools is to move beyond the language of numbers, shapes or colors, to narrate the real-world phenomena they are meant to represent, so that learners develop knowledge about the phenomenon that endures even in the absence of the visualization tool. The interactive discussion here built on the jigsaw activity to encourage students to re-represent not only the data, but the patterns and phenomena they were co-constructing to compare migrations, both verbally and gesturally. Asking learners to re-represent patterns and concepts prompted co-construction of the ideas being discussed, free of the complexity of the data interface, while providing opportunities for formative assessment to guide the ongoing lesson. The outcomes of this strategy seen here suggest the value of further examining its potential for supporting deep learning and transfer.

## Conclusion

Following recent calls by leaders in CSCL (Ludvigsen et al., 2016; Wise & Schwarz, 2017) we examined how the design of classroom orchestration could reap the educational potential of sophisticated representational systems, which are used in professional and everyday contexts, in the college classroom. We propose that disciplinary task models can serve as a guide for making orchestration design decisions. We demonstrated how mapping configurations of group work to the conceptual reasoning moves that professionals make in using these tools could lead to disciplinary learning goals while using everyday tools like webmaps that were not designed as learning environments. Understanding how to orchestrate the classroom to cultivate students' disciplinary
thinking with tools that may not have built-in collaboration and reasoning scaffolds holds promise as a valuable area to explore in future CSCL work. Focusing on such tools may offer advantages beyond the support of disciplinary learning. Various visualization tools are becoming prevalent in daily life in online news, social networking, and other sites, making competency in interpreting these visualizations an important skill for civic participation. Using such tools in their everyday form in the classroom may lead to better transfer to everyday settings. In addition, as Roschelle et al have argued (2008), an alternative strategy for assuring the products of CSCL research have impact at scale is that rather than “scale up,” CSCL innovations be integrated into materials and tools that are already used at scale. Using disciplinary models to design orchestration around tools that are already widely available may be a way to achieve advances in college disciplinary learning at scale.

## References

Bowen, G. M., Roth, W. M., & McGinn, M. K. (1999). Interpretations of graphs by university biology students and practicing scientists: Toward a social practice view of scientific representation practices. *Journal of Research in Science Teaching, 36*(9), 1020-1043.

Dillenbourg, P., Prieto, L. P., & Olsen, J. K. (2018). Classroom orchestration. In *International handbook of the learning sciences: Routledge*.

Fischer, F., Kollar, I., Weinberger, A., Stegmann, K., Wecker, C., & Zottmann, J. (2013). Collaboration scripts in CSCL. *The international handbook of collaborative learning*, 403-419.

Goodwin, C. (1994). Professional vision. *American Anthropologist*, 96(3), 606-633.

Kaendler, C., Wiedmann, M., Rummel, N., & Spada, H. (2015). Teacher competencies for the implementation of collaborative learning in the classroom. *Educational Psychology Review*, 27(3), 505-536.

Knowles, A. K. (2008). Gis and history. In A. K. Knowles & A. Hillier (Eds.), *Placing history: How maps, spatial data, and gis are changing historical scholarship* (pp. 1-25). Redlands. CA: ESRI, Inc.

Ludvigsen, S., Cress, U., Law, N., Rosé, C. P., & Stahl, G. (2016). Future-looking conversations in escl. *International Journal of Computer-Supported Collaborative Learning*, 11(3), 255-262.

Owens, J. B. (2007). Toward a geographically-integrated, connected world history: Employing geographic information systems (GIS). *History Compass*, 5(6), 2014-2040.

President's Council of Advisors on Science & Technology Strategy to Improve K-12 STEM Education. (2012). *Engage to excel: Producing one million additional college graduates with degrees in science, technology, engineering, and mathematics*. Washington, DC: Executive Office of the President.

Prieto, L. P., Holenko Dlab, M., Gutiérrez, I., Abdulwahed, M., & Balid, W. (2011). Orchestrating technology enhanced learning: A literature review and a conceptual framework. *International Journal of Technology Enhanced Learning*, 3(6), 583-598.

Puntambekar, S., & Kolodner, J. L. (2005). Toward implementing distributed scaffolding: Helping students learn science from design. *Journal of Research in Science Teaching*, 42(2), 185-217. doi:10.1002/tea.20048

Radinsky, J., Milz, D., Zellner, M., Pudlock, K., Witek, C., Hoch, C., & Lyons, L. (2017). How planners and stakeholders learn with visualization tools: using learning sciences methods to examine planning processes. *Journal of Environmental Planning and Management*, 60(7), 1296-1323.

Raes, A., Schellens, T., De Wever, B., & Vanderhoven, E. (2012). Scaffolding information problem solving in web-based collaborative inquiry learning. *Computers & Education*, 59(1), 82-94.

Roschelle, J., Tatar, D., & Kaput, J. (2008). Getting to scale with innovations that deeply restructure how students come to know mathematics. In E. K. Anthony, R. A. Lesh, & J. Y. Baek (Eds.), *Handbook of design research methods in education* (pp. 369-395). New York, NY: Routledge.

Seixas, P. (2017). A model of historical thinking. *Educational Philosophy and Theory*, 49(6), 593-605.

Tabak, I. (2004). Synergy: A complement to emerging patterns of distributed scaffolding. *Journal of the Learning Sciences*, 13(3), 305-335.

Tabak, I., & Reiser, B. (2008). Software-realized inquiry support for cultivating a disciplinary stance. *Pragmatics & Cognition*, 16(2), 307-355.

Vogel, F., Wecker, C., Kollar, I., & Fischer, F. (2017). Socio-cognitive scaffolding with computer-supported collaboration scripts: A meta-analysis. *Educational Psychology Review*, 29(3), 477-511.

Wise, A. F., & Schwarz, B. B. (2017). Visions of CSCL: Eight provocations for the future of the field. *International Journal of Computer-Supported Collaborative Learning*, 12(4), 423-467.

## Acknowledgments

Funded in part by the I-CORE Program of the Planning and Budgeting Committee (LINKS center) and The Israel Science Foundation grant 1716/12, and by NSF grants DRL-0953448 and INSPIRE-1248052.
# Data Wrangling Practices and Process in Modeling Family Migration Narratives with Big Data Visualization Technologies

Shiyan Jiang, Carnegie Mellon University, shiyanj@cs.cmu.edu

Jennifer B. Kahn, University of Miami, jkahnthorne@miami.edu

**Abstract:** Big data technologies are powerful tools for telling evidence-based narratives about oneself and the world. In this paper, we examine the sociotechnical practices of data wrangling—strategies for selecting and managing datasets to produce a model and story in a big data interface—for youth assembling models and stories about family migration using interactive data visualization tools. Through interaction analysis of video data, we identified ten data wrangling practices and developed a conceptual model of the data wrangling process that contains four interrelated recursive stages. These data wrangling practices and the process of data wrangling are important to understand for supporting future data science education opportunities that facilitate learning and discussion about scientific and socioeconomic issues. This study also sheds light on how the family migration modeling context positioned the youth as having agency and authority over big data.

**Keywords:** data wrangling, modeling, storytelling, family migration, data science education

## Introduction

As the interdisciplinary field of data science has grown, large-scale datasets, also known as big data, and interactive data visualization tools have become increasingly open and accessible, creating opportunities for learning. Assembling models and narratives with big data is an important STEM practice for many professions (Kosara & Mackinlay, 2013) and for participating in civic discourse (Philip, Schuler-Brown, & Way, 2013). The role of big data and visualizations in public conversations can be attributed to these technologies' capacities to support critical inquiry (boyd & Crawford, 2012). These tools are especially powerful for enriching narratives about the social and scientific world and providing evidence to challenge or confront misinformation.

Opportunities for youth to engage in data science activities that support analyses of social and scientific issues from local and global perspectives are needed. However, few studies have closely investigated how youth engage with big data interfaces at the level of discursive and embodied interaction. In this paper, we examine the sociotechnical practices of data wrangling (introduced in Kahn, under review), the strategies for selecting and managing datasets, for youth assembling models and stories about family migration with public big data using interactive visualization tools. We consider data wrangling as necessary for storytelling and modeling with big data, and an examination of the active interactions between youth modelers and visualization tools is necessary for understanding data wrangling. Our analysis addresses the following research questions: 1) *What are the enactive practices that describe participants' data wrangling interactions with big data visualization tools?* 2) *What is the conceptual process of data wrangling to assemble family data storylines?*

## Theoretical framework

First, this research follows an interactionist perspective (Greeno, 1994) that views learning as occurring through participation situated in activity (Bandura, 1986; Greeno & Engestrom, 2014; Lave & Wenger, 1991). Second, we approach family storytelling with big data as a representational, sociotechnical activity that involves coordination of bodies and tools (Hall & Nemirovsky, 2012). Third, learners' capacities, dispositions, and histories are central in our learning environment design and study of learning processes.

Furthermore, our study builds on a collection of data science education studies that have found that interactive, multivariable data visualization tools support learning across STEM and social studies disciplines (e.g., Philip, Olivares-Pasillas, & Rocha, 2016; Polman & Hope, 2014; Radinsky, Hospelhorn, Melendez, Riel, & Washington, 2014; Rubel, Hall-Wieckert, & Lim, 2017) and understandings of statistical concepts (e.g., Engel, 2017; Harrison, Yang, Franconeri, & Chan, 2014). These studies also call for more opportunities for youth to develop critical data literacy by using visualization tools to ask/answer questions and make inferences about data in personally and culturally meaningful ways (Borner, Peppler, Kennedy, Uzzo, & Heimlich, 2017).
# Methods

## Context and participants

We report on a single iteration of a design-based research program focused on exploring how youth and young adults learn to tell stories and build models about social and scientific issues with big data. Previous iterations found that making personal connections to large-scale phenomena represented by big data could be productive for generating critical perspectives. In turn, this design iteration sought to explore the benefit of embedding storytelling and modeling with big data in a personal context and to better understand the role of big data interfaces in relating personal experiences to a larger scale—the scale of the phenomena being represented by the big data. Conceptually, our design was thus intended to support learning across temporal, spatial and social scales (Hall & Leander, 2010). We chose family migration because the sharing of family histories benefits (Fivush, Bohanek, & Zaman, 2011), and migration continues to be a pressing issue globally, nationally, and in our communities. We hoped that this study would offer insight into how to facilitate learning and productive dialogue around such a timely matter.

In the current study, middle and high school youth (N = 17; self-identified as 6 male; 11 female; 13 African-American; 3 White; 1 Asian; Mean hours of attendance = 13; sample included 6 sibling pairs) created family data storylines to explore reasons for personal family mobility (What moved my family?) as well as national and global migration (What moves families?) in a free summer workshop at a city public library. Youth represented family decision-making and social conditions with online modeling and mapping tools and related the lives of their ancestors to their own experiences and futures. The three weekly workshop sessions (2 days per week, 5 hours per day) culminated in a public community exhibit.

Assembling family data storylines involved the following tasks: First, participants chose a side of the family to focus on. Second, participants chose one of two interactive web-based data tools accessed via their laptop's Internet browser: Social Explorer (Figure 1) or Gapminder (Figure 2). Both tools were selected because they afford (Gibson, 1979/1986) interactivity, as opposed to static data displays. Social Explorer is a historical thematic mapping tool that uses US demographic data. It accesses hundreds of variables from Census and other demographic datasets that go as far back as 1790. Data can be encoded as multiple visualization types—as dots, with colorful shading, with bubble size—and at national, regional, and local scales. In Social Explorer, users can also create different side-by-side temporal and spatial comparisons. Gapminder is a multivariable, dynamic graphing tool that uses public global socioeconomic data. It has five possible quantities or variables that can be selected. Y-axis, x-axis, color, and bubble size each have over 500 health and wealth indicators or measures to choose from. Timescales of datasets vary: Some start as early as 1800, and others only have one year of available data. In Gapminder, users can also alternate between logarithmic and linear scales, and select particular country bubbles to leave data trails over time. Third, participants selected variables from each tool's available datasets, like education level or household income in the US Census. Participants captured screenshots of models or maps and inserted them into a Microsoft PowerPoint, accompanied by slides or texts that explained data selections and what participants learned from the data.

## Learning environment design

Our learning environment design aligns with a 4E learning model. First, our design recognizes STEM modeling as distributed and embodied activities (Hall & Nemirovsky, 2012; Hutchins, 1995). We thus included activities beyond interacting with computers. For example, once or twice in each workshop session, the day started with a “four walls” game that set up comparisons between experiences of the youth and that of their ancestors, related to circumstances surrounding family migration (i.e., family size, educational attainment, occupation, neighborhood racial diversity). We asked questions such as: “How far along in school did your family member go?” (Wall 1: Grade school, Wall 2: High school, Wall 3: College and beyond, Wall 4: No formal schooling), which was followed by “How far along do YOU expect to go?” In Sessions 2 and 3, we performed a walking-scale timeline that, like the four walls game, asked participants to stand in for their family in historical time. These embodied comparison activities asked the participants to consider the historical social conditions that motivated or forced their families to move and that possibly could be explored with the data tools.

Second, we consider our design to be an example of extended learning with big data. Learning was distributed across workshop activities and family members. Youth assembled their storylines with siblings who were in the workshop and with parents, via phone calls, text messages, visits during the workshop day, and at-