home conversations over intervening nights and days, in which pieces of family story were filled in or changed or corrected by family members.

Third, our design embedded the sociotechnical practices of data science in a personal topic (family migration) and in a common, cultural, collaborative activity (family storytelling).

Finally, as described earlier, the focus for the current analysis is on enactive learning, or how the youth participants interacted with designed activities and context for storytelling and modeling with big data.

## Data collection and analysis

We video and audio recorded all activities and recorded participants' work on laptop computers with screen capture software. Our qualitative analysis took an interactionist approach (Greeno, 1994; Azevedo & Mann, 2018) towards engagements between individual agents and the learning environment. This approach, with a focus on both bodily interactions with the physical environment and tools as well as conceptual practices and perceptions of participants, supports the study of enactive and multimodal, embodied learning. We used interaction analysis methods (Jordan & Henderson, 1995) to understand participants' data wrangling strategies. As we reviewed participant trajectories (Jiang, 2018) across activities, we developed analytic memos around what appeared to be—through continuous or constant comparisons (Glaser, 1965) of individual participants—conceptual and technical practices to describe participants' data wrangling. We then selected episodes for microanalysis, in which we analyzed gesture, discourse, and tool use as sources of knowledge and learning. We often recreated maps or models in Gapminder or Social Explorer to better understand participant data explorations. The comparison of participants also focused on critical reflections on the data (i.e., data quality, data stakeholders), social history, and family migration. We reviewed and compared participant records until no new categories of data wrangling practices emerge (Strauss & Corbin, 1994). After identifying the categories of practices, we reanalyzed the selected episodes for microanalysis to trace the trajectories of practices and developed themes around the process of data wrangling.

## Findings

We define data wrangling as the practices that manage multiple datasets and measures in order to connect narratives with aggregate data. Data wrangling in an open data interface appears to involve the application of technical and statistical knowledge as well as an individual's social values and sense of identity. For instance, many of the participants' interactions with the data tools involved locating places that held personal meaning for them, whether in their family histories or own experiences, or selecting datasets that they identified with. In previous iterations, we have called this "getting personal with big data" (Kahn & Hall, 2016). Consequently, in our instructional context, data wrangling served as the means for participants to establish self-society relations (Kahn, under review).

## RQ1: What are the enactive practices that describe participants' data wrangling interactions with big data visualization tools?

We identified ten sociotechnical practices that comprise data wrangling for youth: 1) filtering data; 2) selecting indicators or variables; 3) data visual encoding; 4) interpreting data points; 5) identifying data patterns; 6) pursuing data surprises; 7) reasoning about data relationships; 8) countering data; 9) approximating data; and 10) making data predictions. These practices tended to be in the service of building comparisons to tell a story about both family history and broader socioeconomic trends (e.g., a comparison of origin place A to destination B that shows better economic conditions in B). Below, we introduce and define each practice and provide several brief illustrations.

In **filtering data**, participants either select a country bubble and all others fade in Gapminder or locate specific locations in a map in Social Explorer using the zoom feature (e.g., Figure 1). This practice results in highlighting specific data points or locations through a change of data visualization, but there is no change to the dataset being used to build the data visualization. Making such selections is challenging in an open data interface, when there are hundreds of datasets and variables to choose from. Consequently, personal connections tended to drive participant selections from the beginning, such as when youth selected a country in which their family members lived or zoomed into their neighborhoods on a map.
<figure><img src="image_97.png" /><figcaption>**Figure 1.** A participant is viewing the violent crime rate of her neighborhood in Social Explorer.</figcaption></figure>

**Selecting indicators or variables** involves participants' changing default indicators or variables by following specific constraints. This practice changes the dataset being used to build the data visualization. While the constraints guiding indicator selections varied among participants, the most common constraints included a) finding a dataset based on a predeveloped storyline, such as one participant's selection of *Mean Years in School for Men 25 Years and Older* in Gapminder to explore whether her father moved to the U.S for better educational opportunities; b) following a scale, such as one participant's successive selections of *Household Income Less Than $10,000, Less Than $50,000, and Less Than $200,000* in Social Explorer; c) identifying oneself or people that participants were familiar with in the data, such as one participant (13 years old, female) selecting *Female Population Aged 10-14 Years* in Gapminder.

**Data visual encoding** is the practice of interpreting or understanding the relationship between two representations in which data is mapped into visual structures (e.g., color legends). For instance, in a choropleth map, areas are shaded in proportion to the measurement of variables (e.g., dark colors represent higher population density while light colors represent lower population density). Data visual encoding usually occurred when participants were trying to a) understand how the data is encoded in different visual representations, such as when one participant said, “Sure are a lot of crime rates” after hovering her mouse over a dark red area in Social Explorer; or b) change the details of visual encoding to leverage the data visualization for storytelling purposes. For example, in order to support the claim of moving due to safety concerns, one participant changed visual encoding from “one dot represents the value of 10,000” to “one dot represents the value of 5,000” in Social Explorer to make the differences in the number of crimes between two states more visible. As another example, guided by a researcher, one participant changed the x- and y-axis scales from linear to logarithmic in Gapminder to explore potentials of each visualization in explaining family story. Although the participant did not understand the statistical concept, she was aware of the change in visualization. A persistent challenge across interfaces for youth was differentiating between data encoding for rates or percentages and counts.

**Interpreting data points** refers to the practice of interpreting a single data point at one time. This practice was marked by highlighting with the cursor tooltip or selecting (filtering) a particular bubble in Gapminder or discrete geographic area in Social Explorer and by providing verbal explanations of the meaning of the numbers in the tooltip in discourse, as illustrated below.

### Excerpt 1 [Week 1, Day 1]

Francine (*hovers mouse over US bubble in Gapminder, turns to her tablemate*): See, right here. It's United States. It's about how much money they earn, 53.4, and how long they live, 79.1. (1)

**Identifying data patterns** represents the practice of describing a data trend, such as the trend of a variable for a single geographic area over time (e.g., for a state 1920–1970) or the trend of a variable for multiple areas at one point in time (e.g., a regional trend in the year 2000). For instance, in the excerpt below, participant Sage, whose mother came from Thailand and father was born in the US, described a data pattern for life expectancy while watching an animation in Gapminder in which she compared the two countries.

### Excerpt 2 [Week 1, Day 2]
Sage: It's going down a little bit for the United States (Figure 2a).

Sage (while seeing a drop in life expectancy in Thailand and in the US in 1919 [Figure 2b]): Oh, no!

Sage: Life expectancy is going up quite a bit in both countries now (Figure 2c).

<figure><img src="image_98.png" /><figcaption>Figure 2a, 2b, 2c. In Sage’s Gapminder model, the US and Thailand are selected, and time plays from 1800 through 2015. Life expectancy (y-axis) and income (x-axis) generally increased over time for the US and Thailand, except for a dramatic drop in life expectancy in both countries in 1919 (2b).</figcaption></figure>

**Pursuing data surprises** involves participants' actively looking for or noticing outlier data, typically represented by a divergent bubble trail in Gapminder or a very light or dark region in Social Explorer. For instance, while noticing the drop in Figure 2b, Sage went to search online for possible reasons for a dramatic decrease in life expectancy in Thailand in 1919. This example indicates the close relationship between the practices of identifying data patterns and pursuing data surprises.

**Reasoning about data relationships** refers to when participants describe relationships between two or more variables or covariation to explain family migration. In most cases, participants had difficulties relating two variables and tended to examine indicators (e.g., household income, educational attainment) independently.

**Countering data** involves sharing a personal experience or account that does not align with the data. For example, Francine explained that her father went to school every year while noticing that the average number of years of schooling for men in Sudan, according to Gapminder, was only 3 years in the 1980s. Another participant, Naimah, who assembled a Social Explorer comparison showing an increase in the Black population in the North and a decrease in Black population in the South between 1920 and 1970, noted that Social Explorer did not include how “moving from South Carolina affected [her] family's connections, and how it was a milestone in [her] history.” We view this kind of critique as a challenge to what the Census takers deem as important for understanding social history. This practice shows that participants had agency and authority in questioning big data, which challenges the contention that personal experience serves to confirm data for youth (Enyedy & Mukhopadhyay, 2007).

**Approximating data** describes when participants make selections that are approximations for the data they were looking for when there is no data for the exact year, category, indicator, country or place that they desire. For example, for Naimah, survey years 1920 and 1970 roughly corresponded with the birth of her paternal great-grandparents in South Carolina and their decision to leave (1920) and her grandparents' return to the South (Alabama) from Illinois (1970). These approximations can be understood as emergent fixes to manage uncertainty (Star, 1985) or missing data in data wrangling.

**Making data predictions** represents the practice of imagining what the data could show in the future. In our analysis, this practice did not occur while participants interacted with Gapminder and Social Explorer, although participants considered their own futures in our activities. For instance, in her family data storyline, Hannah compared the percentage of persons 25 years and over with some college experience or more between Chicago and Oakville in 1990 to examine whether her father moved to Oakville for college. She subsequently included a slide in her family storyline describing where she wanted to attend college in the future.
## RQ2: What is the conceptual process of data wrangling to assemble family data storylines?

We developed a conceptual model (Figure 3) of the data wrangling process that contains four interrelated recursive stages: 1) *finding my family and me in big data*, 2) *understanding my family and me in relation to the society*, 3) *challenging big data with authority and agency*, and 4) *constructing a big data model to share my family story*. During the stage of *finding my family and me in big data*, participants selected data and indicators related to themselves. They described how historical events affected family migration in the stage of *understanding my family and me in relation to the society*. In the stage of *challenging big data with authority and agency*, participants critically analyzed data when the data was not consistent with their initial understandings of family stories and actively explored alternative indicators that might contribute to family migration. They selected indicators and visualizations that aligned with the family story in the stage of *constructing a big data model to share my family story*. Participants tended to start with finding data related to themselves in this process (*finding my family and me in big data*), which is a “first move” in data wrangling, and ended with constructing models for assembling their family data storylines (*constructing a big data model to share my family story*), which they presented to their peers and were displayed in a public exhibit.

The four stages are recursive, and participants moved back and forth between these stages. Although participants' conceptual movements between these stages indicate a variety of paths in the process of data wrangling, some movements were more challenging for the youth. For instance, Sage engaged in comparing different variables (e.g., income) between Thailand and the U.S in the stage of *finding my family and me in big data*. Her mother moved from Thailand to the U.S for love, which is an abstract concept that she had difficulties in finding variables to represent in the stage of *constructing a big data model to share my family story*. In her case, the movement between these two stages rarely occurred after several failures in finding appropriate variables to represent the abstract concept.

<figure><img src="image_99.png" /><figcaption>Figure 3. A conceptual model of data wrangling to assemble family data storylines.</figcaption></figure>

## Discussion and implication

Our study illustrates what data wrangling looks like in an informal learning setting and how students engaged with various sociotechnical practices in the process. Participants engaged in learning about their families to learning about society through data wrangling. Typically, one practice went hand-in-hand with other practices (e.g., interpreting the meaning of a dark red county involves both the practices of visual data encoding and interpreting data points) in interrelated recursive stages.

The study shows that the “first move” in data wrangling tends to be finding data related to oneself while youth navigate data visualization tools to assemble their family storylines. This finding supports the current understanding of the youth’s interactions with big data while contributing new insights: Whereas research in data science and STEM education has evidenced the value and challenges of culturally relevant pedagogy (e.g., Enyedy & Mukhopadhyay, 2007), this study illuminates that getting personal with big data provides an entry point and access to interactions with big data technologies. While other instructional designs with these tools reduced the number of indicators available for exploration (e.g., Radinsky et al., 2014), we found that it is important to provide a rich and inclusive dataset to offer equitable learning opportunities in data
wrangling. This finding suggests that future designs could engage youth in exploring big data technologies with
intentional scaffolds for getting personal with data.

Also, this study addresses tensions of confirming data presented in Enyedy and Mukhopadhyay's
(2007) study, in which youth use data to confirm pre-existing conclusions without questioning the model. Our
study demonstrates that family data storylines fostered the youth's authority and agency over big data, and this
produced both questions about and challenges to the big data. This finding has implications for designing
learning activities with big data in which youth have their own voices. The literature supports this design
principle and studies have shown the importance of building a close relation between the self and digital
artifacts (e.g., Philip et al., 2016). Future research can pursue other personal contexts for big data exploration,
such as daily mobility datasets or personal health data.

In addition, our study presents several challenges that need to be addressed in data wrangling practices
in future studies. The first challenge is ensuring the understanding of statistical concepts, such as the difference
between counts and rates, as well as the understanding of how to make a reasonable comparison, such as
zooming two maps to the same level to set up comparisons. A second challenge is engaging youth in discussions
of whether variables in a dataset can be used to measure abstract concepts (e.g., love). A third challenge is
encouraging more data exploration when the model does not correspond with family stories. In one such case,
data wrangling consequently went unresolved (i.e., no data was included in Isis's final family data storylines).

Although each youth assembled their own family data storyline on an individual laptop, participants
frequently interacted with tablemates, who were often their siblings, when they discovered data that they were
personally connected to, such as data representing the city where participants lived. Future research is needed to
investigate how to support getting personal with data in more collaborative learning settings, such as data
modeling activities designed for intergenerational families or teams of peers. In addition, this study shows that
youth had more exchanges with each other when they encountered trouble in data wrangling, such as when they
encountered data that did not align with their initial understandings of why their families moved. These
exchanges, particularly between siblings, entailed negotiations of authority in interpreting data and telling the
family history, such as Isis told her older sister Naimah, "I do not need you to tell me about my father's story."
Much more needs to be learned about the effect of agency and authority on peer interactions with big data.

Our findings regarding data wrangling practices and process should be the starting point in research on
designs for learning and instruction that promote modeling narratives with big data visualization technologies in
different contexts. For example, future studies can examine whether some practices (e.g., identifying data
patterns) are more influenced by the tool affordances (e.g., bubble trail in Gapminder) while some practices
(e.g., pursuing data surprises) are more universal across data wrangling activity and interfaces. Although these
practices were used with two interfaces (motion charts and web maps), we expect they would apply to other data
interfaces involving multimodal, multivariable dynamic representations (e.g., stacked graph and radar chart).
We believe that big data technologies are valuable instructional tools for offering learning opportunities and
communication with others about important social and scientific issues.

## Endnotes

(1) All names are pseudonyms.

## References

Azevedo, F. S., & Mann, M. J. (2018). Seeing in the dark: Embodied cognition in amateur astronomy practice. *Journal of the Learning Sciences, 27*(1), 89-136.

Bandura, A. (1986) *Social foundations of thought and action: A social-cognitive view*. Englewood Cliffs, NJ: Prentice-Hall.

Borner, K., Peppler, K., Kennedy, B., Uzzo, S., and Heimlich, J. (2017). "Data Visualization Literacy: Research and Tools That Advance Public Understanding of Scientific Data" *NSF - National Science Foundation* Available at https://www.nsf.gov/awardsearch/showAward?AWD_ID=1713567

boyd, D., & Crawford, K. (2012). Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon. *Information, communication & society, 15*(5), 662-679.

Engel, J. (2017). Statistical literacy for active citizenship: A call for data science education. *Statistics Education Research Journal, 16*(1), 44-49.

Enyedy, N., & Mukhopadhyay, S. (2007). They don't show nothing I didn't know: Emergent tensions between culturally relevant pedagogy and mathematics pedagogy. *The Journal of the Learning Sciences, 16*(2), 139-174.

Fivush, R., Bohanek, J. G., & Zaman, W. (2011). Personal and intergenerational narratives in relation to adolescents' well-being. *New directions for child and adolescent development, 2011*(131), 45-57.
Gibson, J. J. (1986). *The ecological approach to visual perception*. Hillsdale, NJ: Erlbaum. (Original work published 1979)

Glaser, B. G. (1965). The constant comparative method of qualitative analysis. *Social problems, 12*(4), 436-445.

Greeno, J. G. (1994). Gibson's affordances. *Psychological Review, 101*, 336-342

Greeno, J. G., & Engeström, Y. (2014). Learning in activity. In K. Sawyer (Ed.), *The Cambridge handbook of the learning sciences* (pp. 128–147) (2nd ed.). London: Cambridge University Press.

Hall, R. & Leander, K. (2010, July). Scaling practices of spatial analysis and modeling. In Hall, R. (Symposium Chair). Scaling practices of spatial analysis and modeling. International Conference of the Learning Sciences.

Hall, R., & Nemirovsky, R. (2012). Introduction to the special issue: Modalities of body engagement in mathematical activity and learning. *Journal of the Learning Sciences, 21*(2), 207-215.

Harrison, L., Yang, F., Franconeri, S., & Chang, R. (2014). Ranking Visualizations of Correlation Using Weber's Law. *IEEE Trans. Vis. Comput. Graph.*, 20(12), 1943-1952.

Hutchins, E. (1995). How a cockpit remembers its speeds. *Cognitive Science*, 19(3), 265-288.

Jiang, S. (2018). STEM+ L: Investigating Adolescents' Participation Trajectories in a Collaborative Multimodal Composing Environment. *ProQuest LLC*.

Jordan, B., & Henderson, A. (1995). Interaction analysis: Foundations and practice. *Journal of the Learning Sciences, 4*(1), 39-103.

Kahn, J. (under review). Learning at the intersection of self and society: The family geobiography as a context for data science education. *Journal of the Learning Sciences Special Issue: Situating Data Science*.

Kahn, J., & Hall, R. (2016, April). Getting personal with big data: Stories with multivariable models about global health and wealth. Paper presented at the American Education Research Association 2016 Annual Meeting, Washington D.C.. Available in the AERA Online Paper Repository.

Kosara, R., & Mackinlay, J. (2013). Storytelling: The next step for visualization. *Computer, 46*(5), 44-50.

Lave, J., & Wenger, E. (1991). *Situated learning: Legitimate peripheral participation*. Cambridge, England: Cambridge University Press.

Philip, T. M., Olivares-Pasillas, M. C., & Rocha, J. (2016). Becoming racially literate about data and data-literate about race: Data visualizations in the classroom as a site of racial-ideological micro-contestations. *Cognition and Instruction, 34*(4), 361-388.

Philip, T. M., Schuler-Brown, S., & Way, W. (2013). A framework for learning about big data with mobile technologies for democratic participation: Possibilities, limitations, and unanticipated obstacles. *Technology, Knowledge and Learning, 18*(3), 103-120.

Polman, J. L., & Hope, J. M. (2014). Science news stories as boundary objects affecting engagement with science. *Journal of Research in Science Teaching, 51*(3), 315-341.

Radinsky, J., Hospelhorn, E., Melendez, J. W., Riel, J., & Washington, S. (2014). Teaching American migrations with GIS census webmaps: A modified “backwards design” approach in middle-school and college classrooms. *Journal of Social Studies Research, 38*(3), 143-158.

Rubel, L. H., Hall-Wieckert, M., & Lim, V. Y. (2017). Making Space for Place: Mapping Tools and Practices to Teach for Spatial Justice. *Journal of the Learning Sciences, 26*(4), 643-687.

Star, S. L. (1985). Scientific work and uncertainty. *Social studies of Science, 15*(3), 391-427.
# Promoting and Tracing High School Students' Identity Change in an Augmented Virtual Learning Environment

Hamideh Talafian, Mamta Shah, Amanda Barany, and Aroutis Foster

Ht343@drexel.edu, mamta@drexel.edu, amb595@drexel.edu, aroutis@drexel.edu

School of Education, Drexel University

**Abstract:** This paper illustrates the application of Projective Reflection (PR) as a theoretical framework in an ongoing NSF-CAREER study to trace learning as identity change among 20 high school students. A 9-week augmented virtual learning environment (AVLE) course, *Virtual City Planning*, was created and implemented in a science museum classroom. In-game and in-class student data and mentor observations were collected and analyzed using Quantitative Ethnography (QE) techniques to identify patterns of identity exploration enacted over time. This paper offers group findings to illustrate general trends of identity exploration during the AVLE, and one illustrative case study to provide an in-depth look at an integrated identity exploration trajectory. This work concludes with a discussion of the need for promoting identity exploration as an educational goal, and a means to support learners to adapt to the changing workforce of tomorrow. The affordances of AVLEs are discussed to advance research in this nascent area.

## Introduction

Educational research has increasingly emphasized ways to develop learner skills in *identity exploration*, or the intentional repeated examination of who a learner is and who he/she wants to become (Kaplan & Flum, 2012), which can lead to identity change in targeted directions over time (i.e. change towards a future in a STEM career) (Foster, 2014). Virtual learning environments (VLEs) such as games have been increasingly identified and researched for their potential to support change in both cognitive (i.e. knowledge) and affective (i.e. motivation) elements of self (e.g. Kamarainen, Metcalf, Grotzer, & Dede, 2015). To rigorously leverage these affordances, however, research points to the need for robust conceptual and design frameworks for developing game environments (DeVane, 2010), and the purposeful inclusion of real-world pedagogy that can supplement game content as part of an integrated learning experience (Clark, Tanner-Smith, & Killingsworth, 2015). We refer to this merging of a primarily virtual environment with real-world augmentations as an augmented virtual learning environment (AVLE) (Milgram, Takemura, Utsumi & Kishino, 1994), which integrates the benefits of both worlds to comprehensively provide identity exploration opportunities. Rowsell & Abrams' (2011) *I/identity* framework offers a useful starting point with its “multidisciplinary approach to mediating I/identity in the 21st century,” with consideration for both individual (i)identity change, or “an embedded sense of self that inherently fuels and/or shapes one’s behavior” (p. 4), and collaborative aspects of (I)identity change that are “rooted in practices and overt forms of meaning-making” (p. 4) offered by AVLEs.

Thus, in this paper, we illustrate one way of conceptualizing learning as a process of exploring possible role selves through experiences in AVLEs to gain the knowledge and skills to reconstruct oneself through a theory called as Projective Reflection (Foster, 2014). The paper describes the application of Projective Reflection in an ongoing NSF-CAREER study that informed the design and implementation of *Philadelphia Land Science (PLS)*, a text and web-based VLE for exploring environmental science and urban planning roles. The in-game experiences mediated by PLS were augmented by an external in-class curriculum to further scaffold students’ exploration of the two roles. Thus, the combination of PLS + the external curriculum was referred to as the Virtual City Planning AVLE, which was offered to 20 high school students over a 9-week period in a science museum in a northeastern US city. Data corpus included in-game logged data, research memos, pre-post interview questions, written and visual artifacts created in-game and during external individual and group curricular, reflection and discussion activities. These data sources were used to examine the following research question: “To what extent did participants engage in an integrated identity exploration in Virtual City Planning as a result of exploring the roles of an environmental scientist and urban planning?” Implications are discussed to advance educational research on framing, facilitating and tracking learning and identity in AVLEs through the use of emerging frameworks such as Projective Reflection.

## Theoretical framework

Projective Reflection (PR) is a theoretical framework that conceptualizes learning as identity exploration that facilitates learners' engagement in self-transformation or identity change in AVLEs and explicates the process of
learning as identity exploration over time (See Figure 1) (Foster, 2014). Four theoretical constructs in PR support identity exploration, and structure assessments of learners' cognitive and affective change over time: (1) knowledge, (2) interests and valuing, (3) self-organization and self-control, and (4) self-perception and self-definition (Foster, Shah, & Barany, 2017). Six questions are used to synthesize these changes into a comprehensive characterizations of a learner's initial current self, their exploration of possible selves (measured repeatedly across the experience), and the new self at the end of the intervention (Foster, 2014): 1) what the learner knows – current knowledge, 2) what the learner cares about – self and interest/valuing, 3) what/who the learner expects to be throughout the virtual experience and their long term-future self, 4) what the learner wants to be – possible self, 5) how the learner thinks – self and interest, and 6) how the learner sees him/herself – self-perception and self-definition (see Table 1).

<figure><img src="image_100.png" /><figcaption>**Figure 1.** The Projective Reflection framework for conceptualizing learning as a process of identity change over time.</figcaption></figure>

Over the duration of an intervention, a learner is encouraged to engage in targeted and intentional reflection on aspects of self along the four PR constructs, including (a) her/his starting self at the beginning of a learning experience, (b) repeatedly throughout the exploration of possible selves as supported by the AVLE, and (c) her/his new self at the end of the intervention. Furthermore, AVLEs are designed to have opportunities for participants to play, engage in curricular activities, reflect and discuss in relation to the roles explored both in the game and outside (Foster & Shah, 2016). For each student, the process of tracing change is carried out chronologically, guided by the six questions which allow us to explicate the extent to which the learner explored a possible role by developing intentional changes in their knowledge, interest and valuing, self-perception and self-definition, and self-organization and control skills- indicating the extent to which the identity exploration was comprehensive or integrated. In this paper, we report the case of a female student as an elucidation of one integrated identity exploration trajectory over the 9-weeks from the start of her participation in Virtual City Planning (weeks 1-2), during the course (weeks 3-7), and the end of the course (weeks 8-9). Group patterns of identity exploration are also identified to not only better understand the case study, but also to highlight the overall changes in the four constructs informing identity exploration as a result of engaging in the AVLE over the 9 weeks.

## Methodology

This research was conducted as part of an ongoing 5-year NSF CAREER project awarded to advance theory and research on promoting identity exploration and change in science using virtual learning environments using Projective Reflection (Foster, 2014). Building on this broader agenda, researchers designed and implemented an AVLE that featured the weekly use of the VLE *Philadelphia Land Science (PLS)* and supportive real-world augmentation in the classroom (for more information see Shah et al., 2018). The resulting AVLE, titled Virtual City Planning was offered to a total of 54 Philadelphia high school students who participated in weekly STEM career programming at their local science museum. *Virtual City Planning* was offered to facilitate learning as identity change by engaging participating students in real-world activities on environmental science and urban planning. *Philadelphia Land Science (PLS)* allowed players to explore roles related to urban planning and environmental science careers as they connect to the Philadelphia context and redesigned/rezoned the map of the city with different stakeholders. In *PLS*, groups of students were synchronously guided by online and in-person mentors to collaborate with their peers in redesigning the map of the city. Building on the strengths of *PLS*, the research team designed real-world features (i.e. non-digital design and reflection activities) that leveraged affordances of classroom context to further support identity exploration.

Three versions of Virtual City Planning were offered from September, 2016 to May, 2017, each successively refined from the previous offering (9-weeks in Fall with 20 students, 8-weeks in Winter with 19 students, and 4-weeks in Spring with 19 students) to strengthen the extent to which the activities in the AVLE could support students' process of projective reflection (Cobb, Confrey, DiSessa, Lehrer, & Schauble, 2003). This paper focuses on the first iteration of Virtual City Planning, which was enacted over nine weekly 90-minute
sessions with 20 students in the science museum. 60% of them were females (n = 12), 30% were males (n = 6) and the other 10% preferred not to reveal their sex (n = 2). They attended a local magnet school which focused on STEM preparation and had several programs for teachers' professional development and students' STEM enrichment integrated with/within the museum.

## Procedures and data collection

Data sources included logged in-game data, classroom artifacts, and researcher observations. Once collected, data was organized chronologically by student to track PR changes from the beginning, at repeated points during, and at the end of the AVLE.

Over the 9-weeks, Virtual City Planning allowed participants to learn about the process of proposing a rezoning proposal for the city of Philadelphia that addresses the competing and complementary needs of four stakeholders in a well-balanced manner. Activities in Philadelphia Land Science and outside of it were designed to facilitate intentional shifts in what learners know, how they think, what they care about, how they see themselves, what they want to be and expect to be in relation to environmental planning and urban science. For the first half of the course, learners were assigned to one of the four stakeholder groups to explore specific environmental and/or economic needs of the city. The four stakeholders were Philadelphia Institute for Environment Preservation, Philadelphia Economic Affairs Coalition, Environmental Council for Greater Philadelphia, and Bridgeway Community Action Group. In the second half of the course, students were jigsawed in a manner that every new group had a representative from each of the four stakeholder groups; thus, simulating the real-world scenario of multiple interests and expertise coming together to address real-world issues. The two parts of the course prepared students towards the goal of proposing a rezoning plan.

In Session 1, we engaged players in learning about their teams and understanding the expected workflow of PLS. In this session, we also gathered data to establish students' starting self by way of an intake interview in PLS, background survey, and a focus group discussion. Sessions 2 and 3 required participants to research the stakeholders' concerns from various factions of the city (commercial, residential, environmental, civic, social) and understand what is expected of the urban scientists in the 'request for a re-zoning proposal.' Sessions 4 and 5 engaged players in creating models to create and test the extent to which their re-zoning proposal balanced the needs of the stakeholders on an interactive map of Philadelphia using a mapping tool called iPlan. They received feedback from the stakeholders and their mentors which lead to iterative refinements in sessions 6-8. Data were obtained from sessions by way of in-game logged data (chats, notebook entries, iplan maps, researcher memos) to understand what each participant was focusing on more or less along the four constructs as he/she was exploring the role assigned and those that emerged naturally, and how the participant was engaged in the identity exploration process individually, in participation with PLS, and within the classroom setting (interaction with peers, mentors). Finally, in session 9, players worked towards finalizing a written document explaining the rezoning plan and representing the plan on a map. Data was collected to establish participants new self by way of an in-game exit interview, and a focus group discussion.

## Data analysis

Data was coded both inductively and deductively to answer the question of “To what extent did participants engage in an integrated identity exploration in Virtual City Planning as a result of exploring the roles of an environmental scientist and urban planning?” Overall, Quantitative Ethnography (Shaffer, 2017) was adopted to guide the data analyses procedures. First, researchers engaged in a deductive coding process for each case (Krippendorff, 2004) using the qualitative analysis software MAXQDA. Lines of student data were coded as illustrating self-reflection on demonstration of the four PR constructs, with the agreement reached by two graduate-level coders. Thereafter, qualitatively coded chronological data was quantified using a binary approach. Each line was coded for the occurrence (=1) or non-occurrence (= 0) of four constructs in order to prepare the data for Epistemic Network Analysis. Epistemic Network Analysis (ENA) (Shaffer, 2017), a tool that can be used with Quantitative Ethnography, was employed for modeling the structure of connections in the data to look for discernable patterns between the identity of participating students demonstrated at the start, during, and at the end of their participation in Virtual City Planning. This association structure between the changes in their knowledge, interest and valuing, self-organization and self-control and self-perception and self-definition, was modeled mathematically by creating an adjacency matrix based on their co-occurrence in the course over time, by the three data points. As such, ENA offered a unique way to recognize the patterns of identity exploration at both the group (Big I) and individual levels (Little i) for engaging in projective reflection as a result of Virtual City Planning. We referred back to the interactions and activities coded in the data to close the interpretive loop and thus fully understand the phenomenon mirrored in the model for each student and the group at large.
## Findings

Group findings are presented in this section as the themes across four constructs of PR: knowledge, interest and valuing, self-organization and self-control, and self-perception and self-definition. Then, individual findings for one case “Andrea” will be illustrated by Epistemic Network Analysis (ENA) and explained.

### Group findings

#### Change across knowledge

Based on the results of the inductive-deductive qualitative coding of student data, three thematic patterns related to student knowledge changes were identified: (1) demonstrations of core content knowledge and problem-solving and critical thinking, (2) shifts from general to specific knowledge demonstrations over time, and (3) connecting new information to existing awareness of situated context.

*Demonstrations of core content knowledge and problem-solving and critical thinking.* The majority of codes applied to student data identified instances in which students demonstrated core content knowledge gains, such as an understanding of what urban planning is and how it is conducted. These trends exist largely due to the design of the AVLE. For example, the intake survey prompted James to describe what urban planners do, to which he responded: “They collect data about the places around us and get important people to change it.” Similarly, as a result of the design of the AVLE, a large proportion of codes identified instances in which learners engaged in problem-solving and critical thinking by generating solutions to in-game problems, or by offering justifications for their design choices. For example, Alice explained her choice to add more industrial zoning to downtown Philadelphia, saying “people need more places to work and find jobs to work” which supports the affordances of AVLEs for developing meta-knowledge skills.

*Shifts from general to specific knowledge demonstrations over time.* A review of shifts in knowledge demonstrations over time within individuals reveals a general shift from broad understandings of urban planning and environmental science towards more specific recognition of the complex relationships between domain variables over time. For example, changes in Kimberly’s responses to the question “What factors might an urban planner need to consider when planning/designing a city?” reveal increased specificity and a deeper understanding of urban planning processes from engagement in the AVLE. In week 1, she wrote that urban planners “need to factor in the needs of other people” while in week 9 she was able to explain how “[urban] planners must think and fit the needs of stakeholder. Stakeholders focus on things they like to see or not like to see for specific cases.”

*Connecting new information to existing awareness of situated context.* As students demonstrated increases in urban planning and environmental science over time, they also regularly connected the urban planning problems, processes, and AVLE context (Philadelphia) to the knowledge of similar issues as they experience them in their own communities. For example, as Ali encountered the economic and environmental issues emphasized in the AVLE, he was able to reflect on the issues he had noticed in his own neighborhood (pedestrian safety). He later applied his new knowledge of how to identify problems and design solutions to his own context: “if I would want to change things, I would put more safety signs on streets to make people feel safe. my changes may help because driving lately has been more of a problem because people are frightened of others on either the highway or just a simple street.” These connections demonstrate the potential of the augmented AVLE for promoting deeper learning integrated into learners’ understandings of their own interest and values, patterns of self-organization and self-control, and self-perception and self-definition.

#### Change across interest and valuing

After the analyses of qualitative data for interest and valuing, two themes emerged: (1) change from personal interest to less personal goals and interests, (2) overlapping changes in interest and knowledge.

*Change from personal interest to less personal goals and interest.* The students’ interest and valuing changes showed a significant difference from the beginning to the end of the AVLE. Yet, the direction of the change from more personal goals and interests in the beginning weeks to less personal interest was an evident trend in all students. Deductively, many of the students’ reflections and responses to survey questions were coded personal interest in initial weeks, while similar questions toward the end of the AVLE led to more “global relevance codes” which demonstrates a shift to less personal goals when the player recognizes how tools and content related to a community or society and satisfies the needs of the group members. For example, when Andrea was prompted with the question of “How the design of Philadelphia has affected you?” in post-survey she responded: “... [environmental issues] can impact us by changing the city it can make a better environment. If we change the ecosystem to be better, it can make Philly look good and it will be healthier for us” which shows a well-suited response for “global relevance” as well as “personal relevance”. This was just one example of many responses demonstrating students’ less personal interests toward the end of the AVLE.
*Overlapping changes in interest and knowledge.* Another evident theme in students’ responses regarding interest and valuing was that change in this construct happened in concert with a change in knowledge, they demonstrated their interest using new words and expressions which were inductively coded as knowledge, too. For example, when Ellen was asked to report her team’s recommended value of bluebirds which are among endangered species in the Northeast, she answered “The recommended value of bluebirds is 2270 bluebirds. The stakeholders believe this is the perfect amount because if there were more there would be too much environmental space and not as much housing. However, if there were less than the city would not have the environmental space that it needs.” A justification which shows Ellen both has the knowledge of keeping endangered species in her surrounding environment and values people’s needs in having environmental space they need to.

## Change across self-organization and self-control

The emerged themes came out of the researchers’ memos from the students’ demonstration of self-organization and self-control: (1) Asking help from the more knowledgeable peers/stakeholder, (2) Working toward a common goal and demonstrating metaknowledge.

*Asking help from more knowledgeable peers/stakeholders.* Asking help from the knowledgeable peers/stakeholders came mainly from the researchers’ memos recording instances when students were asking for help when discussing their final map changes with their peers and stakeholders which was coded co-regulated learning. In co-regulated learning examples, the students were asking for help from more knowledgeable peers, stakeholder, and sometimes instructors.

*Working toward a common goal and demonstrating metaknowledge.* When the students were working in groups toward a common goal in addressing the stakeholders’ concerns, they were not only demonstrating socially-shared regulated learning but also problem solving, critical thinking, communication and collaboration skills which were meta-knowledge components under knowledge construct. For example, when they were asked to justify the rezoning of their map "...[zoning for business] is important because there is not a lot of space for business so we are thinking to make more space for it." First, they indicated that they are looking for a way to solve the issue (problem-solving and critical thinking), second, they were indicating that they are working toward a common goal by bringing a plural pronoun (we). This example with many other examples of this sort informed us that the students worked toward the common goal with their stakeholder, and brainstormed ideas with the members of the group to resolve the issues and rezone the maps.

## Change across self-perception and self-definition

The change in this construct was traced by coding self-efficacy, self-concept, and possible self-explored as the three main sub-constructs under it which led to one overarching theme: (1) No eloquent change in students’ self-perception and self-definition.

*No eloquent change in students’ self-perception and self-definition.* The analysis of students’ qualitative data did not show drastic changes in students’ self-perceptions and self-definition. Nearly all the students responded the same to the prompt of “what do you want to be in the future? what do you NOT want to be in the future?” before and after the AVLE. For instance, Ali said “basketball player” Ciara said “wealthy, happy, educated” Elijah said “business owner” and many other similar responses which were not changed in the post-survey. This indicates that the students’ perceptions of themselves, their interest and their future desires barely change.

## Case findings

The illustrative case example Andrea demonstrated intentional reflection and integrated change in all four areas: knowledge, interests and values, patterns of self-organization and self-control, and perceptions and definitions of self over time in three levels of starting self, possible selves explored, and new self which is summarized Table 1. Figures 3-6 illustrate the strength of PR connections in these levels and the strength is calculated from the number of coded segments using ENA. Overall, Andrea demonstrated many instances of foundational knowledge in relation to self-perception and self-definition with the hope of working in the science field and “contribut[ing] to some type of urban planning” in initial weeks of using AVLE (Figure 3). Later in the course, Andrea’s identity exploration focused more on the interest and valuing aspects as a result of engaging in the AVLE; this was integrated her emergent and detailed knowledge of urban planning (see Figure 4). Finally, a good balance between her knowledge, interest and valuing, and self-perceptions and self-definitions were revealed which is also shown in Figure 4.
<figure><img src="image_101.png" /><figcaption>**Figure 3.** Andrea's Starting Self. The connection between knowledge and self-perception and self-definition is the</figcaption></figure>

<caption>**Figure 4.** Andrea's Exploring Possible self. The connection between knowledge and interest</caption>

<caption>**Figure 5.** Andrea's new self, shows a good balance between her knowledge, interest and valuing, and self-perception and self-definition.</caption>

## Andrea's starting self

The adjacency matrix of knowledge, interest and valuing, and self-perception and self-definition in Andrea's starting self (Figure 3) during the first two weeks of the course reveal that the connection between knowledge and self-perception and self-definition constructs are stronger than others. This strength comes from her “Agree” responses to Likert-scale questions which were assessing both knowledge and self-perception and self-definitions constructs such as: “I would be able to express my view in form of a group of people” or “I would be able to create a plan that addresses the issue”. Also, the connection of these constructs with interest and valuing construct is not as week as their connection to self-organization and self-control. Andrea demonstrated interest and valuing by describing her interest in the environment “I want to help make the environment we live in clean” understanding the relevance of designing the city to her personal life “the design [of the city] affects me because of too many buildings and it gets so crowded”. She also proposed suggestions for rezoning the city by telling “we should get rid of some buildings or separate them into different areas of Philadelphia, so we can have enough room and make it feel less crowded” which demonstrated her problem-solving abilities in relation to relevance during the first two weeks of the course.

## Andrea's exploring possible self

Figure 4 shows the adjacency matrix for Andrea's exploring possible self which illustrates her change from week 3-7. Comparing to her starting self, Andrea demonstrated more knowledge and interest and valuing construct, and her adjacency matrix showed weaker connection to other constructs. Totally, knowledge and interest and valuing constructs together in all activities were coded (n=205) times which obviously have a bigger portion in Figure 4. For knowledge, she demonstrated increasingly detailed knowledge of urban planning by suggesting her detailed recommended land parcel changes “We changed the industrial areas to open wetlands so that the wetlands couldn't be polluted. WE decided that the open wetlands should be just on the wet side of the map, near the water so that the lands or the people around the area didn't pollute it and poison any of the animals or the creatures on that part of the map/area.” She capitalized “WE” which means that she is emphasizing her collaboration with other team members and socially shared learning. She also demonstrated many instances of relevance which made her adjacency matrix thicker in this construct, too: “Me as a citizen has to show how people should respect how to treat the environment. I think that our changes will affect the area in a good way and in a bad way because people will need to find more industrial areas and it will be good because the environment will be safe.” However, the link between knowledge and self-perception and self-definition is weaker as well as self-perception and self-definition and interest and valuing comparing to Figure 3 (her starting self). One explanation may be that she had a better understanding of each of these tasks because the AVLE had simulated real-life activities and allowed her to self-reflect during the program, thus, picked a less direct assertive response to the questions that were assessing those constructs.

## Andrea's new self

Figure 5 illustrates Andrea's adjacency matrix for her new self in final weeks of 8-9 which shows a good balance between her knowledge, interest and valuing, and self-perception and self-definition. This figure is well supported by qualitative themes as we discussed earlier “overlapping changes in interest and knowledge”. For the majority of students, many instances of knowledge co-occurred to the interest and valuing utterances and in some cases, they were both presents in a single utterance. For example, when Andrea was asked “Whose needs and opinions should a city plan reflect?” she responded, “it should reflect of the citizens of United States of America” compared to some other peers who had answered “people” or “citizens of the city”. This is a detailed utterance which both demonstrates a good understanding of the city planning goals and the needs of the people who live in a country asserting the fact that any decision in rezoning the city will impact the whole population of America indirectly.
This utterance was coded both for core content foundational knowledge and global relevance which was a sub-category of interest and valuing.

Analyzing Andrea's adjacency matrices from starting self to new self, and the similarity of starting self to new self, uncovered the fact that Andrea's responses to many questions of knowledge and self-perception and self-definition were unrealistic at the beginning of the program. Using AVLE, she could explore her identity and challenged some of her notions about her capabilities (see Figure 4). However, by using AVLE she could develop many aspects of the Projective Reflection constructs in a way that made it similar to her starting point. This time her responses were more realistic in the sense that she gave less affirmative but more detailed responses to the prompts which were asked initially.

<table><caption>Table 1: Summary of findings from the iteration 1 group and from an illustrative student case</caption><thead><tr><td>PR Construct</td><td>Starting self (initial reflections; Weeks 1-2)</td><td>Possible selves explored (during AVLE; week 3-7)</td><td>New self (concluding reflections; week 8-9)</td></tr></thead><tbody><tr><td>Knowledge</td><td>Described experiences with digital tools, provided general, factually correct responses to knowledge questions</td><td>Demonstrated increasingly detailed knowledge of urban planning (i.e. understanding relationships between specific design variables), and strong literacy using PLS compared to peers</td><td>Shared clear, detailed knowledge of urban planning concepts and general design process</td></tr><tr><td>Interests and valuing</td><td>Shared that she valued “making the environment we live in clean”</td><td>Expressed enjoyment of AVLE experience, regularly identified relevance of urban planning to desired future career in science</td><td>Described expanded interests in not only environmental conservation but also “the man-made items that surround it.”</td></tr><tr><td>Self-organization and self-control</td><td>Shared enjoyment of group work and confidence in her ability to complete learning tasks</td><td>Participated as a group leader - volunteered design strategies in group meetings, offered technical assistance to peers, described feedback she received and her strategies for improvement</td><td>Affirmed how her participation “helped me see what it was like being an urban planner”</td></tr><tr><td>Self-perception and self-definition</td><td>Hoped to work “in the science field” as an engineer or scientist; recognized that this future career means she would “have to [have] contributed to some type of urban planning”</td><td>Shifting towards a future career in medicine, stating “urban planning would affect me in a way because of jobs and how there would be enough places to go and study medicine.”</td><td>Hoped to be “a doctor, scientist, astronomer;” strongly agreed when asked if she could see herself as an urban planner in future</td></tr></tbody></table>

Discussion

Case findings and group themes are summarized in Table 1. Synthesis of Andrea's changes over time revealed *integrated* identity exploration through engagement with the urban planning process: the development of increasingly-specific content knowledge and AVLE literacy, repeated and diverse connections to personal interest and values and an enjoyment of the experience, active peer-to-peer leadership and the enactment of self-regulation strategies, and connections to her current and desired future roles as they emerged. As a result, Andrea was consistently able to find personal value and enjoyment in relation to her perceived self and active knowledge gaining over time. These connections demonstrate the potential of the augmented AVLE for promoting deeper learning integrated into learners' understandings of their own interest and values, patterns of self-organization and self-control, and self-perception and self-definition. As such, the AVLE, which leveraged the affordances of both virtual environments and real-world augmentations (i.e. roleplay and discussion) has the capacity to promote a comprehensive identity exploration trajectory as defined by the four PR constructs.

Group findings suggest that the AVLE experience encouraged at least some aspects of identity exploration around urban planning careers for all students – each player concluded the experience able to justify whether or not they wished to become an urban planner based on aspects of their AVLE participation. However, a review of group findings revealed that not all students enacted every aspect of PR and that many students approached identity exploration in different ways (i.e. increasing confidence versus frustration over time). These results illustrate the need for further inquiry into patterns of identity exploration so that additional supports may be designed in AVLEs to better support PR across learners with different starting self-characteristics (i.e. interests, knowledge, etc.).
**Theoretical and educational significance**

Projective Reflection (PR) serves as a theoretical lens for supporting student learning as a process of identity exploration that leads to change over time. The ability of adaptive collaboration to engage in (a) the intentional reflection on the self, (b) the process of exploring possible future selves related to new careers and roles in collaboration with stakeholders and other group members, and (c) the enactment targeted and intentional steps toward a desired new self, will serve as a particularly valuable skill for learners preparing for an evolving 21st century workforce. The development and enactment of *Virtual City Planning* offers useful insights for addressing how learning experiences may be optimally designed using a robust theoretical framework to support targeted identity exploration and change processes (DeVane, 2010), and how students’ trajectories of identity exploration and change in the same learning experience may differ (Foster, 2014). As practical and theoretical understandings of identity exploration trajectories emerge through this research, the capacity of learning practitioners to design early targeted supports for learners based on their starting self characteristics increases.

**References**

Bagley, E. A., & Shaffer, D. W. (2015). Stop talking and type: comparing virtual and face-to-face mentoring in an epistemic game. *Journal of Computer Assisted Learning, 31*(6), 606-622.

Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. S. (2016). Digital games, design, and learning: A systematic review and meta-analysis. *Review of educational research, 86*(1), 79-122.

Cobb, P., Confrey, J., DiSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. *Educational researcher, 32*(1), 9-13.

Creswell, J. W., &amp; Miller, D. L. (2000). Determining validity in qualitative inquiry. *Theory into practice, 39*(3), 124-130.

DeVane, B. M. (2010). *Toward sociocultural design tools for digital learning environments: Understanding identity in game-based learning communities*. University of Wisconsin--Madison.

Ferdig, R. E. (2006). Assessing technologies for teaching and learning: understanding the importance of technological pedagogical content knowledge. *British Journal of Educational Technology, 37*, 749-760.

Foster, A. (2014). CAREER: Projective reflection: Learning as identity exploration within games for science. *Drexel University: National Science Foundation*.

Foster, A., & Shah, M. (2016). Knew me and new me: Facilitating student identity exploration and learning through game integration. *International Journal of Gaming and Computer-Mediated Simulations (IJGCMS), 8*(3), 39-58.

Foster, A., Shah, M., & Barany, A. (2017). Facilitating learning as identity change through game-based learning. In Y. Baek (Ed.), *Game-Based Learning: Theory, Strategies and Performance Outcomes*. New York, NY. Nova Publishers.

Shah, M., Foster, A., Barany, A., Cellitti, J., Duka, M., Swiecki, Z., Evenstone, A., Kinley, H., Quigley, P., & Shaffer, D.W. (2018). Learning and Identity in Virtual Learning Environments: Iterative Design and Implementation of Philadelphia Land Science. Paper to be presented at the 1st Connected Learning Summit, MIT Media Lab, Cambridge, MA, August 1-3, 2018.

Kamarainen, A. M., Metcalf, S., Grotzer, T., &amp; Dede, C. (2015). Exploring ecosystems from the inside: how immersive multi-user virtual environments can support development of epistemologically grounded modeling practices in ecosystem science instruction. *Journal of Science Education and Technology, 24*(2-3), 148-167.

Kaplan, A., Flum, H. (2012). Identity formation in educational settings: A critical focus for education in the 21st century. *Contemporary Educational Psychology, 37*(3), 171-175.

Kaplan, A., Sinai, M., & Flum, H. (2014). Design-based interventions for promoting students’ identity exploration within the school curriculum *Motivational interventions* (pp. 243-291): Emerald Group Publishing Limited.

Krippendorff, K. (2004). *Content analysis: An introduction to its methodology*. Sage Publications.

Maxwell, J. A. (2013). *Qualitative research design: An interactive approach* (Third ed.). Thousand Oaks, Calif.: SAGE Publications.

Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1995). *Augmented reality: A class of displays on the reality-virtuality continuum*. Paper presented at the Telemanipulator and telepresence technologies.

Shaffer, D. W. (2017). *Quantitative ethnography*: Cathcart Press.

**Acknowledgments**

This material is based upon work supported by the National Science Foundation. The views expressed are those of the researchers.
# Collaborative Talk Across Two Pair-Programming Configurations

Zarifa Zakaria, Danielle Boulden, Jessica Vandenberg, Jennifer Tsan, Collin F. Lynch and Eric N. Wiebe
zzakari@ncsu.edu, dmboulde@ncsu.edu, jvanden2@ncsu.edu, jtsan@ncsu.edu, cflynch@ncsu.edu and wiebe@ncsu.edu
North Carolina State University

Kristy Elizabeth Boyer, University of Florida, keboyer@ufl.edu

**Abstract:** Given that pair programming has proved to be an effective pedagogical approach for teaching programming skills, it is now important to explore alternative collaborative configurations. One popular configuration is where dyads collaborate by sharing a single computer sitting side-by-side. However, prior research points to potential challenges for elementary students when sharing a single computer when collaborating. This prompted us to explore another configuration where dyads sit side by side but collaborate on a shared virtual platform with individual computers. We compared the discourse of students' collaboration under these two settings. Results show that although there are no significant differences in the amount of collaborative talk between the two configurations, there is qualitative evidence of how differing affordances of two configurations shape collaborative elementary students' practices.

## Introduction

A plethora of studies suggest that collaboration is an effective pedagogical approach for teaching students programming skills (e.g., Hank et al. 2011; Williams et al., 2000) Additionally, the ability to collaborate effectively helps prepare students for the future workforce (National Research Council, 2013). However, there are many alternative configurations to implement collaborative programming in classrooms. For example, dyads can each use individual computers in a two-computer pair-programming configuration or a single computer to do one-computer pair programming (Figure 1). There can be differences in the capacity of collaboration under different physical configurations. In pair programming, a common configuration used in industry is sharing one computer (Hank et al. 2011). Here, one partner takes turns as a driver and the other as a navigator of the programming environment. Empirical studies exploring the benefits of such one-computer programming shows that students or employees working together on a single computer increases enjoyment (Muller, 2006), retention rate (Carver et al., 2007) and fast problem-solving (Xu & Rajlich, 2006).

While adapting industry practices for use in educational contexts makes sense, there is reason to believe that changes in context would need to be accounted for to optimize its benefits. First, studies have shown that collaboration can be improved in terms of enjoyment, engagement, and time-efficiency if there is equality of individual control over the computer (Infante, 2009; Scott, 2003). Secondly, the traditional one-computer configuration has mostly been studied either in industry or with college students (e.g., Salleh et al., 2011). For younger students there can be issues with equity (Shah et al., 2014; Tsan et al., 2018) which may hamper productive collaborative talk, raising questions as to the appropriateness of pedagogical strategies requiring a high degree of interpersonal negotiation like turn taking.

Thus, we aim to explore a two-computer configuration where partners collaborate in a virtual workspace sitting side-by-side at individual computers. There are studies with similar configurations: collaboration with multiple mice on a shared screen (Echeverría, 2012; Infante, 2009; Scott et al., 2003), multiple touch options on shared screens such as tablets (Hsiao et al., 2014), or individual screens (Cockburn, 2005; Dewan et al., 2009). However, little is known about collaboration with individual computers sharing a virtual space in close proximity (i.e., two-computer configuration). This configuration affords individual agency with system input but allows face-to-face dialogue with a partner. Our goal is to contrast the discourse level in such two-computer with one-computer collaboration in an emerging area of programming in K-12 academics.

<figure><img src="image_102.png" /><figcaption>**Figure 1.** The One- and Two-computer Configurations in Pair Programming.</figcaption></figure>
# Theoretical framework

We focus our study on differentiating students’ collaborative discourse throughout two different configurations of collaboration. When doing so, it is important to explore constructs that consider how the differences in these configurations might shape interactional levels of collective problem-solving. Mercer proposed the concept of Intermental Development Zone (IDZ) that focuses on the nature of interactive processes between teacher and student or peer-to-peer (Mercer, 2000). It draws on both Vygotsky’s Zone of Proximal Development (ZPD) (Vygotsky, 1980) and Bruner’s Scaffolding (Wood et al., 1976), to explain productive talk between peers. In a joint activity, three types of conversations can occur which Mercer termed ‘Cumulative talk,’ ‘Disputational talk’ and ‘Exploratory talk.’ In *Cumulative talk*, speakers build positively but uncritically on what the other has said, while *Disputational talk* is characterized by disagreement and individualized decision making. In *Exploratory talk*, participants engage critically, but productively, by challenging and offering alternative hypothesis to each other’s ideas. Exploratory talk has been shown to expand the joint ZPD by enabling partners to achieve a better mutual understanding of the problem (Fernandez et al., 2015). Other research has shown that challenging viewpoints can foster conceptual change (Clark et al., 2003). As language is a prime medium of collaboration, even in computer-supported collaboration (Dillenbourg et al., 2009), it is appropriate to utilize Mercer’s framework to explore language artifacts (i.e., spoken utterances) of dyadic collaboration.

Results from this study will explore how the two configurations might facilitate computational problem-solving during programming activities by quantitatively and qualitatively comparing these configurations. We sought to contrast the conversational attributes of these two configurations by documenting the three types of collaborative talk situated in the IDZ framework. We also describe a process of translating the framework into practice in the context of an elementary level collaborative programming environment.

Guided by Mercer’s (2000) IDZ conceptual framework and prior empirical findings, we explore the following research questions: 1) What is the balance of different types of collaborative talk under different pair-programming configurations? 2) How are these types of talk different or similar across the two pair-programming configurations?

# Method

## Participants

This study is part of a design and development project investigating collaborative programming in elementary schools. Participants were from a fifth-grade classroom of a suburban school in the Southeastern United States. All the students were in an Academically and Intellectually Gifted program. We used data on 11 students (10-11 years), among whom 4 (36%) were girls and 7 (63%) were boys. In total, we had 7 dyads in one-computer and 6 dyads in two-computer configuration. Students were paired in alternative combinations. No set of two students were paired more than once. Except for one, all students participated in activities under both the configurations.

## Procedure

Students participated in problem-solving activities with a 5-day programming curriculum developed by the authors. In these activities, students used the NetsBlox block-based programming environment to develop a program by dragging blocks and combining them. All the pairs solved problems of the same difficulty level. Dyads collaborated using the two different configurations on alternating days. In the one-computer configuration, students were informed that one partner would act as the driver who would control the input devices, while the other partner, the navigator, would talk to the driver about the problems and anticipate actions. The teacher used a timer to make sure each partner had equal opportunity to be a driver and a navigator. Dyads in the two-computer configuration shared a virtual platform with individual computers.

## Data collection and analysis

The data source was video recordings of dyads collaborating on solving a programming problem. In the one-computer configuration, video recordings captured both the students’ faces, voices, and the computer screen using the Open Broadcaster Software. For the two-computer configuration, the same software recorded each students’ face, voice, and screens separately which were then merged using Adobe Premiere for analysis. Collaboration sessions lasted an average of 53 minutes. We utilized around 20 minutes of each dyad’s collaboration. Criteria for selecting a section of videos were: audio-video clarity, comparable difficulty level of the activities, and if both the partners took turns as driver and navigator in a one-computer configuration.

We analyzed the data using a mixed method approach (Creswell & Creswell, 2017). First, we coded the videos to identify the distribution of three types of talk and analyzed these distributions using non-parametric
tests. After the quantitative phase, we employed qualitative analysis to identify patterns and complexities. For the qualitative analysis, the video-recorded dialogue of all pairs and their respective schematic coding were qualitatively analyzed as a multiple case study design (Yin, 2014). To guide these analyses, we relied upon our theoretical propositions using Mercer's IDZ framework and prior literature on pair programming to aid in yielding descriptions of the contextual conditions of each thematic case. These descriptions enabled us to offer plausible explanations for the results from the quantitative phase.

## Categorization and coding

Our qualitative analysis was initially inspired by a study conducted by T'sas (2018) that focused on the three types of talk from IDZ framework to classify conversation. Guided by T'sas' articulation of the talk types, a scheme aligned with Mercer's original framing was utilized. Further piloting refined T'sas' approach by changing a few problematic sub-categories. For example, 'counter-challenge' changed to 'counter-challenge followed by consensus' and placed in Exploratory talk, while 'counter-proposition' changed to 'counter proposition with no consensus' and put in Disputational talk. Also, utterances in a time interval which did not fall into either Exploratory or Disputational category were considered as Cumulative talk given that they were characterized without any disagreement or critical exchange and thus considered as assertive, yet uncritical. The finalized coding scheme consisted of three categories: Exploratory, Cumulative and Disputational (Table 1).

A professional transcription service transcribed the video recordings for coding. We utilized a time interval coding method (Bakeman, 2000) similar to that used in some other dialogue studies (e.g., Baines et al., 2009) to code not just an utterance but an aggregated series of responses in an interval, representing a particular type of talk. Piloting determined that 10-second intervals optimized capture of unique phrases with appropriate context. Thus, we divided a 20-minute recording into 120 ten second intervals and then labeled each interval exclusively to one category. Utilizing concurrent video with the transcripts allowed us to leverage facial expressions, body postures, and gestures to interpret the transcript. After training on the coding scheme, three authors coded 50% of the videos together (average Cohen's kappa = 0.805) and then coded the other 50% individually. Coding discrepancies were then discussed until consensus was reached.

<table><caption>Table 1: Definitions of the Categories and k-agreements</caption><thead><tr><th></th><th>Exploratory</th><th>Cumulative</th><th>Disputational</th></tr></thead><tbody><tr><th>Major Characteristics</th><td>Challenge, alternative hypothesis, critical reasoning. k=0.714</td><td>Uncritical addition of ideas, Agreement. k=0.893</td><td>Disagreement without critical reasoning, k=0.803</td></tr><tr><th>Elaborated Characteristics</th><td>- Offered alternative hypothesis.<br>- Initiations challenged, and counter-challenged followed by consensus.<br>- Justifications given.<br>- Joint acceptance</td><td>-Agreement without critical discussion.<br>- Friendly and conflicts avoidance.<br>- Positively but uncritically sharing ideas.<br>- Superficial amendments.</td><td>-Disagreement without outcome.<br>- Individualized decision-making.<br>- Initiations directly rejected.<br>- No/little constructive criticism.<br>- Counter proposition with no consensus.<br>- No resolutions.</td></tr></tbody></table>

## Results and discussion

To answer our first research question, we examined the distribution of talk by quantitatively examining the coded frequencies of the three types of talk. We first investigated the overall level of discourse between the two configurations. The unit of overall quantitative analysis was students' talk per minute. We took each coded talk in 10 second intervals and aggregated them in number of different types of talk per minute. Non-parametric statistical test results indicated a non-significant difference, U=20.5, p=.943, between the two-computer configuration (66.9%) and one-computer (81.7%). The trend of a lower percentage of talk in the two-computer configuration could be explained by the fact that each student had equal autonomy in terms of computer access, limiting the need for negotiation. A Kruskal-Wallis test was used in an analysis into the distribution across the percentage of different types of talk per minute, and showed a statistically significant difference, H= 24.9, p<.001. Distribution was 12.7% Exploratory, 82.6% Cumulative and 4.7% Disputational. A series of Wilcoxon tests between pairs of each of the talk types revealed the following: Exploratory-Cumulative, Z=-3.18, p<.01, Cumulative-Disputational, Z=-3.18, p<.01, Exploratory-Disputational, Z=-2.85, p<.01. Previous literature
indicates that, generally, without any intervention, Cumulative and Disputational talk tend to be higher than Exploratory (T'sas, 2018). While no significant differences were found between the configurations for any of the types of talk, Disputational trended higher for one-computer (7.0%) than two-computer (2.6%) (Figure 2).

<figure><img src="image_103.png" /><figcaption>**Figure 2.** Percentage of each type of talk per minute, by computer configuration.</figcaption></figure>

Our second research question allowed us to explore the quality of the talk across the conditions. We qualitatively analyzed the collaborative conversations and found some salient aspects of talk which elucidate differences between collaboration in one- and two-computer configurations. From the coded talk, we found a few sub-categories like, “challenge”, “explain”, “cooperation” etc. Three themes emerged from these categories.

## Exploratory talk often preceded/followed by Disputational talk in One-computer

Results of qualitative analyses suggest that the transition between Exploratory talk and the other types of talk differed based on the configurational conditions. We provide examples to better illustrate this phenomenon.

Tom and Sandy (pseudonyms) present one example of a one-computer configuration where Exploratory talk was preceded by Disputational talk. They repeatedly disagree with one another without any explanation to each other. There are disagreements like "No" and "Nope" multiple times without trying to come to a consensus. This falls into the characteristic of Disputational talk. However, after several statements of dispute, Tom hypothesize, "that is as far as it's gonna go," whereby Sandy challenges, "How do you know?" This challenge forces him to counter her question with further explanation which becomes Exploratory talk.

One-Computer

Tom: Nope, not far enough, negative 150.

Sandy: Negative 110.

Tom: No, negative 150.

Sandy: Let's go with 10, maybe. 10?

Tom: Nope, five.

Sandy: One. So we need to go, like, negative-

Tom: Well, no, it's- it's going to X.215.

Sandy: Negative ...

Tom: That's as far as it's gonna go.

Sandy: How do you know?

Tom: Because it's a negative. You see, 'cause that's still the same number.

Two-Computer

Tony: What'd you do?

Sandy: I made, whoa, it's supposed to go right when I hit the right arrow but instead, it's like flying away.

Tony: Yeah.

Sandy: Why though?

Tony: Oh, that happens.

Sandy: Why?

Tony: I don't know if it's just 'cause we don't understand what's happening. Because whenever we did that, it'd change that too much and it'd make it. I forgot how we did it last time.

Sandy: It might have something to do with that.

<caption>**Figure 3.** Excerpts illustrating the relationship of Disputational talk with Exploratory (one-computer) or Cumulative (two-computer).</caption>

In comparison, consider these transitions between types of talk in two-computer configurations, where Cumulative talk progressed into Exploratory talk and then is followed by more Cumulative talk. In the excerpt in Figure 3, the same girl, Sandy, is paired with another boy, Tony. Here, Exploratory talk originates from Cumulative talk where Sandy requests Tony's assistance by saying "Tony, help me." This request then segues into a repetitive pattern of Exploratory phrases comprising a series of stimulating questions when each student responds to their partner's inquiry using several 'why' questions and 'because' explanations.

These examples are demonstrative of a divergent sequential form for these two conditions, where in the one-computer configuration Exploratory talk transitioned into Disputational talk while in the two-computer configuration Exploratory talk transitioned back into Cumulative talk. A possible explanation may be that because students in two-computer have equal access to their own devices, allowing them agency to express their ideas directly without requiring negotiation, the likelihood of control-based dispute is less likely to ensue.

Past research on student conflict during collaborative activities has demonstrated that this is a naturally occurring phenomenon (Jeong, 2008; Rubin, Pruitt, & Kim, 1994), and can be beneficial or harmful to outcomes
depending on its characteristics. For example, Lee, Huh, and Reigegluth (2015) found task-related conflict to have a positive influence on peer collaboration as students are forced to self-reflect and build upon one another's ideas. Whereas they contend that when students are having disputes about the collaboration process itself or interpersonal conflicts, it can be detrimental to productive collaboration. There is concern that the one-computer configuration is perhaps amplifying the potential for unproductive disputes over process or based on interpersonal dynamics. The two-computer configuration still requires negotiation over the final programming artifact but removes the need for sharing input devices, perhaps striking a better balance of agency and control.

## Balance of challenger and explainer in Exploratory talk

In the one-computer configuration, we found evidence that one learner may end up challenging their partner more by asking questions and demanding justification for any proposals or claims; we consider this type of learner to be a *challenger*. In the excerpts below, we can see the other partner took the role of an *explainer* and ended up justifying what she proposed or edited. As challenging is identified as a major characteristic of Exploratory talk, this pairing of *challenger* and *explainer* was found in this type of talk. Interestingly, among the excerpts, where this combination was found, it was the driver (who had control of the input) who played the role of challenger. With Steve driving and Melony navigating, Steve was more of a challenger, asking more questions to Melony who was answering them. In one instance, they were discussing deleting a block and Steve (driver) was constantly challenging Melony's idea with questions such as "Okay. So, what?", "How do you have two layers?", "Like, does it still work?", "Okay, So what?" one after another (see Figure 4). This tendency of the driver to be more challenging was found in other dyads in the one-computer configuration too. In another pair, Sandy, the driver, was challenging Anthony's ideas with a lot of 'why' and 'how' questions. Previous research suggests that 'why' and 'how' are Exploratory key words for being counter challenging and 'because', 'but', 'what if' are Exploratory keywords for explanation (Polo et al., 2015; Wegerif et al., 2005).

<table><caption>Figure 4. Excerpts illustrating the balance of challenger and explainer in Exploratory talk.</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td><strong>One-Computer</strong><br><u>Steve</u>: Okay. So, what?<br><u>Melony</u>: And so, like here, wouldn't you do, um..<br><u>Steve</u>: How do you have two layers?<br><u>Melony</u>: I don't know.<br><u>Melony</u>: Maybe you've to put wind direction off.<br><u>Steve</u>: Like, does it still work?<br><u>Melony</u>: Yeah, it works, but like ...<br><u>Steve</u>: Okay. So, what?<br><u>Melony</u>: And so, like here, wouldn't you do, um..</td><td><strong>Two-Computer</strong><br><u>Tony</u>: Because it was like this, when we pressed up arrow<br>... oh wait he has a...<br><u>Steve</u>: But I think we're actually connected these, when you<br>press it this goes to-<br><u>Tony</u>: What happen if would go down more than he would<br>go up?<br><u>Steve</u>: He doesn't even go down though.</td></tr></tbody></table>

In contrast, in the two-computer configuration, there was evidence that showed both the partners taking the roles of challenger and explainer in a more interleaved fashion. In Figure 4, Tony and Steve are found to be continuously challenging and explaining to each other with words such as ‘but,’ ‘because,’ ‘what happens if,’ ‘even though’ etc. Another pair, Luke and Clara were similarly questioning each other’s ideas and demanding an explanation. In one instance, Clara was suggesting an ‘if/else’ block as well as challenging Luke’s idea of inputting a ‘stop’ block. Luke too, was simultaneously proposing the opposite idea and kept questioning Clara.

This differing combination of *challenger* and *explainer* suggests that this relationship is not always equitable with the challenger playing more of an authoritative role in the pair. Researchers have strived to analyze and understand the equity of collaborative relationships in many subjects, including computer science (e.g., Deitrick et al., 2016; Shah et al., 2014; Stanton et al., 2002). While we are still unsure of the effects of an inequitable relationship in terms of the challengers and explainers, we hypothesize that collaboration is productive if the dyads have balanced, interleaved challenging and explaining interactions.

## Cooperative work rather than collaborative work in two-computer configuration

In the two-computer configuration, both partners have access to a laptop, thus they get the opportunity to edit individually. This increase possibilities of getting involved in cooperative work rather than collaborative work, which has a clear distinction in terms of the socio-cultural learning mechanism (Davidson, 2014; Oxford, 1997). Kagan and Kagan (2009) identify a pair of critical attributes of cooperative learning: interdependence and accountability. Here, students can work in separate sections of the project, yet still simultaneously seek help from each other. For example, Rupert and Melony divided the work between themselves by saying “Yeah, we need to.”
I'll do the top one, you can do the bottom one," indicating different code blocks. However, they both sought help when they came upon problems. Melony, facing difficulty, asked "In this one? And then the other way around?". Rupert then assisted her, answering "And then you can put negative 85 in the second one." This pattern of interdependence then continued. Figure 5 shows another prominent example, where Sandy and Tony start cooperating, Sandy responds to a teacher saying, "I'm gonna work on the sprite getting it to move back and forth and he's gonna work on the different enemies." However, after a few minutes of cooperative work, they start going back and forth to each other's section, hovering over their partner's computer to help them edit their sections. While doing cooperation, they were more involved in Cumulative talk. But when they started collaboration on same section of the code, more Exploratory instances were observed.

<table><caption>**Figure 5.** Excerpts illustrating the individual vs. collaborative talk in two-computer configuration.</caption><thead><tr><th>Two-Computer (Cooperative)</th><th>Two-Computer (Collaborative)</th></tr></thead><tbody><tr><td><p><strong>Sandy:</strong> You work on the ball, I'll work on the platforms.</p><p><strong>Tony:</strong> Let me just duplicate this last group and then add it.</p><p><strong>Sandy:</strong> Wait, I'll work on the ball actually, you keep going the enemies.</p><p><strong>Tony:</strong> Okay, let me fix. okay, so, he shouldn't be doing this forever now. Let me check.</p><p><strong>Tony:</strong> Yay, I've got enemy one. But now I have enemy two doing the exact same thing.</p></td><td><p><strong>Sandy:</strong> Broadcast. We're gonna need some sort of broadcast in there.</p><p><strong>Melony:</strong> Okay, broadcast...Scissors...</p><p><strong>Sandy:</strong> Oof! Hello. Oof! Why is it repeating that? Oh, no, what did we do? We broke it! We broke it. We broke it.</p><p><strong>Melony:</strong> Don't worry, I know why.</p><p><strong>Sandy:</strong> Is it flashing for you, too?</p><p><strong>Melony:</strong> Yeah. I put it in forever. There. Oh, no. There we go. Now it'll only say it once. See?</p><p><strong>Sandy:</strong> But then it just disappears. What did you do, Melony?</p><p><strong>Melony:</strong> What?</p></td></tr></tbody></table>

In the two-computer configuration, we also found both the students working on the same sprites but different blocks. The collaborative excerpt in Figure 5 shows Sandy and Melony working on the same section of code, challenging and helping each other as they work through it. While, as we see above that the two-computer collaboration could devolve into cooperative work, dyads seldom did work completely by themselves, rather they often verbalize what they were doing and sought help. Thus, these interactions are best characterized as moving along a collaborative-cooperative continuum. Relevant literature supports the idea that once each student has their own input device, they tend to cooperate rather than collaborate (Stanton et al., 2002).

Cooperative learning has been found to be an effective learning strategy to enhance achievement (McConnell, 2014) and thus a reason for exploration in pair-programming configurations. For future research we would explore productive outcomes of pair-programming with cooperative or collaborative interactions.

## Conclusion

Guided by research informed by Mercer's (2000) framework, we believe helping students to productively collaborate with a higher percentage of Exploratory talk is a laudable goal when designing a pair-programming platform for elementary students. Exploratory talk offers potential for learning where evidence was found to extend students' learning within the ZPD (Littleton, 2005). Our findings that Exploratory talk occurred much less frequently than Cumulative were congruent with past studies (Bungnum, 2018; T'sas, 2018), which showed that without any intervention most conversation generally fall into positive but uncritical utterances, characteristic of Cumulative talk. However, our focus will be to use a combination of system and teacher level supports that enhance the quality and quantity of productive, Exploratory talk.

While there were no significant quantitative differences in the three types of talk between the configurations, qualitative analysis revealed interesting patterns that possibly point to how one and two-computer configurations shape collaborative work by students of this age. The one-computer configuration that provided direct control of editing code to only one student seemed to create an imbalance in an agency with regards to both proposing and evaluating potential solution paths. Evidence pointed to challenging ideas residing primarily with the driver, with the navigator left defending their ideas. We wonder if this is related to the pattern of Disputational talk proceeding Exploratory talk more often in the one-computer configuration. This result was not surprising because the driver is the learner that has control over the computer, thus control over the changes in the code. While we believe that the inequity of the challenger-explainer interaction may be due to the nature of the pair-programming roles, we plan to conduct statistical comparison to verify the differences between the two configurations. Our goal will be to develop interventions and adaptive support to improve the equity of these types of interactions. Examples of interventions include collaboration trainings to educate students on how to challenge their partner's idea and having intelligent collaboration agents that can urge both the students to get involved in challenging and explaining. Of course, one way to limit conflict over editing the program is to provide both students with input control. However, that configuration has the risk of dissipating the collaborative relationship,
as evidence suggested that collaboration in two-computer configuration has a possibility of turning into cooperation. That said, our findings point to a healthy cooperative interdependence of the students in the two-computer configuration, as evidenced by the interleaving of challenger-explainer roles and regular consultation when working on separate codes.

In future research, we will continue to examine and operationalize Mercer's framework for measuring and analyzing dialogue in pair programming. We want to explore Cumulative talk more exhaustively to identify if there are methods of disaggregation of the Cumulative category that are parsimonious with theory and useful empirically to our research and development. Larger scale data collection with more students will also allow us to reach more conclusions about the quantity of both Exploratory and Disputational talk between the two conditions. These findings also have important implications for our future design efforts on systems for pair programming where virtual agents could be used to facilitate the use of Exploratory talk.

## Limitations

Our study consisted of a relatively small, non-representative sample of students, limiting our statistical power and ability to generalize findings. Second, we were unable to analyze partner differences by condition due to instructional and logistical limitations. Relatedly, our analysis did not attempt to account for individual differences (e.g., prior programming knowledge, gender, etc.) which might have impacted dyad's collaborative dynamics. Future studies are planned to address these limitations.

## References

Bakeman, R. (2000). Behavioral observation and coding. In H. T. Reis & C. M. Judd (Eds.), *Handbook of research methods in social and personality psychology* (pp. 138-159). New York, NY: Cambridge University Press.

Baines, E., Rubie-Davies, C., & Blatchford, P. (2009). Improving pupil group work interaction and dialogue in primary classrooms: results from a year-long intervention study. *Cambridge Journal of Education, 39*(1), 95-117.

Bungum, B., Bøe, M. V., & Henriksen, E. K. (2018). Quantum talk: How small-group discussions may enhance students' understanding in quantum physics. *Science Education, 102*(4), 856-877.

Carver, J. C., Henderson, L., He, L., Hodges, J., & Reese, D. (2007, July). Increased retention of early computer science and software engineering students using pair programming. In *Software Engineering Education & Training, 2007. CSEET'07*. (pp. 115-122). IEEE.

Clark, A. M., Anderson, R. C., Kuo, L. J., Kim, I. H., Archodidou, A., & Nguyen-Jahiel, K. (2003). Collaborative reasoning: Expanding ways for children to talk and think in school. *Educational Psychology Review, 15*(2), 181-198.

Cockburn, A. (2005). Crystal Clear. Addison-Wesley.

Creswell, J. W., & Creswell, J. D. (2017). *Research design: Qualitative, quantitative, and mixed methods approaches*. Thousand Oaks, CA: Sage.

Davidson, N., & Major, C. H. (2014). Boundary crossings: Cooperative learning, collaborative learning, and problem-based learning. *Journal on Excellence in College Teaching, 25*.

Deitrick, E., Shapiro, R. B., & Gravel, B. (2016). How Do We Assess Equity in Programming Pairs? In Looi, C. K., Polman, J. L., Cress, U., and Reimann, P. (Eds.). *Transforming Learning, Empowering Learners: The International Conference of the Learning Sciences (ICLS) 2016, Volume 1*. Singapore: International Society of the Learning Sciences.

Dewan, P., Agarwal, P., Shroff, G., & Hegde, R. (2009, May). Distributed side-by-side programming. In *Proceedings of the 2009 ICSE workshop on cooperative and human aspects on software engineering* (pp. 48-55). IEEE Computer Society.

Dillenbourg P., Järvelä S., Fischer F. (2009) The Evolution of Research on Computer-Supported Collaborative Learning. In: Balacheff N., Ludvigsen S., de Jong T., Lazonder A., Barnes S. (eds) *Technology-Enhanced Learning*. Springer, Dordrecht.

Echeverría, A., Améstica, M., Gil, F., Nussbaum, M., Barrios, E., & Leclerc, S. (2012). Exploring different technological platforms for supporting co-located collaborative games in the classroom. *Computers in Human Behavior, 28*(4), 1170-1177.

Fernández, M., Wegerif, R., Mercer, N., & Rojas-Drummond, S. (2001). Re-conceptualizing "Scaffolding" and the Zone of Proximal Development in the Context of Symmetrical Collaborative Learning. *The Journal of Classroom Interaction, 36/37*(2/1), 40-54.

Hanks, B., Fitzgerald, S., McCauley, R., Murphy, L., & Zander, C. (2011). Pair programming in education: a literature review. *Computer Science Education, 21*(2), 135-173.
Hsiao, H. S., Chang, C. S., Lin, C. Y., Chang, C. C., & Chen, J. C. (2014). The influence of collaborative learning games within different devices on student's learning performance and behaviours. *Australasian Journal of Educational Technology, 30*(6).

Infante, C., Hidalgo, P., Nussbaum, M., Alarcón, R., & Gottlieb, A. (2009). Multiple Mice based collaborative one-to-one learning. *Computers & Education, 53*(2), 393-401.

Jeong, H. W. (2008). *Understanding conflict and conflict analysis*. Thousand Oaks, CA: Sage.

Kagan, S. (2009). *Kagan's cooperative learning*. San Clemente, CA: Kagan Publishing.

Mercer, N. (2000). *Words and minds: How we use language to think together*. New York, NY: Routledge.

Müller, M. M. (2006). A preliminary study on the impact of a pair design phase on pair programming and solo programming. *Information and Software Technology, 48*(5), 335-344.

Lee, D., Huh, Y., & Reigeluth, C. M. (2015). Collaboration, intragroup conflict, and social skills in project-based learning. *Instructional Science, 43*(5), 561-590.

Littleton, K., Mercer, N., Dawes, L., Wegerif, R., Rowe, D., & Sams, C. (2005). Talking and thinking together at Key Stage 1. *Early Years, 25*(2), 167-182.

Oxford, R. L. (1997). Cooperative learning, collaborative learning, and interaction: Three communicative strands in the language classroom. *The Modern Language Journal, 81*(4), 443-456.

McConnell, D. (2014). Implementing computing supported cooperative learning. Routledge.

National Research Council. (2013). *Education for life and work: Developing transferable knowledge and skills in the 21st century*. Washington, DC: National Academies Press.

Polo, C., Lund, K., Plantin, C., & Niccolai, G. (2015). *Analyzing Exploratory Talk as a Socio-Cognitive Practice: Identity, Group Argumentation, and Class Debate Quality*. (hal-01208319)

Rubin, J. Z., Pruitt, D. G., & Kim, S. H. (1994). *Social conflict: Escalation, stalemate, and settlement*. New York, NY: McGraw-Hill.

Salleh, N., Mendes, E., & Grundy, J. (2011). Empirical studies of pair programming for CS/SE teaching in higher education: A systematic literature review. *IEEE Transactions on Software Engineering, 37*(4), 509-525.

Scott, S. D., Mandryk, R. L., & Inkpen, K. M. (2003). Understanding children's collaborative interactions in shared environments. *Journal of Computer Assisted Learning, 19*(2), 220-228.

Shah, N., Lewis, C., & Caires, R. (2014). Analyzing equity in collaborative learning situations: A comparative case study in elementary computer science. In *Proceedings of the International Conference of the Learning Sciences*, 495-502. International Society of the Learning Sciences. Boulder, CO.

Stanton, D., Neale, H., & Bayon, V. (2002, January). Interfaces to support children's co-present collaboration: multiple mice and tangible technologies. In *Proceedings of the Conference on Computer Support for Collaborative Learning: Foundations for a CSCL Community*, 342-351. International Society of the Learning Sciences.

Tsan, J., Lynch, C. F., & Boyer, K. E. (2018). “Alright, what do we need?”: A study of young coders’ collaborative dialogue. *International Journal of Child-Computer Interaction, 17*, 61-71, ISSN 2212-8689.

T'Sas, J. (2018). *Learning outcomes of exploratory talk in collaborative activities* (Doctoral dissertation, University of Antwerp).

Vygotsky, L. S. (1980). *Mind in society: The development of higher psychological processes*. Harvard University Press. Cambridge, Massachusetts.

Wegerif, R., Linares, J., Rojas-Drummond, S., Mercer, N., & Velez, M. (2005). Thinking Together in the UK and Mexico: Transfer of an Educational Innovation. *The Journal of Classroom Interaction, 40*(1), 40-48.

Williams, L., Kessler, R. R., Cunningham, W., & Jeffries, R. (2000). Strengthening the case for pair programming. *IEEE Software, 17*(4), 19-25.

Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. *Journal of Child Psychology and Psychiatry, 17*(2), 89-100.

Xu, S., & Rajlich, V. (2006, July). Empirical validation of test-driven pair programming in game development. In *Computer and Information Science, 2006 and 2006 1st IEEE/ACIS International Workshop on Component-Based Software Engineering* (pp. 500-505). IEEE.

Yin, R. K. (2017). *Case study research and applications: Design and methods*. Thousand Oaks, CA: Sage.

## Acknowledgements

This work is supported in part by the National Science Foundation (NSF) through the grant DRL-1721000.

Everything expressed in this material are those of the authors and do not necessarily reflect the views of NSF.
CoProStory: A Tangible Programming Tool for Children's Collaboration

Xiaozhou Deng, Danli Wang, and Qiao Jin

dengxiaozhou2018@ia.ac.cn, danli.wang@ia.ac.cn, jinqiao2018@ia.ac.cn

The State Key Laboratory of Management and Control for Complex Systems,
Institute of Automation, Chinese Academy of Sciences, Beijing, China

School of Computer and Control

Engineering, University of Chinese Academy of Sciences, Beijing, China

**ABSTRACT:** In this paper, we present CoProStory, a collaborative tangible programming tool to support children learning programming. In recent years, many collaborative tangible programming tools were developed for children. Compared to the other tools, Sync Blocks which were designed to coordinate children's programming were used in our tool. The tool consists of two parts: tangible programming blocks and game tasks. Game tasks are provided with 3D cartoon story scenes on computer screen and need to be finished by collaborative programming with our programming blocks. To evaluate our tool, a contrast user study was conducted with 28 children aged 6 to 10. The results demonstrated that CoProStory could support children to program collaboratively and children had a more positive attitude toward programming with Sync Blocks.

Introduction

Children programming has been a wide field of research since 1960s. Research has shown that programming education can have a positive and measurable effect on children's achievement, not only in areas such as math and science, but also in language skills, creativity, and social emotional interaction (Douglas, 1999). Besides, learning programming is an efficient way to cultivate computational thinking which has been described as a fundamental skill for everyone, not only for computer scientists (Jeannette, 2006).

However, programming appears to be quite a challenge for young children. With the traditional way of programming, children have difficulties not only in learning rigid syntax and text symbols, but also in using the complex programming environment (Andy & Andrew, 2007). Therefore, lowering the difficulty of programming and offering children proper programming tools are highly valuable in education. TUI is an efficient interaction method embracing the richness of human's interaction with the physical world, which contributes to children's learning (Michael & Robert, 2007). With tangible programming tools, children can write programs by assembling the physical objects without keystrokes, which is much easier to involve children in programming (Michael & R.Jordan et al, 2012; Timothy, 2004). Besides, TUI has the advantages of involving multiple children in the same process, as TUI can easily provide an open and shared environment among children (Hiroshi & Brygg, 1997).

Face-to-face collaboration with classmates or friends is an important part of children's daily lives. Collaboration provides children with an efficient way to improve social communication skills. It helps children analyze information from the perspectives of other users when communicating face-to-face (Regan & Kori et al, 2001). Collaborative programming can promote the awareness of the teamwork, and the skills of communicating and collaborating (Scott, Regan & Kori, 2003). However, for children, if the tasks are not assigned well, the inequality of task loads can result in significant decrease of children's practice time, interests and concentrations. And, research has indicated that tangible user interfaces could provide broad and shared interaction environments, which have the potential to involve multiple children at the same time and support face-to-face collaboration (Leslie, 2007). Furthermore, research implicated that collaborative technology should support concurrent interaction which can help to engage children in a collaborative activity and enable them to participate equally (Neha & Laurie et al, 2004; Charlie & Linda et al, 2002).

Based on the analysis above, we applied the concept of process synchronization and developed CoProStory, a tangible programming system supporting children's collaboration in programming (Figure 1). Compared with previous works, CoProStory allows two children to program their own characters with programming blocks in parallel, which aims to complete a common task. In this method, with clear role division, conflicts could be well reduced. Furthermore, Sync Blocks are provided to coordinate children's collaboration.
<figure><img src="image_104.png" /><figcaption>Figure 1. Children playing with CoProStory.</figcaption></figure>

## Related work

One of the earliest tangible programming projects that support collaboration is AlgoBlock (Hideyuki & Hiroshi, 1995). Children write their own programs to play a marine game by connecting the objects. In their user study, the authors proved AlgoBlock's value in improving the communication among children by recording and analyzing their speech and activities. Tangicons 3.0 is an educational game designed for children age 6 - 9 (Florian & Thomas et al, 2012). It provides a collaboration environment by allowing at most four children to program together to move virtual characters on a map. However, its programming blocks are built based on Sifteo cubes, which increase the cost of the tool. Digital Dream Lab (DDL) is a tangible storytelling system for children to learn programming concepts. DDL focused on the concepts of variables and classes by using the metaphor provided by irregular puzzle pieces (Hyunjoo & Anisha, 2013). The system has six kinds of blocks, including Character Block, Animation Block, Color Block, Size Block, Variable Block and Background Block. However, the tool was also designed for older children (7-12 years old) and younger children are short in collaboration compared to those of older children. Plugramming was designed to support young children's face-to-face collaborative programming. However, in this tool tasks are divided into two separated branches of a program (Tomohito & Yasushi et al, 2017).

From the works above, we find the open characteristics of TUIs can effectively support face-to-face collaboration between children. Based on previous work, we designed and developed a new tangible programming system, CoProStory, which allows two children play and program face-to-face. With clear role division and Sync Blocks, CoProStory could support two children to program concurrently and collaboratively.

## Design and implement

Our design and implement process involved two major revisions in which we tried to explore how to encourage children's collaboration better. The difference between the two versions is that one is designed to be completed with Sync Blocks while the other isn't. The details are described as follows:

### Programming blocks

Programming blocks are small square boxes (6cm*6cm*6cm) covered with color figures representing certain semantics or manipulation constraints (Figure 2). Programming blocks are the primary tools for children interacting with the animation game running on the computer.

Blocks with RFID reader are very flexible, and are suitable for parameters with a lot of optional values. In these blocks, a RFID reader was fixed beneath the top surface. And with a shallow groove on its top surface, a mini Interactive Element Card (IECard) could be embedded, together making up a complete semantic. IECards are made up with small circular RFID tags covered with figures, which could represent different interactive elements in game scenes, such as a tree, a gate and even a character. And with infrared sensor fixed inside, the message of related position could be provided.

Programming blocks can be divided into four categories: Character Blocks, Attribute Blocks, Animation Blocks and Sync Blocks. Children could use Character Blocks and Attribute Blocks to initialize their characters, which provides with clear role division. Since Attribute Blocks provide children with individual changes to visual character, it may be easier to make children immerse themselves to our game. Animation Blocks are used
as commands to program the characters' actions. With different kinds of IECards, children could use Animation Blocks to program the visual character finishing the given tasks. Most importantly, we design a special type of blocks, Sync Blocks, to coordinate children's collaboration.

Sync Blocks are designed to encourage children’s collaboration in the game. Three kinds of Sync Blocks are currently provided: Wait Blocks, Notice Blocks and WaitForSeconds Blocks. Wait Blocks and Notice Blocks are always used in our programs. Once it comes to a wait command, the corresponding character would stand there and keep waiting until another character comes to a notice command. To use the Sync Blocks properly, children need to decompose the given task together before programming and figure out a solution with provided blocks.

<figure><img src="image_105.png" /><figcaption>**Figure 2.** Programming Blocks & Interactive Element Cards.</figcaption></figure>

Sync Blocks help children to keep in the same speed to some extent. If children ignore their partner's process, the real-time feedback would provide corresponding error messages. For instant, one character needs to cross a river but he should wait for the other to drop the suspension bridge and children should use Sync Blocks to complete these moves.

## Visual games

We provide two versions of games which need different degrees of collaboration. Specifically, the work in the first version is designed without Sync Blocks.

In the game of version1, children should use programming blocks to move their characters to the given destination. One of characters needs to go to the tree from the house while the others direction is opposite. During this process they should make their characters finish certain movements such as opening an umbrella or striding over a rock in order. They need to program the correct sequences without Sync Blocks.

In the game of version2, children need to finish their task with Sync Blocks. After several movements, one character should wait for the other dropping down the suspension bridge by a series of operations to cross the river. After the character got cross the river, the other character should lift the suspension bridge again. In this story scene, two characters' movements are time-related, as a result, children need to use Sync Blocks to finish this given task.

<figure><img src="image_106.png" /><figcaption>**Figure 3.** Different versions of game tasks.</figcaption></figure>
Programming process

The whole process of learning consists of three stages: initialization stage, programming stage and running stage. Children would use Character Blocks and Attribute Blocks to initialize their characters. Then simple background stories would be told to children which contains game task. After children get clear with their goals, they are asked to program characters to complete given tasks. During this programming stage, the real time feedback showed in figure 4 would help children and reduce their cognitive load of programming. Specifically, there would be states messages on the screen which could show children their progress in real time. And if children place wrong blocks, there would be error messages to help children debug their program. After children finish their program, the running state would show the result of children's programming with animation. We use these states to introduce children the normal process of programming.

To evaluate our tool, we conducted a contrast user study and compare these two versions to find the
benefits of Sync Blocks.

<figure><img src="image_107.png" /><figcaption>Figure 4. Programming state and running state.</figcaption></figure>

User study

Participants and setting

28 children (13 girls and 15 boys) aged 6 to 10 with a mean age of 7.79 were involved in our user study. To assess the collaborative design of CoProStory, 28 children were divided into two groups evenly: G1 (7 girls and 7 boys with mean age of 7.5) and G2 (6 girls and 8 boys with mean age of 8.14). Children in G1 were experimented with Version1 of CoProStory and children in G2 experimented with Version2. Experiments were conducted in a spacious room. In order to capture children’s behaviors and voice, a video recorder was set up.

Experimental procedure

The experimental procedure mainly consists of three parts: introduction, programming and interview. At the beginning, we introduced CoProStory to children (5 minutes). After that, children were asked to complete the programming tasks in pairs (30 minutes) and the interactions, conversations, utterances and time they spent on each task would be recorded with cameras. Next, we made a brief interview for children (10 minutes). Altogether, one experiment takes one regular school lesson (45 minutes).

Measures

In ordered to find the advantages of the design of Sync Blocks with quantitative data, we conducted quantitative coding and analysis of video data of the participants' behaviors act by act using Bale's Interaction Process Analysis coding scheme (Robert, 1950). And in order to raise measurement accuracy, each video data would be analyzed by two coders. Firstly, the two coders would learn the coding schema and coding rules, then coded 2 groups of video data independently. Afterward, they discussed their disagreements, adjusted their mismatched codes and then coded the remaining data separately.

According to Bale's Interaction Process Analysis coding scheme, we used these four kinds of codes to evaluate the interaction of children while programming: 1) positive reactions, 2) negative reactions, 3) initiative helping behaviors, 4) behaviors of asking for assistance. With the quantitative data of these codes, we could measure the collaboration of children and compare the results of the two groups to find the positive effects of Sync Blocks. Table1 is the description of each code.
<table><caption>Table 1: Coding scheme used for our analysis of the recorded video</caption><thead><tr><td>Code</td><td>Description</td></tr></thead><tbody><tr><td>C 1</td><td>Positive Reaction, including 4 kinds of behaviors: task-related conversation, task-related respond, passing blocks and correct partner's errors.</td></tr><tr><td>C 2</td><td>Negative Reactions including refuse to help/answer and ignore partner's task-related talk and request.</td></tr><tr><td>C3</td><td>Take the initiative to provide help and answer task-related questions.</td></tr><tr><td>C4</td><td>Ask for help and ask task-related questions.</td></tr></tbody></table>

And we made a brief interview after children programmed. The interview questionnaire is a Likert-type scale composed of 8 questions scored from one to five, where one means the minimum and five means the max score. Q1-Q4 concerned about CoProStory's interests, ease to use and learn. Q5-Q8 are used to evaluate how CoProStory support children's collaboration. Questions are as follows:

* Q1: How much do you like the game?

* Q2: Do you think the game is easy to learn?

* Q3: Do you think it is easy to control roles in the game by blocks?

* Q4: Was the real-time feedback in the game helpful in the programming stage?

* Q5: How much do you like playing the game with your partner?

* Q6: Do you pay attention to your partner when you were playing the game?

* Q7: How much help did you offer your partner?

* Q8: How much help did your partner offer you?

## Results

We analyzed the quantitative data of four codes combined with video data. The quantitative coding results are shown in Figure 5:

<figure><img src="image_108.png" /><figcaption>Figure 5. Quantitative coding results of G1 (blue) and G2 (red).</figcaption></figure>

For interactive behaviors, the average positive data (C1) of G1 and G2 are 8.71(SD=4.40) and 10.57(SD=3.74), which indicates that CoProStory's design affects participants' collaborative behaviors and children would show more positive reactions in version 2. The average negative data (C2) are 0.14(SD=0.35) and 0(SD=0). The negative data are both low, which indicates that children would unlikely show negative reactions while programming with CoProStory.
For helping behaviors (C3), children who used G1 showed relatively fewer helping behaviors (M=3.29, SD=3.1) compared to children who used G2 (M=5.29, SD=2.12). And for C4, children in G2 were more likely to ask for help (M=0.86, SD=0.99) than G1 (M=1.71, SD=2.19). The results of G2 were relatively higher than G1 which means the design of Sync Blocks can promote children's helping behaviors. Combined video data, we can find that children are more likely to help their partners while programming with Sync Blocks. Here is an example in G2:

**Children1:** Now I need the suspension bridge down to continue. Using Wait Blocks to wait...where do you get? I need you place the Notice Blocks.

**Children2:** I haven't got there. And I need the climb Animation Block.

**Children1:** Here is your Block. What's next?

**Children2:** Let's see...

As the example shows, children in G2 were more likely to pay attention to their partners with Sync Blocks while programming. By contrast, according to the video data, children in G1 often offered initiative help until they finished their own tasks. Here is an example in G1:

**Children1:** I have finished my part!

**Children2:** I still need...

**Children1:** It's easy. Let me help you.

The results of our interview are shown in Figure 6. Overall, scores of interviews show that CoProStory could support children programming collaboratively.

<figure><img src="image_109.png" /><figcaption>**Figure 6.** Scores of Q1-Q8 (blue columns are scores of G1, red columns are scores of G2).</figcaption></figure>

The result of the interview shows that CoProStory is interesting, easy to use for children for the reason that children in both G1 and G2 gave CoProStory average 4.5(SD=0.63). But the scores of Q2, both G1 and G2 result in 3.57, indicate that we may make the semantics of the blocks clearer to reduce the cognitive load. As for Q3, the average scores (4.14 with SD=0.64 and 3.93 with SD=1.03) of two groups were positive which means children thought it is easy to use programming blocks to control the characters. Q4 is about whether the real-time feedback in the game helpful for children in the programming stage. Children in different groups gave average scores of 3.79(SD=1.08) and 4.07(SD=0.46). The result shows a positive evaluation of CoProStory's real-time feedback and indicates that the real-time feedback was helpful for children to complete the tasks.

In Q5, children were asked how much they like playing the game with their partner, they gave the question average 4.64 (SD=0.48) and 4.71(SD=0.45). Both groups gave high scores. In Q6, we asked the children if they paid attention to their partner when they were playing the game. They gave the question average 3.79 (G1, SD=0.86) and 4.07(G2, SD=0.59). A relative higher score of G2 indicates that children may pay more attention to
their partner with Sync Blocks. In Q7, we investigated how much help did they offer their partner, children gave an average of 3.71 (G1, SD=1.03) and 4.1(G2, SD=0.52). Children also gave an average of 3.71(G1, SD=1.16) and 4.21(G2, SD=0.67) for Q8-investigating how much help their partner offered them and the difference between G1 and G2 indicates that Sync Blocks may lead children help each other.

## Discussion

CoProStory is a new tangible programming tool for children's collaboration and the design of Sync Blocks is to promote collaboration and avoid inequality of loads of programming. According to the results of user study, children were more engaging and showed more positive reactions while programming with Sync Blocks. But as the results of interview shows, there isn't significant difference in the scores of interests between these 2 versions and children gave lower scores of eases to learn in G2. The reason may be that the new Sync Blocks also increase the cognitive load of programming while coordinating children's programming.

To avoid inequality of loads, we make clear role division and assign tasks to children as expected. However, the separation may cause less collaboration and the difference of programming speed may still cause inequality, which is reflected in G1: children in G1 often offered initiative help until they finished their own tasks and as consequence the children got help might take less loads. To avoid inequality and promote collaboration, we use Sync Blocks to make the separated tasks time-related and coordinate children's programming. And based on our experiment, we think it's necessary to classify the help behaviors and find new measurement of inequality to get a better comparison. Besides, the design of visual games with Sync Blocks needs further consideration. For instance, the positions of time-related nodes affect children's collaboration and the method of nodes assignment needs further research.

## Conclusion

This paper described CoProStory, a brand-new tangible programming tool which could encourage collaboration between children. The tool supports children to program together in the same physical environment while work on a shared quest. Besides, with the comparison of the two versions, we found the advantage of the design of Sync Blocks.

We try to applied the concept of process synchronization to promote children's collaboration. As described, we design Sync Blocks to coordinate children's programming and made a contrast user study to explore the better way to support children's collaboration better. And as the results of our experiment show, children were more positive in version 2. In particular, some children who program faster usually would help their partners if their program was blocked by Wait Blocks due to the turning of attention to their partners. Besides, children would observe other programming while using Sync Blocks, which usually indicates positive discussions.

In the future, this work could be improved in several ways. As the results of the interview show, we need to improve the interaction of CoProStory. For instance, we can make our real-time feedback clearer. Furthermore, the interplay between quality of collaboration and learning outcome is worth to be further studied in depth through designs of more specific learning modules and corresponding evaluation methods.

## References

* Andy Cockburn, Andrew Bryant. (1997). Leogo: An equal opportunity user interface for programming: Journal of Visual Languages & Computing 8, 5: 601-619.

* Charlie McDowell, Linda Werner, Heather Bullock and Julian Fernald. (2002). The Effects of Pair-Programming on Performance in an Introductory Programming Course. In Proceedings of the SIGCSE technical symposium on Computer science education (SIGCSE'02), 38-42.

* Douglas H. Clements. (1999). The future of educational computing research: the case of computer programming. Information Technology in Childhood Education Annual, 147-179.

* Florian Scharf, Thomas Winkler, Claudia Hahn, Christian Wolters and Michael Herczeg. (2012). Tangicons 3.0: an educational non-competitive collaborative game. In Proceedings of the 11th International Conference on Interaction Design and Children, 144-151.

* Hideyuki Suzuki and Hiroshi Kato. (1995). Interaction-Level Support for Collaborative Learning: AlgoBlock—an open programming language. In Proceedings of the first international conference on Computer support for collaborative learning, 349-355.

* Hiroshi Ishii, Brygg Ullmer. (1997). Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems, 234-241.
Hyunjoo Oh, Anisha Deshmane, Feiran Li, Ji Yeon Han, Matt Stewart, Michael Tsai, Xing Xu, and Ian Oakley. (2013). The Digital Dream Lab: Tabletop Puzzle Blocks for Exploring Programmatic Concepts. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction, 51-56.

Jeannette M. Wing. (2006). Computational thinking. Communications of the ACM 49, 3: 33-35.

Leslie Eugene Chipman. (2007). Collaborative technology for young children's outdoor education[M]. ProQuest.

Michael S. Horn, R. Jordan Crouser, Marina U. Bers. (2012). Tangible interaction and learning: the case for a hybrid approach. Personal and Ubiquitous Computing 16, 4: 379-389.

Michael S. Horn, Robert J. K. Jacob. (2007). Tangible Programming in the Classroom with Tern. In Proceedings of the CHI '07 Extended Abstracts on Human Factors in Computing Systems.

Neha Katira, Laurie Williams, Eric Wiebe, Carol Miller, Suzanne Balik and Ed Gehringer. (2004). On understanding Compatibility of Student Pair Programmers. In Proceedings of the SIGCSE technical symposium on Computer science education (SIGCSE'04), 7-11.

Regan L. Mandryk, Kori M. Inkpen, Mark Bilezikjian, Scott R. Klemmer and James A. Landay. (2001). Supporting children's collaboration across handheld computers. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems, 255-256.

Robert F. Bales. (1950). A set of categories for the analysis of small group interaction. American Sociological Review, 257-263.

Scott S D, Regan L. Mandryk, Kori M. Inkpen. (2003). Understanding children's collaborative interactions in shared environments[J]. Journal of Computer Assisted Learning, 19(2): 220-228.

Timothy S. McNerney. (2004). From turtles to Tangible Programming Bricks: explorations in physical language design. Personal and Ubiquitous Computing, Volume 8, 326-337.

Tomohito Yashiro, Yasushi Harada, and Kazushi Mukaiyama. (2017). Plugramming: A Tangible Programming Tool for Children's Collaborative Learning. International Conference on Human-Computer Interaction, 398-409.

## Acknowledgments

This research is supported by the National Key Research and Development Program under Grant No. 2016YFB0401202, and the National Natural Science Foundation of China under Grant No. 61872363, 61672507, 61272325, 61501463 and 61562063.
# A Spiral Model of Collaborative Knowledge Improvement to Support Collaborative Argumentation for Science Learning: Technological and Pedagogical Design

Wenli Chen, Si Zhang, Yun Wen, Chee-Kit Looi, and Jennifer Yeo

wenli.chen@nie.edu.sg, si.zhang@nie.edu.sg, yun.wen@nie.edu.sg, cheekit.looi@nie.edu.sg, jennifer.yeo@nie.edu.sg

Nanyang Technological University, Singapore

**Abstract:** Innovations in teaching and learning are not merely about merely the design of technologies but the integration of technologies and pedagogical practices in supporting meaningful learning. This paper presents the design, implementation and evaluation of a web-based system to support secondary school students' collaborative argumentation (CA) in science learning in Singapore. A pedagogical model named the Spiral Model of Collaborative Knowledge Improvement (SMCKI) is proposed in this study to inform the system and learning activity design. Starting with a stage of individual brainstorming, the pedagogical model scaffolds students to go through five stages of intra-group and inter-group knowledge improvement and refinement, to support the advancement of their collective and individual knowledge. The results showed that the students significantly improved on their scientific content knowledge through the staged collaboration argumentation activities in the web based learning environment.

## Introduction

In recent years there is a shift in science education from focusing on exploration and experiment to the construction of argument and explanation (Duschl & Osborne, 2002). Argumentation refers to the process of discussion and negotiation among peoples of different point of view (Osborne, Erduran, & Simon, 2004; Sampson & Clark, 2009). Argumentation is part of the practice of science for evaluating, refining and establishing new theories (Duschl & Osborne, 2002). It has been widely recognized as an effective approach for science learning (Osborne & Patterson, 2011) as it helps students improve their conceptual understanding (Bouyias & Demetriadis, 2012), understand the nature of science, promotes deeper learning of content (Nussbaum, 2008), enhance knowledge creation (Erduran, Simon, & Osborne, 2004), and develop metacognitive skill (Böttcher & Meisert, 2011).

Many effective argumentation happens among students (Scheuer, Loll, Pinkwart, & McLaren, 2010) who engage in proposing, critiquing, coordinating evidence with claims to construct arguments and explanations, reflecting, and evaluating each other's ideas. Educational researchers have developed some pedagogical approaches and tools to support students' collaborative argumentation (Scheuer et al., 2010). However, students were often found as working ineffectively and inefficiently when doing the argumentation. Students are still not substantively engaged in the process of discussion and negotiation (Yun & Kim, 2015). One of the critical issues is that students' discussions do not lead to significant improvement of idea improvement due to the lack of interdependence among group members. More carefully designed collaborative argumentation activities which can support idea imporvement is needed.

This paper presents a web-based platform to support students' collaborative argumentation in science learning. Developed by National Institute of Education, Nanyang Technological University Singapore, this system is designed for supporting generalized coordination of collaborative argumentation among students and the teacher in the following three aspects: 1) argumentation: developing graph-based argumentation to represent argument elements, and relationships between them, 2) collaboration: scaffolding student's continuous knowledge improvement through a staged Spiral Model for Collaborative Knowledge improvement, 3) peer assessment and critique: supporting intra-group peer assessment and critique by quantitative rating and qualitative feedbacks.

## Literature review

### Argumentation and learning

Argumentation is viewed as a vital type of knowledge construction activity that can lead to knowledge advancement (Weinberger & Fischer, 2006). Science teachers have designed various learning activities to support students' argumentation (Emig, McDonald, Zembal-Saul, & Strauss, 2014), and students are encouraged to participate in scientific practice activities, reflective dialogue, and actively use evidence to support their own
claims (Bulgren et al., 2014; Duschl et al., 2002). Argumentation is as an effective learning process and educational outcome for science education (Osborne & Patterson, 2011).

Many argumentative frameworks have been proposed to support students’ argumentation in science education. Toulmin Argument Pattern (TAP) is the most commonly used framework (Toulmin, 1958). TAP consists of six elements: claim, qualifier, data, warrant, backing, and rebuttal. A *claim* is an assertion, or statement, about a belief or idea. *Grounds* are statements or reasons that support the *claim*. *Warrants* are an elaboration on the reasoning behind why the person believes their *claim* to be true. A *qualifier* provides strength and clarification to the *grounds* and *warrant*. With a *qualifier*, the *claim* is valid only during a specific circumstance. A *rebuttal* is a particular condition in which the *warrant* becomes void and the *claim* is not valid. While a *qualifier* can provide strength and clarification, the *backing* provides support to the *warrant* by stating why the *warrant* is acceptable.

Another argumentation framework is Walton's argumentation scheme (Walton & Reed, 2005). Walton breaks down argumentation into six different types of dialogue: persuasion dialogue, the inquiry, negotiation, information-seeking dialogue, deliberation, and eristic dialogue. Persuasion dialogue is when one person is trying to persuade another person that some particular suggestion is true through the use of arguments that show or prove that it is true. Inquiry dialogue involves a group of people investigating the reasons for some event or phenomenon. Negotiation dialogue involves bargaining, while information-seeking dialogue is when one person has, or appears to have, information that another person wants. Deliberation dialogue is making a decision to solve a problem and lastly, eristic dialogue is a quarrel between two people. The argumentative patterns are widely used to evaluate students' argumentation ability and its development (Garcia-Mila, Gilabert, Erduran, & Felton, 2013).

In addition to providing argumentative frameworks to help students develop their ability to argue, many studies have found that graph-based computer application tools can help students develop argumentation skills and scientific knowledge (Dwyer, Hogan, & Stewart, 2012; Hsu, Dyke, Smith, & Looi, 2018). Compared with these previous studies, this study especially emphasizes the integration of technology and pedagogy. While technical tools support scientific argumentation, pedagogical design better supports students’ collaborative learning.

## Collaborative argumentation

As reflected in literature, students encounter a lot of difficulties in collaborative argumentation. Except for unequal participation and low level engagement as often observed in collaborative learning activities (Choo, Eshaq, Samsudin, & Guru, 2009), students might also not use appropriate and/or enough evidence to warrant their claims, or justify their choice of evidence in the arguments they produce (Erduran, Simon, & Osborne, 2004; Weinberger, Stegmann, & Fischer, 2010). The self-explanations made during argumentation were only sporadically generated by even good students (Larusson & Alterman, 2009). Students did not evaluate the validity or acceptability of an explanation for a given phenomenon in a satisfying way as well (Bouyias & Demetriadis, 2012). There was also evidence that students had problem producing “a good argumentation sequence” (Larusson, & Alterman, 2009). When required to engage in scientific argumentation, a critical type of discourse epistemic practice in doing and communicating science (Lazarou, Sutherland, & Erduran, 2016), students were confronted with more challenges. Existing research has indicated that students often lacked ability to determine the acceptance, rejection, or modification of ideas (Hogan & Maglienti, 2001). Their acts of distorting, trivializing, and ignoring certain evidence frequently resulted in misconceptions (Sampson & Clark, 2008). In the realm of CSCL research, collaborative argumentation is regarded as a key type of knowledge construction process that should be mastered by students to enable knowledge advancement. Yet in the argumentation processes, students without necessary training were often noted as working ineffectively and inefficiently. They did not ask each other questions, not explain or clarify their own opinions, not articulate the reasoning behind, not elaborate and reflect on knowledge (Kollar, Fischer, & Hesse, 2006; Martin & Hand, 2009). They also hardly take alternative perspectives into consideration (Sampson, Grooms, & Walker, 2011).

In science education, engaging in collaborative argumentation is challenging for most students (Gould & Parekh, 2018). During the process of collaborative argumentation, it is difficult for students to construct scientific explanations, and students adopt the wrong ways of arguing and only a few students can complete the argumentation task (Ryu & Sandoval, 2012). The main reason for this phenomenon is that students lack the basic skills of collaborative argumentation. Students don't know how to create a logical point of view and present evidences for support it. To improve students' performance of collaborative argumentation, one kind of method is to provide students with a structured argumentation scaffold (Suthers, 2003).

## Web-based system design

The web-based system development is based on the J2EE platform, and adopts the SSH (Spring, Struts, and Hibernate) technology. The system includes three modules: graph-based argumentation, collaborative knowledge
improvement, and peer assessment and critique. Figure 1 shows the screenshots of the system. The central area of the screen is students' graph-based argument workspace, where students use evidence to support or oppose the claim. The activity description (such as activity topic, activity introduction, role assignment, experimental data and evaluation rules, etc) and chat window are shown on the top right of the page.

## Graph-based argumentation design

The system is a graph-based argumentation application that uses different shapes to represent the argument elements. The oval node represents the claim, the cloud node represents the idea (not sure if it is claim or evidence), and the rectangle node represents the evidence. The explanation of shapes of the argument elements in the system are shown in figure 1. The arrows represent the relationships among the nodes. Green arrows represent "evidence for" whereas red arrows represent "evidence against".

<figure><img src="image_110.png" /><figcaption>**Figure 1.** The Screenshot and explanation of augment elements of the System.</figcaption></figure>

The system supports uploading and downloading attachments (including pictures and documents) with the nodes. In addition, the system supports the transitions between different shapes, for example, the idea can be changed to the claim, or changed to the evidence if it was connected with the claim. Because the team members are engaged in collaborative argumentation, members from the same group can modify or delete the nodes/relationships created by other members of the group.

## Collaboration design: Spiral Model of Collaborative Knowledge Improvement (SMCKI)

Instead of collaboration scripts that generally provide a detailed set of guidelines, rules and structured tools for describing how the group members should interact, we propose a pedagogical model to guide the interaction processes so that Collaborative Knowledge Improvement can happen more effectively in classrooms. The 5-stage Spiral Model of Collaborative Knowledge Improvement (SMCKI) provides a tangible structure for one operational collaborative activity design beginning with brainstorming and a structured process of constant knowledge improvement. The model focuses on democratic knowledge sharing as well as cycles of individual, group and class knowledge enhancement (see Figure 2).

This model is based on the authors' previous work on Funnel Model for rapid collaborative knowledge improvement (Wen, Looi, & Chen, 2011), which consists of three stages of collaborative learning process. By respecting and encouraging cognitive diversity, the first stage encourages the creation of diverse ideas. The subsequent stages tap on this diversity to seek synergy of ideas, and a stage of convergence and consensus seeking leading to knowledge convergence and advancement of the individuals, groups and class. The CMCKI model entails 5 stages (Figure 2). The teacher used verbal instructions and system buttons to regulate the 5 stages of scientific argumentation activity.
I: Individual brainstorming: Students individually construct argument with claims and evidences of the scientific phenomena. The argument represents the best knowledge of the individuals.

II: Intra-group synergizing: Students discuss, synergize and consolidate group members' work by deleting, adding, modifying nodes/relationships. A group graph-based argumentation diagram is created which represents the best knowledge of the group.

III: Inter-group peer assessment and critique: Students go to other groups to provide quantitative ratings and qualitative comments by identifying the strengths and areas for improvements.

IV: Intra-group refinement: Students go back to their own groups and refine the group work based on other groups' ratings and feedbacks. After further verbal negotiation, they were required to seek consensus and finalize their group idea.

V: Individual idea perfection: Individually, students write an argumentative essay or reflection report to explain the scientific phenomena.

<figure><img src="image_111.png" /><figcaption>Figure 2. Spiral Model of Collaborative Knowledge Improvement.</figcaption></figure>

## Peer assessment and critique

In stage III of SMCKI, students go to other groups to provide quantitative rating and qualitative feedbacks on the content and structure of the argument. Students can rate both the nodes and the relationships in the argument diagram. When students double-click the node or relationship, an assessment window will pop up (see Figure 3). All the students' qualitative feedbacks will be displayed in a table format. In order to encourage thoughtful critique and rating, each element of the argument diagram can only be rated once. Students name will not be shown in this stage.

To make the ratings more intuitive for students, the color of the lines of those nodes and relations which received positive ratings will be highlighted whereas the color of the lines of those nodes and relations which received negative ratings will be lighter. This feature will help channel teacher and students' attention when they do stage 3 and 4 argumentation activities. On top of assessing the individual elements of the argument diagram, students can assess the whole diagram based on the rubrics given by the teacher.

<figure><img src="image_112.png" /><figcaption>Figure 3. The screen shot of the peer assessment and critique (Stage III of SMCKI).</figcaption></figure>

## Research questions

In this study, we focus on investigating whether and how was the use of web-based system informed by the SMCKI helpful for improving students' conceptual knowledge learning?
# The enactment of SMCKI in science classroom

## Participants

A secondary grade 4 (16 years old on average) class with a total of 33 students in Singapore participated in the project. They studied physics phenomenon through SMCKI approach using the system. All the students in this school are equipped with a personal iPad 24/7, and they are proficient in using iPad as a learning tool. The teacher has rich experiences in science teaching and he is tech-savvy. Before the project started, researchers conducted training for the teacher with the aim to improve teacher's understandings of the technological and pedagogical design of the system. All the students were heterogeneously grouped by the teacher according to their ability. There were nine groups of 3-4 students in the class.

The collaborative argumentation activities were co-designed by the teachers and researchers. Three lessons were implemented (50 minutes per lesson) from July to August in 2018. The data analyzed in this paper were from the third lesson on the electromagnetic induction phenomenon. Following the SMCKI approach, students went through 5 staged of collaborative argumentation. The time allocation for each stage is: Individual brainstorming (7 minutes), within-group synergizing (10 minutes), inter-group assessment and critique (8 minutes), intra-group refinement (10 minutes), and individual perfection (after class). The paper examines the students' conceptual development in the process for 5-staged collaborative argumentation activities.

## Methods and instruments

During the collaborative argumentation activities, students were asked to provide explanation about the phenomena that they observed from an iPad which showed the induced current flowed in a solenoid over time when a magnet fell through it. Students presented their explanations based on an explanation framework which consisted of three elements: claim, evidence, and reasoning (McNeill, Lizotte, Krajcik, & Marx, 2006). A total of 48 explanations generated in students' collaborative argumentation activities. Content analysis was employed to analyze the nature of peer comments in Stage 3. The unit of content analysis is one comment.

*Pre-test and post-test design.* Pretest and posttest were used to measure students' conceptual knowledge before and after the collaborative learning activity and used the same text paper, which contained 4 questions, one point for each question. The test questions were closely related to the knowledge points in this lesson.

*Coding scheme of peer feedbacks.* A coding scheme adapted from Clark & Sampson (2007) was used to code students' qualitative feedbacks in stage 3. The coding scheme consists of five codes: rebuttal, support, query, emotive appeal, and off-task comments. The second and third authors independently coded all the 79 comments, with an inter-rater reliability of 0.864 (Cohen's Kappa).

*Students' interview design.* Students were required to reflect on their experiences of collaborative learning based on one guiding questions in the post-intervention interview: What did you do at each stage of collaborative learning activity?

## Data analyses and results

### Comparison of students' conceptual understanding between Stage I and Stage V.

In Stage I and V, individual students are required to construct argument and explanation of the scientific phenomenon individually. A comparison was made to measure students' conceptual knowledge between stage I and V. Descriptive statistics were used to analyze the mean and standard deviation of conceptual understanding, see Table 1. As the data do not conform to normal distribution, the Wilcoxon signed Ranks test was used to detect differences on conceptual knowledge between 2 stages. The results in table 1 showed that students' conceptual understanding in Stage V was significantly higher than the stage I (z=-4.647**, p<0.001).

<table><caption>Table 1: Wilcoxon signed Ranks test results</caption><thead><tr><th></th><th>N</th><th>Minimum</th><th>Maximum</th><th>Mean</th><th>Std.Dev</th><th>Z</th></tr></thead><tbody><tr><td>Stage I</td><td>33</td><td>0</td><td>4</td><td>2.58</td><td>0.830</td><td>-4.647***</td></tr><tr><td>Stage V</td><td>33</td><td>2</td><td>4</td><td>3.70</td><td>0.529</td><td></td></tr></tbody></table>

Note. *** p<0.001.

### Students' conceptual development throughout staged activities: Group 3 as the case study
We provide a qualitative account of group 3's process of doing the staged collaborative argumentation. In order to clearly show the process of students' revision of their explanations, we used bold font and underlining in the text to show the position of students' revision.

*Stage I individual posting.* All 4 students (Jenny, Hellen, Nichole, and Sabrina, all pseudo names) managed to post their explanation of the scientific phenomena based on their pre-existing knowledge.

*Stage II intra-group synergizing.* After seeing each other's posting, students compared each other's explanations and tried to synergize their work through negotiation and discussion. In this stage, Helen added a scientific knowledge element to her previous explanation in stage 1.

> "When the magnet first enters the solenoid, there is a momentary increase in rate of cutting of magnetic field lines by the coil, hence by Faraday's Law, an e.m.f. is induced in the coil and there is a current. In the moment that the magnet is completely in the solenoid, there is no rate of cutting of magnetic field lines by the coil, so there is no e.m.f. induced in that moment and the current is 0. When the magnet falls out of the solenoid, there is a momentary decrease in rate of cutting of magnetic field lines by the coil, hence by Faraday's Law, an e.m.f. is induced in the coil. Since the motion of the magnet is in the opposite direction as compared to when it enters the solenoid, the induced e.m.f. is in the opposite direction and the current is also in the opposite direction, **by Lenz's Law**." (14:02, posted by Helen)

In this stage, Sabrina repeatedly revised the statements that represent her explanation. In the process of revision, the representation of Sabrina's explanation moved closer to the expert scientific concept knowledge.

> "When the magnet is dropped into the solenoid, **there is a momentary rate of cutting of magnetic field lines and thus an e.m.f is induced and a current in the solenoid is recorded. According to Lenz's law of motion, as the magnet changes direction to move out of the magnet, the current recorded would be of similar magnitude and amplitude but in a different direction.**" (14:06, posted by Sabrina)

In stage II, students not only revised their own explanations, but also added new explanations. Sabrina added a new claim and evidence to improve own explanation. Jenny put forward a backing to Helen's explanation. During the post-activity interviews, the 4 students explained to researcher how they synergized their group argumentation diagram in Stage 2. Students discussed with her group mates first and then modified her explanation (from the interview transcript with Sabrina). If student's concept is wrong, she would modify her explanation (from the interview transcript with Sabrina). If students thought their partners' explanations make more sense, they will add others' idea to her explanation (from the interview transcript with Helen).

*Stage III Intra-group assessment and critique.* Group 4 students went to group 3 to provide ratings and feedbacks. To examine the role of stage 3 peer critique for students knowledge improvement in argumentation, content analyses was done to understand the nature of peer comment given. Group 4 gave a total of 7 comments in this stage, including 1 support, 3 rebuttals, and 3 emotive appeals. Table 2 shows an example of comments given by group 4.

<table><caption>Table 2: An example of comments given by group 4 to group 3</caption><tr><td>1. Explanation</td><td>Group 3 (Student Jenny)<br/>"When the magnet is dropped into the coil, there is a momentary rate of cutting of the magnetic field lines from the magnet by the coil. By faraday's law, an emf is induced in the coil, which drives an induced current, hence a sudden increase in magnitude of current was recorded. This repeated in the opposite direction when the magnet fell into the solenoid, hence by lenz's Law, the direction of the current is opposite to that at first, so a current of similar magnitude but in opposite direction was recorded."</td></tr><tr><td>1.1 Rebuttal</td><td>Group 4 (comments to group 3)<br/>"why does the direction change?"</td></tr><tr><td>1.2 Rebuttal</td><td>Group 4 (comments to group 3)<br/>"What does "direction of current is opposite to that at first" a bit confusing."</td></tr><tr><td>1.3 Emotive appeal</td><td>Group 4 (comments to group 3)<br/>"True"</td></tr></table>
During the post-activity interviews, students explained their experiences in this stage. In this stage, students of group 4 gave comments to group 3 based on the following criteria: (1) key words, (2) explanation was clear or not, (3) the number of evidence supported one claim, (4) students' understanding of the conceptual knowledge, and (5) the concept was correct or accurate. If students thought that the other group's explanations were better, they would bring other groups' ideas to their own group.

*Stage IV Intragroup refinement.* Based on the explanations and comments from other group, students in group 3 further revised their explanations. Students added scientific conceptual knowledge in their explanations or had their doubts about some conceptual knowledge cleared.

## Discussion and conclusion

This study proposes web-based platform to support students' collaborative argumentation in science education by scaffolding students' graph based argumentation informed by a spiral model of collaborative knowledge improvement. An empirical study was conducted to investigate how student construct argumentation for science learning throughout the stage-collaborative argumentation activity. The paper provides a descriptive account of the interactions of a group as the students engaged in spiral cycles of knowledge improvement. The case study demonstrates that students' conceptual understanding improved continuously through the 5 stages of SMCKI. This study also showed that revision, deletion and synthesis of explanation were sophisticated skills, and the web-based system needed to provide scaffoldings for the development of students' these skills. In addition, teachers' feedback was also very important. In the second and fourth stages, students would perform better if teachers could provide some feedback on scientific knowledge and collaboration process.

Informed by SMCKI, our web-based system has 3 characteristics: “low floor”, “wide walls”, and “high ceiling”. “Low floor” means all students can easily participate and contribute to the group and class’s knowledge improvement; “Wide walls” means the networked online system enables everyone to connected through individual-group-class-group-individual interaction pattern. There are many opportunities for students be exposed to diversified ideas and perspectives. “High ceiling” means that students can tap on each other’s ideas to improve the knowledge throughout the staged spiral model. Our empirical study shows that the quality of ideas of the one stage is always higher than the preceding stage. Students work on increasingly complicated ideas by communication, negotiating and critique, all of which require higher order thinking.

The SMCKI is also meant to scaffold teachers to enact and orchestrate the collaborative learning activities in the classroom as well as to build capacity to be able to design such learning activities themselves. It has the potential for teachers to embark on collaborative learning activities in the classroom and to manage the risks of the activities breaking down or not reaching any kind of fruitful collaboration. SMCKI is not specific to collaborative argumentation or science learning. It can be applied in any collaborative learning activity. Further study is needed to investigate the application of SMCKI in other contexts of collaborative learning.

## References

* Bouyias, Y., & Demetriadis, S. (2012). Peer-monitoring vs. micro-script fading for enhancing knowledge acquisition when learning in computer-supported argumentation environments. *Computers & Education, 59*(2), 236-249.

* Böttcher, F., & Meisert, A. (2011). Argumentation in science education: A model-based framework. *Science & Education, 20*, 103-140.

* Bulgren, J. A., Ellis, J. D., & Marquis, J. G. (2014). The use and effectiveness of an argumentation and evaluation intervention in science classes. *Journal of Science Education and Technology, 23*(1), 82-97.

* Choo, K. A., Eshaq, A. R. M., Samsudin, K. A., & Guru, B. K. (2009). An evaluation of a constructivist online collaborative learning activity: A case study on Geometry. *Turkish Online Journal of Educational Technology, 8*(1), 15-25.

* Clark, D. B., Sampson, V., Weinberger, A., & Erkens, G. (2007). Analytic frameworks for assessing dialogic argumentation in online learning environments. *Educational Psychology Review, 19*(3), 343-374.

* Duschl, R. A., & Osborne, J. (2002). Supporting and promoting argumentation discourse in science education. *Studies in Science Education, 38*, 39-72.

* Dwyer, C. P., Hogan, M. J., & Stewart, I. (2012). An evaluation of argument mapping as a method of enhancing critical thinking performance in e-learning environments. *Metacognition and Learning, 7*(3), 219-244.

* Emig, B. R., McDONALD, S. C. O. T. T., ZEMBAL - SAUL, C. A. R. L. A., & Strauss, S. G. (2014). Inviting argument by analogy: Analogical - mapping - based comparison activities as a scaffold for small - group argumentation. *Science Education, 98*(2), 243-268.

* Erduran, S., Simon, S., & Osborne, J. (2004). TAPping into argumentation: Developments in the application of Toulmin's argument pattern for studying science discourse. *Science education, 88*(6), 915-933.
Garcia-Mila, M., Gilabert, S., Erduran, S., & Felton, M. (2013). The effect of argumentative task goal on the quality of argumentative discourse. *Science Education*, 97(4), 497-523.

Gould, D. L., & Parekh, P. (2018). Mentoring and Argumentation in a Game-Infused Science Curriculum. *Journal of Science Education and Technology*, 27(2), 188-203.

Hogan, K., & Maglienti, M. (2001). Comparing the epistemological underpinnings of students' and scientists' reasoning about conclusions. *Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching*, 38(6), 663-687.

Hsu, P.-S., Dyke, M. V., Smith, T. J., & Looi, C.-K. (2018). Argue like a scientist with technology: the effect of within-gender versus cross-gender team argumentation on science knowledge and argumentation skills among middle-level students. *Educational technology research and development*, 1-34.

Kollar, I., Fischer, F., & Hesse, F. W. (2006). Collaboration scripts–a conceptual analysis. *Educational Psychology Review*, 18(2), 159-185.

Larusson, J. A., & Alterman, R. (2009). Wikis to support the “collaborative” part of collaborative learning. *International Journal of Computer-Supported Collaborative Learning*, 4(4), 371-402.

Lazarou, D., Erduran, S., & Sutherland, R. (2017). Argumentation in science education as an evolving concept: Following the object of activity. *Learning, Culture and Social Interaction*, 14, 51-66.

Martin, A. M., & Hand, B. (2009). Factors affecting the implementation of argument in the elementary science classroom. A longitudinal case study. *Research in Science Education*, 39(1), 17-38.

McNeill, K. L., Lizotte, D. J., Krajcik, J., & Marx, R. W. (2006). Supporting students’ construction of scientific explanations by fading scaffolds in instructional materials. *Jrl of the Learning Sciences*, 15(2), 153-191.

Nussbaum, E. M. (2008). Collaborative discourse, argumentation, and learning: Preface and literature review. *Contemporary Educational Psychology*, 33(3), 345-359.

Osborne, J., Erduran, S., & Simon, S. (2004). Enhancing the quality of argumentation in school science. *Journal of research in science teaching*, 41(10), 994-1020.

Osborne, J. F., & Patterson, A. (2011). Scientific argument and explanation: A necessary distinction? *Science Education*, 95(4), 627-638.

Ryu, S., & Sandoval, W. A. (2012). Improvements to elementary children's epistemic understanding from sustained argumentation. *Science Education*, 96(3), 488-526.

Sampson, V., & Clark, D. (2009). The impact of collaboration on the outcomes of scientific argumentation. *Science education*, 93(3), 448-484.

Sampson, V., & Clark, D. B. (2008). Assessment of the ways students generate arguments in science education: Current perspectives and recommendations for future directions. *Science Education*, 92(3), 447-472.

Sampson, V., Grooms, J., & Walker, J. P. (2011). Argument - Driven Inquiry as a way to help students learn how to participate in scientific argumentation and craft written arguments: An exploratory study. *Science Education*, 95(2), 217-257.

Scheuer, O., Loll, F., Pinkwart, N., & McLaren, B. M. (2010). Computer-supported argumentation: A review of the state of the art. *International Journal of Computer-supported collaborative learning*, 5(1), 43-102.

Suthers, D. D. (2003). Representational guidance for collaborative inquiry. In *Arguing to learn* (pp. 27-46). Springer, Dordrecht.

Toulmin, S. (1958). *The uses of argument*. New York: Cambridge University Press.

Walton, D., & Reed, C. A. (2005). *Argumentation schemes and enthymemes*. Synthese, 145, 339-370.

Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction in computer-supported collaborative learning. *Computers & education*, 46(1), 71-95.

Weinberger, A., Stegmann, K., & Fischer, F. (2010). Learning to argue online: Scripted groups surpass individuals (unscripted groups do not). *Computers in Human behavior*, 26(4), 506-515.

Wen, Y., Looi, C.-K., & Chen, W. L. (2011). Towards a model for rapid collaborative knowledge improvement in classroom language learning. In H. Spada, G. Stahl, N. Miyake & N. Law (Eds.), *Proceedings of Computer Supported Collaborative Learning (CSCL) 2011 (Part 2, pp. 836-840)*. Hong Kong, China: International Society of the Learning Sciences.

Yun, S. M., & Kim, H. B. (2015). Changes in students’ participation and small group norms in scientific argumentation. *Research in Science Education*, 45(3), 465-484.

## Acknowledgments

This study was funded by Singapore Ministry of Education (MOE) under the Education Research Funding Programme (OER 07/17 CWL) and administered by National Institute of Education (NIE), Nanyang Technological University, Singapore. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Singapore MOE and NIE.
# Flexible CSCL Orchestration Technology: Mechanisms for Elasticity and Dynamism in Pyramid Script Flows

Kalpani Manathunga and Davinia Hernández-Leo

kalpi.nisansala@gmail.com, davinia.hernandez-leo@upf.edu

ICT Department, Universitat Pompeu Fabra, Barcelona

**Abstract:** Flow patterns (e.g., Pyramid or Snowball) formulate good practices to script collaborative learning scenarios, which have been experimented in small-scale settings widely. Applying flow patterns on large-scale contexts present challenges to educators in terms of orchestration load. Orchestration technology can support educators to manage collaborative activities; yet existing technology do not address flexibility challenges like accommodating growing numbers of students or tolerating dynamic conditions in learning settings. We define elasticity and dynamism as two key elements in the flexibility of a script. Elasticity is related to the capacity of an orchestration technology to incorporate varying participant counts. Dynamism is the capacity to maintain a pedagogically meaningful script progression in presence of different individual behaviors. In this paper we propose flow creation and flow control mechanisms to address elasticity and dynamism in orchestration technology for Pyramid flows. These mechanisms, implemented in the PyramidApp tool, have been evaluated across four scenarios varying from small to large settings. The results show that rules enabling pyramid creation on-demand and the use of timers are useful to achieve elasticity and dynamism in the pyramid formation and progression in an automatic manner.

## Introduction

Collaboration is a coordinated process by which individuals construct and maintain shared conceptions where knowledge is co-constructed socially (Roschelle & Teasley, 1995). In collaborative learning, situations are created in which particular forms of interactions among learners are expected to occur, leading to productive learning experiences. Computer-Supported Collaborative Learning (CSCL) contributes mechanisms and technologies supporting creation of such collaborative learning situations (Dillenbourg, Sanna, & Fischer, 2009; Roschelle & Teasley, 1995). Hence, CSCL environments need to be carefully designed and implemented incorporating interaction generation and regulation mechanisms. Moreover such CSCL contexts should scaffold productive interactions and/or to facilitate activity monitoring and intervening when required (Dillenbourg et al., 2009) since free collaboration does not necessarily result in fruitful learning (Dillenbourg & Jermann, 2010).

CSCL scripting means shaping up the way that collaborations are desired to happen with technology-mediation (Dillenbourg & Tchounikine, 2007) triggering specific types of interactions beneficial for learner cognition, while achieving educational objectives. Scripts define the activity sequence, group formation, phase changing, role allocation and rotation, resource distribution, mediating communication and coordination, constrain peer interactions in social and cognitive activities that would otherwise occur rarely or not at all (Dillenbourg & Tchounikine, 2007; Kobbe et al., 2007). In this notion, scripting is possible as *micro-scripts*, emphasizing on individual learner's actions with finer granularity and *macro-scripts*, defining interactions and regulations in coarse-grained activity flows (Dillenbourg & Tchounikine, 2007; Kobbe et al., 2007). Collaborative Learning Flow Patterns (CLFPs) are examples of macro-scripts reflecting best practices to orchestrate collaborative learning which are broadly accepted and repetitively utilized by practitioners (Hernández-Leo et al., 2006). Examples of CLFPs are Jigsaw, Pyramid or Snowball, Think-Pair-Share (TPS) and Thinking Aloud Pair Problem Solving (TAPPS). Each pattern is driven by its governing pedagogy that should not be modified during the design. For example, Pyramid pattern is considered as good practice to structure collaborative learning across multiple epistemic tasks and educational levels, fostering individual accountability and positive interdependence (Davis, 2002; Hernández-Leo et al., 2006). The pedagogy of this pattern is such that individuals study a given problem initially and propose a preliminary solution. Such solutions are discussed and compared to propose a shared solution in groups. This discussing and negotiation will repeat in growing group sizes (e.g. two groups in a level of the pyramid join as a single group in the next level) until the whole group reaches consensus to propose a common solution.

Practitioners are required to invest some effort to understand pattern definitions and types of constraints to design effective, meaningful scripts (Dillenbourg & Tchounikine, 2007; Hernández-Leo et al., 2006). Moreover, in its enactment with students, they need to orchestrate or manage in real time the script mechanics (Dillenbourg & Jermann, 2010). Flexible orchestration allows to adapt CSCL scripts in real-time
with a degree of freedom to modify various orchestration aspects like group formation, role allocation and rotation (Dillenbourg & Jermann, 2010). Script modifiability is non-trivial, due to unexpected situations like learners not being present or leaving the activity in the middle spoiling on-going collaborations. As a result, orchestrating activity flows manually can be challenging for practitioners (Dillenbourg & Tchounikine, 2007; Sharples, 2013). Therefore, technology-mediated or semi-automated orchestration services are beneficial to real-time manage on-going activity flows. Previous work in the field had provided extensive knowledge in designing scripted collaborative learning flows effectively (Hernández-Leo et al., 2006; Pérez-Sanagustín, Burgos, Hernández-Leo, & Blat, 2011; Rodríguez-Triana, 2014), yet applicable mostly either at small-scale or co-located learning settings (Manathunga & Hernández-Leo, 2017). Different learning settings scaling, from small face-to-face classrooms to massive online learning communities, present varied script design and orchestration requirements. Diverse scales for collaborative learning orchestration raises a number of challenges, including adaptation challenges, tolerating unexpected conditions like drop-outs or late-joiners, needs for (re)designing scripts on-the-fly or managing the orchestration load (Sharples, 2013).

The main motivation of this research is to seek how already existing pattern-inspired scripts (in our case, Pyramid CLFP) can be enhanced to achieve flexible meaningful orchestration in order to be applied upon various learning scenarios. On the contrary to pre-defined, rigid scripts which can not be modified on-the-fly, flexible scripts allow practitioners to design and adapt in real-time with a freedom for modifications. Such modifications could be embedded during the design of the script or at the execution time as different mechanisms that do not violate the underlying pedagogical definition of the Pyramid pattern. Following sections of the article explain about those mechanisms and models introduced towards flexible CSCL orchestration, experimental settings, an analysis of the proposed mechanisms and a concluding discussion section.

## Mechanisms for flexible orchestration

The journey towards orchestration technology that supports flexible scripted collaborative learning flow is approached from two key elements that we define as *elasticity* and *dynamism*. Elasticity is defined as the capacity of an orchestration technology to accommodate growing numbers of learners to collaborative learning activities without violating the underlying pedagogical rationale of the script. Dynamism is defined as the capacity of an orchestration technology to maintain a pedagogically meaningful script progression in presence of different individual behaviors. Hence, unexpected scenarios, e.g. unanticipated activity drop-outs (Dillenbourg & Tchounikine, 2007), would not harm on-going collaborations.

As indicated above, we study the particular case of the Pyramid collaborative learning flow pattern, as an interesting structure for macro-scripts that has a potential to fit well in scenarios with a varying number of participants. Based on an analysis of the pattern structure and the targeted objectives of elasticity and dynamism, we propose a set of mechanisms named as “Flow Creation” and “Flow Control” rules. As the name implies, flow creation mechanisms are suggested at the script initiation stage, when building the flow based on its learning design, whereas flow control rules are inferred during the script execution. To show its feasibility and to evaluate the mechanisms, we have integrated their implementation into the PyramidApp tool.

## The case of the Pyramid Flow and the PyramidApp

Macro-scripts, such as those based on Jigsaw, Pyramid, Think-Pair-Share patterns, structure collaborations that potentially lead to fruitful learning (Hernández-Leo et al., 2006; Pérez-Sanagustín, Burgos, Hernández-Leo, & Blat, 2011; Rodríguez-Triana, 2014). Pyramid CLFP is structured in a way that individuals attend a given task and suggest an initial solution. Then they are assigned to small groups to discuss on the initially proposed options and agree upon a common option from the group which will be propagated to the next level(s) where much larger groups are formulated enriching collaborations and consensus reaching. At the global level, all participants agree upon one or few selected options that are reflected with the whole class. Pyramid pattern promotes individual accountability, peer interactions and positive interdependence among peers. The pattern can be applied to any subject matter, educational level and using (or not) different technologies (Hernández-Leo et al., 2006). An implementation of the pattern is reflected in PyramidApp, (Manathunga & Hernández-Leo, 2017), a web-based tool that implements an option submission space for participants to attempt the given task individually, a rating feature helping them to reach consensus with an integrated discussion space for negotiations and collaborations. PyramidApp has authoring features for the educators to create desired Pyramid pattern-inspired collaborative activities. It also provides a monitoring functionality for the educators to monitor on-going activities and to keep track of previous activities. The authoring process of PyramidApp implementation is shown in Figure 1 indicating different parameters, incorporating those related to the implementation of flow creation and control mechanisms. When enacting a PyramidApp activity, there are several phases as option submission, rating and discussion for further clarifications and to reach a consensus.
Next sections explain diverse parameters used for flow creation and control mechanisms and how those are embedded in PyramidApp.

<figure><img src="image_113.png" /><figcaption>Figure 1. PyramidApp authoring application with (a) flow creation and (b) flow control mechanisms.</figcaption></figure>

## Script flow creation mechanisms

As stated earlier, flow creation mechanisms were introduced to achieve elasticity, i.e. to accommodate growing numbers of learners without affecting the underlying pedagogy. As illustrated in Figure 2, the implemented rationale is creating multiple pyramids on-demand automatically and allocating students to on-going pyramids as pyramids increase sizes till a maximum threshold.

<figure><img src="image_114.png" /><figcaption>Figure 2. Flow creation mechanisms showing elastic pyramid initiation.</figcaption></figure>

Key aspects introduced here are the pyramid capacity (i.e., both minimum and maximum numbers of students that can be accommodated in a single pyramid) and number of pyramids replicated to allocate further incoming students. Once a pyramid is configured by indicating number of levels in the pyramid, group size and the total number of activity participants, it grows till a maximum volume calculated using the equation: $Max\_pyramid\_size = {(Min\_students\_per\_pyramid * 2) - 1}$. When the activity is initiated, pyramids are formulated and started using the value for minimum students per pyramid and then filled till the maximum size. During the authoring phase, if a practitioner allows multiple pyramid creation (see Figure 2), PyramidApp
replicates the given design automatically to generate several pyramids to occupy the total amount of participants. When the minimum number of students per pyramid is given, the authoring tool will automatically calculate and display the possible group sizes (e.g., 2,3,4,5, etc..) along with the possible number of levels per pyramid. Based on these values and the total class size, the tool calculates the number of ultimate results from the activity after leveling through the pyramids.

## Script flow control mechanisms

Once a Pyramid flow is activated, the flow needs to be controlled with more parameters, embedding dynamic behaviors to ensure a flexible progression during the execution time. In order to achieve dynamism with a smooth pyramid progression, we introduced several timers (see Figure 3) for different PyramidApp stages, to avoid the problem of different submission/rating times and drop-outs (inactive participants) causing the pyramid progression to freeze during option submission and rating stages. Submission timer and rating timer are the two main timers that define the maximum allowed time to complete those phases and their values can vary from minutes to days based on the activity being face-to-face or distance. To maintain a fluid dynamic flow, we use a satisfaction percentage (minimum number of active users completing a particular phase). Upon reaching the satisfaction percentage in a group, a countdown timer (*countdown timer* *< maximum allowed time per phase)* is activated until the maximum time allocated for that phase. The countdown timer notification is displayed in the interface for students to be informed. If all learners complete the task before any timer (either submission/rating timer or countdown timer) expiration, the group is promoted to the next level of the pyramid.

<figure><img src="image_115.png" /><figcaption>Figure 3. Flow control mechanisms showing dynamic flow progression.</figcaption></figure>

Finally, we have integrated Pyramid flow activity awareness features that trigger information related to the activity status such as current pyramid level details, group members, countdown timer notifications and email notifications providing activity updates. Pyramid participants can see how many peers or groups who have not yet completed the current level during the waiting stage. Once the pyramid is finished, both practitioners and participants can view the highly rated option(s) resulted from the activity, which then could be -for example- further discussed and analyzed by the practitioner with the participants.

## Evaluation

Flexibility of the flow orchestration can be construed by means of elasticity and dynamism. Our working hypothesis is that the proposed flow creation and control mechanisms embedded to Pyramid CLFP address flexibility successfully. Hence, the evaluation questions articulated and analyzed in this study are, do proposed mechanisms for flow creation address elasticity and do proposed flow control mechanisms address dynamism with meaningful orchestration? Meaningful orchestration is being pedagogically relevant during the script execution, which means that any novel mechanism introduced to the flow does not violate the essence of the pattern: e.g. pyramid group sizes to be preserved necessarily, every participant should have at least one peer for
collaborations irrespective of activity drop-outs, late-comers are combined with on-going pyramids from the next possible activity phase and let them collaborate, propose default field values for the practitioners to create efficient Pyramid flows like possible number of pyramid levels or preferred group sizes and provide activity awareness measures for both participants and practitioners.

## Experimental settings and data gathering

Several rounds of experiments were carried out in several sessions from two undergraduate courses (Introduction to Information and Communication Technologies (ITIC) offered in the first year and Network Protocols (NP) offered in the third year) of an Engineering School, taught by the same professor. PyramidApp activity authoring requirements varied based on different epistemic tasks and target groups (see Table 1). Some experiments were enacted during face-to-face classroom sessions whereas others were in distance mode, asynchronously. Some PyramidApp activities were administered in a Massive Open Online Course (MOOC) called Innovative Collaborative Learning with ICT (CLAT), a five-week MOOC ended in summer 2017, launched on Canvas platform. To evaluate the proposed flow creation and control mechanisms, diverse attributes related to the mechanisms were introduced at activity authoring and enactment phases in PyramidApp. Various data sources such as activity log files and questionnaires were used as data collection methods. Log files provided more accurate data to analyze time durations, activity participation and behavior during PyramidApp activities. Questionnaires provided practitioner's viewpoint as well as participants' perspective towards the activity. We used a mixed approach for the analysis triangulating both quantitative and qualitative data gathered from above data sources to answer the research question addressed (Twining, Heller, Nussbaum, & Tsai, 2017). Quantitative figures helped to evaluate how successfully proposed parameters could implement elasticity and dynamism whereas qualitative data provided better interpretation for the results acquired in each scenario.

<table><caption>Table 1: PyramidApp experiment settings</caption><thead><tr><td>Activity Name</td><td>Activity Description</td><td>Activity type</td><td>Target group</td><td>Additional Details</td></tr></thead><tbody><tr><td>ITIC (face-to-face class)</td><td>Three cases were given to read. Initially the class was divided into halves and gave two cases to be discussed using PyramidApp. Then the final case was given to the whole class and enacted one PyramidApp round.</td><td>Case study analysis, open-ended question answering and collaborative negotiation</td><td>First year undergraduates (N=31)</td><td>Students used only one smartphone per small group of two or three students at the first rating level.</td></tr><tr><td>ITIC (distance mode)</td><td>Watch a video (discussing ethical dilemmas in ICT) and indicate which of the 24 imperatives in the ACM code of ethics and Professional Conduct are related.</td><td>Case study analysis and collaborative negotiation</td><td>First year undergraduates (N=194)</td><td>Activity was enacted over a weekend as a homework before the next session using either smartphones or laptops individually.</td></tr><tr><td>NP (face-to-face class)</td><td>By observing the given TCP traffic, find some congestion control problems presented and explain your answer.</td><td>Problem solving and collaborative negotiation</td><td>Third year undergraduates (N=39)</td><td>Most students used smartphones individually. Activity was challenging. As expected by the educator, finally selected answer was incorrect.</td></tr><tr><td>CLAT MOOC (distance mode)</td><td>In your view, what are the main benefits of collaborative learning.</td><td>Reflections upon practices and collaborative discourse</td><td>Educators from secondary and higher education (N=617)</td><td>Heterogenous user groups with diverse expertise levels and experiences used their own devices.</td></tr></tbody></table>

## Results and discussion

Scenarios stated in Table 1 have been analyzed indicating PyramidApp authoring configurations composed by the educators over resulted values during the activity enactment. Each case shows how the introduced flow creation and control mechanisms are used for pyramid formulation and how those achieved elasticity and dynamism. In terms of the meaningful orchestration achieved through the mechanisms, it was observed that there was no violation to the rules like maximum pyramid size or collaboration group sizes. Though some participants were dropped without completing all pyramid levels, participants had at least one peer to discuss during the collaboration stages and every group finally witnessed at least one solution irrespective of the number of initial submissions. Some participants joined the activity after the flow was initiated, yet pyramids occupied late-comers without interruptions. PyramidApp authoring features suggested default field values for the practitioners like possible number of levels in a pyramid or preferred group sizes and provided activity awareness measures such as current status of the activity, different groups, their members and responses for both participants and practitioners culminating a meaningful activity flow.
## ITIC face-to-face class scenario

In this case, the educator designed three separate Pyramid flow designs (see Table 2) and each design resulted in a single pyramid after the execution also, as the class size was relatively small. Though the three pyramids were initiated with the minimum size, those had grown to occupy more students ensuring the elasticity of the designs. The satisfaction percentage provided for the activity was 60% and the two countdown timers had been activated 24 times overall. Irrespective of the countdown timer expirations students were able to still submit the initial option and the ratings without pyramids being frozen ensuring fluid, dynamic pyramid progression until the submission phase timer expires. All three pyramids had consumed around 10 minutes for the activity completion, as desired by the practitioner.

<table><caption>Table 2: Flow creation and control mechanisms – ITIC in-class activity</caption><caption>PyramidApp authoring parameters and values</caption><thead><tr><th>Pyramid ID</th><th>No. of pyramids designed</th><th>Class size</th><th>Minimum students per pyramid</th><th>No. of levels</th><th>Group size (first level of rating)</th><th>Submit timer</th><th>Submit count down timer</th><th>Rating timer</th><th>Rating count down timer</th></tr></thead><tbody><tr><td>P1</td><td>1</td><td>15</td><td>15</td><td>2</td><td>3</td><td>4 mins</td><td>2 mins</td><td>4 mins</td><td>2 mins</td></tr><tr><td>P2</td><td>1</td><td>7</td><td>7</td><td>2</td><td>3</td><td>4 mins</td><td>2 mins</td><td>4 mins</td><td>2 mins</td></tr><tr><td>P3</td><td>1</td><td>7</td><td>7</td><td>2</td><td>3</td><td>4 mins</td><td>2 mins</td><td>4 mins</td><td>2 mins</td></tr></tbody></table><p>After execution of the PyramidApp activity</p><table><thead><tr><th>Pyramid ID</th><th>No. of pyramids created</th><th>No. of logins</th><th>No. of options</th><th>Submit count down timer expiration</th><th>Submit timer expiration</th><th>No. of students rated in level 1</th><th>No. of students rated in level 2</th><th>Rating count down timer expiration (both levels)</th><th>Rating timer expiration (both levels)</th></tr></thead><tbody><tr><td>P1</td><td>1</td><td>18</td><td>15</td><td>3</td><td>3</td><td>16</td><td>15</td><td>13</td><td>1</td></tr><tr><td>P2</td><td>1</td><td>12</td><td>6</td><td>3</td><td>6</td><td>10</td><td>10</td><td>3</td><td>2</td></tr><tr><td>P3</td><td>1</td><td>10</td><td>4</td><td>2</td><td>6</td><td>8</td><td>8</td><td>0</td><td>2</td></tr></tbody></table>

## NP face-to-face class scenario

This is also a face-to-face classroom scenario in which the educator designed only one pyramid, but the flow creation rules replicated the design and created two pyramids on-demand to accommodate all activity participants (see Table 3). Though the educator had expected 40 students for the activity, only 32 were present on that day and that miscalculation had no effect on the PyramidApp enactment due to the elasticity mechanisms applied by the application. Here also the satisfaction percentage was set to 60% and 15 submissions had been done after the countdown timer was initiated and 19 students had rated after seeing the rating countdown timer. The fact that every student not rating both levels had no effect in the pyramid progression due to the dynamism mechanisms proposed which provided flexible orchestration till the final level of pyramids. Both pyramids consumed around 9 minutes to complete all the levels including discussions. The task was authored with more time for submission and rating deliberately by the educator as the task was very challenging and wanted students to fail, to establish the conditions of a motivated and rich discussion in the classroom about why they failed, and which would be the right answer. Students enjoyed the activity irrespective of being failed to answer.

<table><caption>Table 3: Flow creation and control mechanisms –NP in-class activity</caption><caption>PyramidApp authoring parameters and values</caption><thead><tr><th>Pyramid ID</th><th>No. of pyramids designed</th><th>Class size</th><th>Minimum students per pyramid</th><th>No. of levels</th><th>Group size (first level of rating)</th><th>Submit timer</th><th>Submit count down timer</th><th>Rating timer</th><th>Rating count down timer</th></tr></thead><tbody><tr><td>P1</td><td>1</td><td>40</td><td>16</td><td>2</td><td>6</td><td>5 mins</td><td>2 mins</td><td>3 mins</td><td>1 min</td></tr></tbody></table><p>After execution of the PyramidApp activity</p><table><thead><tr><th>Pyramid ID</th><th>No. of pyramids created</th><th>No. of logins</th><th>No. of options</th><th>Submit count down timer expiration</th><th>Submit timer expiration</th><th>No. of students rated in level 1</th><th>No. of students rated in level 2</th><th>Rating count down timer expiration (both levels)</th><th>Rating timer expiration (both levels)</th></tr></thead><tbody><tr><td>P1</td><td>2</td><td>16</td><td>11</td><td>9</td><td>3</td><td>12</td><td>9</td><td>11</td><td>2</td></tr><tr><td>P2</td><td></td><td>16</td><td>10</td><td>6</td><td>4</td><td>12</td><td>14</td><td>8</td><td>2</td></tr></tbody></table>

## ITIC distance scenario

This distance mode of PyramidApp activity was enacted as a homework task over a weekend in a relatively large class (n=194). As given in Table 4, the educator designed a distance mode pyramid activity and assigned 16 as the minimum pyramid size. The activity resulted in 6 pyramids that have grown till the maximum volume successfully complying with proposed elasticity mechanisms. Longer timers were assigned for submission and rating since this activity extended over two days in the distance mode. As for the flow control mechanisms, this
version of distance mode PyramidApp did not implement the countdown timer based on the satisfaction percentage. Instead, it showed the remaining timer notification for each phase from the beginning of the activity and notified students via email notifications. Hence, the analysis considers only the submission and rating timer expiration. Still, that did not affect the flow of the pyramids. In all pyramids, number of students rated and collaborated in the second level is lesser than the first level. Around 48% could not complete the submission phase (in P2, P3, P4) because they were added to the on-going pyramids as they had accessed the activity late. Yet, these students were given chance to participate and present opinions in the rating stages assuring meaningful orchestration along the Pyramid flow, rather letting them to be idle till the next activity is available. If students login after pyramid creation timestamp (i.e., submission timer expired), they are straightaway added to the next available on-going pyramid, allowing them to collaborate in rating and discussion stages.

<table><caption>Table 4: Flow creation and control mechanisms – ITIC distance activity</caption><caption>PyramidApp authoring parameters and values</caption><thead><tr><th>Pyramid ID</th><th>No. of pyramids designed</th><th>Class size</th><th>Minimum students per pyramid</th><th>No. of levels</th><th>Group size (first level of rating)</th><th>Submit timer</th><th>Rating timer</th></tr></thead><tbody><tr><td>P1</td><td>1</td><td>120</td><td>16</td><td>2</td><td>5</td><td>18 hrs</td><td>12 hrs</td></tr></tbody></table><table><caption>After execution of the PyramidApp activity</caption><thead><tr><th>Pyramid ID</th><th>No. of pyramids created</th><th>No. of logins</th><th>No. of options</th><th>Submit timer expiration</th><th>No. of students rated in level 1</th><th>No. of students rated in level 2</th><th>Rating timer expiration (both levels)</th></tr></thead><tbody><tr><td>P1</td><td rowspan="6">6</td><td>31</td><td>10</td><td>21</td><td>23</td><td>16</td><td>7</td></tr><tr><td>P2</td><td>31</td><td>16</td><td>15</td><td>27</td><td>19</td><td>1</td></tr><tr><td>P3</td><td>31</td><td>16</td><td>15</td><td>23</td><td>16</td><td>4</td></tr><tr><td>P4</td><td>31</td><td>16</td><td>15</td><td>27</td><td>18</td><td>2</td></tr><tr><td>P5</td><td>31</td><td>25</td><td>6</td><td>21</td><td>18</td><td>4</td></tr><tr><td>P6</td><td>31</td><td>20</td><td>11</td><td>13</td><td>11</td><td>14</td></tr></tbody></table>

## CLAT MOOC scenario

In the MOOC scenario, the educator designed a Pyramid collaborative activity that extended to three pyramids at the end of the course (see Table 5). Only P1 had the minimum number of participants (four students) whereas P2 and P3 had occupied more, preserving the elasticity properties introduced. Here also the distance version of the PyramidApp including timers only for submission and rating along with the email notifications was used. Here also the timers were quite longer than in a usual face-to-face class. Around 67% from the participants had submitted their options and 60% had rated at least one level of the activity. Though, the activity participation is not equal among all students, the pyramids fluidly finished with no freezing in any branch.

<table><caption>Table 5: Flow creation and control mechanisms – CLAT MOOC distance activity</caption><caption>PyramidApp authoring parameters and values</caption><thead><tr><th>Pyramid ID</th><th>No. of pyramids designed</th><th>Class size</th><th>Minimum students per pyramid</th><th>No. of levels</th><th>Group size (first level of rating)</th><th>Submit timer</th><th>Rating timer</th></tr></thead><tbody><tr><td>P1</td><td>1</td><td>4</td><td>4</td><td>2</td><td>2</td><td>2 hrs</td><td>2 hrs</td></tr></tbody></table><table><caption>After execution of the PyramidApp activity</caption><thead><tr><th>Pyramid ID</th><th>No. of pyramids created</th><th>No. of logins</th><th>No. of options</th><th>Submit timer expiration</th><th>No. of students rated in level 1</th><th>No. of students rated in level 2</th><th>Rating timer expiration (both levels)</th></tr></thead><tbody><tr><td>P1</td><td rowspan="3">3</td><td>4</td><td>3</td><td>1</td><td>1</td><td>2</td><td>2</td></tr><tr><td>P2</td><td>6</td><td>3</td><td>3</td><td>2</td><td>3</td><td>3</td></tr><tr><td>P3</td><td>5</td><td>4</td><td>1</td><td>4</td><td>1</td><td>1</td></tr></tbody></table>

## Conclusion

This paper offers a technology-oriented contribution to the orchestration technology research line within CSCL, aiming at facilitating the application of collaborative learning scripts from small to large settings in a flexible way. We have defined script flexibility in terms of *elasticity* and *dynamism* of the collaborative learning activity flow. Moreover, we have proposed mechanisms to address them for the case of scripts structured according to the Pyramid or Snowball collaborative learning flow pattern. A set of flow creation mechanisms (e.g. allocation of participants to pyramid groups considering ranges in desired group sizes) has been defined to enable the elastic incorporation of a varying number of participants to a Pyramid flow in an automatic manner. Flow control mechanisms (including timers and satisfaction parameters) are proposed to maintain a fluid, dynamic pyramid flow execution. These mechanisms have been implemented in the PyramidApp tool. Validation of PyramidApp across different educational settings showed that the flow creation and control mechanisms