# The Knowledge Building Network Pilot Project: An Exploration of Emergent Designs to Enhance Collective Teacher Efficacy

Leanne Ma, OISE/University of Toronto, leanne.ma@mail.utoronto.ca

Monica Resendes, OISE/University of Toronto, monicaresendes@gmail.com

Marlene Scardamalia, OISE/University of Toronto, marlene.scardamalia@utoronto.ca

Karen Dobbie, Ontario Ministry of Education, karen.dobbie@ontario.ca

**Abstract:** Ontario's Leading Student Achievement Project aims to foster professional collaboration among principals, vice-principals, and teachers in order to build capacity for improving student achievement and well-being. During the 2017-2018 school year, a multi-level, multi-district KB network was initiated to spread Knowledge Building in elementary schools across the province. This study traces the evolution of KB and KF practices in relation to changes in teacher efficacy and positive student outcomes. Over the span of a school year, teachers co-designed principle-based practices with their students to foster collective discourse, idea improvement, community norms, and meta-cognition/meta-talk in their classrooms. Results indicate that the initiative was a success, with the majority of educators reporting professional growth through collaboration, as well as willingness to lead their own KB network the next year.

## Introduction

Teacher efficacy is defined as “teachers’ perceptions of their own ability to bring about desired student outcomes” (Takahashi, 2011). Research has consistently demonstrated a strong relationship between teacher efficacy, classroom practices, and student achievement (e.g., Hattie, 2012; Goddard, Hoy, & Woolfolk Hoy, 2000). Teachers with high efficacy set more challenging goals, show greater effort and persistence, and are more willing to try new teaching approaches (Tschannen-Moran & Woolfolk Hoy, 2001). Studies also indicate that teachers’ individual efficacy is interlinked with collective efficacy (Schunk & DiBenedetto, 2006), where collective teacher efficacy refers to the collective perception that “teachers in a given school make an educational difference to their students” (Tschannen-Moran & Barr, 2004). Enabling conditions of collective teacher efficacy include shared goals (Kurz & Knight, 2003), group mastery experiences (Goddard, 2001), and opportunities for teacher leadership (Derrington & Angelle, 2013). Thus, school administrators (i.e., principals and vice-principals) play a vital role in fostering collective teacher efficacy (Ross & Gray, 2006; Leithwood & Jantzi, 2008). Taken together, this line of work suggests that teacher efficacy, both at the individual and collective levels, are important for enhancing teaching practices and student achievement.

Over the past 13 years, the Ontario Principals' Associations (1), in partnership with the Ministry of Education, have been fostering professional collaboration among principals, vice-principals, and teachers in order to build their collective capacity for improving student achievement through the Leading Student Achievement: Networks for Learning project (LSA). The project is guided by the evolving “LSA Theory of Action” (Leithwood, 2018), which highlights conditions for school success along the rational path (e.g., Knowledge Building), the emotional path (e.g., collective efficacy), the organizational path (e.g., professional learning communities), and the family path. As such, one key priority for the LSA project is to create professional learning networks for Knowledge Building/knowledge creation. This study explores supports for implementing Knowledge Building at scale by tracing the evolution of classroom practices in relation to changes in teacher efficacy and positive student outcomes.

## The LSA Knowledge Building network pilot project

During the 2017-2018 school year, a pilot project was created for educators to learn, share, co-design, and refine Knowledge Building practices that supported student learning and well-being. 32 teachers and 8 administrators from 17 elementary schools across 8 boards were brought together to create a multi-level, multi-district network to promote the cross-fertilization of ideas and spread of classroom innovations across the province. Each teacher was committed to implementing Knowledge Building (KB) pedagogy and Knowledge Forum (KF) technology in at least one subject area, such as math, science, or social studies. Members of the network met five times between September and May to engage in collaborative design of KB and KF practices. Throughout the year, teachers were provided with customized capacity-building sessions and on-site KF support as often as needed to help advance their design work. Data sources for this study include: teachers' KB practices presented at bi-monthly
meetings and teachers' responses to end-of-term surveys, including their reflections about the design process and observations of how their practices impacted their students.

## Knowledge Building practices

One of the initial challenges of the pilot project is introducing Knowledge Building (Scardamalia & Bereiter, 2014), which, unlike other pedagogical approaches, adopts a principle-based approach to classroom design. Many teachers found the conceptual complexity of the 12 KB principles (Scardamlia, 2002) to be less than intuitive and struggled with translating them into practice. To help scaffold the development of their understanding, teachers were encouraged to study the KB Gallery (Resendes & Dobbie, 2017), as well as multimedia resources on http://thelearningexchange.ca, including: descriptions of the 12 KB principles, concrete tools and strategies for getting started in the classroom, video series featuring teacher, student, and parent testimonials, and podcast series to provoke reflection and classroom planning. The bi-monthly meetings served as opportunities to discuss their attempts at principle-based classroom design (both successful and failed ones), ask questions and troubleshoot, receive feedback from peers, and plan for their next iteration.

<figure><img src="image_59.png" /><figcaption>Figure 1. a) KB wall, b) KB scaffolds, c) KB table, and d) Improvable Ideas board.</figcaption></figure>

Figure 1 shows some KB practices invented by Ontario teachers in the companion guide to the KB Gallery, including: a) KB walls, which visualizes connections in the community knowledge; b) KB scaffolds, which promotes diverse contributions to the community discourse; c) KB table, which makes student ideas visible to all members of the community; and d) Improvable Ideas board, which shows design iterations on student ideas. These practices have the shared goal of bringing student ideas to the center of classroom activities and fostering students' sense of collective responsibility for knowledge advancement (Scardamalia, 2002). Put differently, these practices help teachers realize various KB principles in their classroom, such as *community knowledge, collective responsibility, Knowledge Building discourse, democratizing knowledge, and improvable ideas*. The year-long challenge for network participants was to adopt these practices into their own classrooms and schools, refine them through deepening engagement with all 12 KB principles, as well as co-design new KB practices with their students.

## KB network meetings

During the KB network meetings, 3 teachers, 3 teaching teams, and 6 school teams presented their classroom designs and re-designs. Each presentation highlighted the implementation of KB practices in areas such as math, science, history, geography, health, and robotics. On average, each classroom design included 4 to 5 practices (Figure 2). The most commonly used practices were: KB scaffolds (9), KF community (9), KF analytic tools (8), KB walls (7) and community norms (6).

<figure><img src="image_60.png" /><figcaption>Figure 2. KB practices used by elementary network teachers.</figcaption></figure>
It can be seen that neither face-to-face practices (e.g., KB walls, KB circles, KB table) nor online practices (e.g., KF community, KF analytic tools) alone could account for half of KB practices presented, suggesting that most teachers adopted a blended approach to their KB practices. In particular, KB scaffolds and community norms were used in both face-to-face and online contexts, which points to their importance in facilitating the integration of KF technology with KB practices in the classroom.

Given the fact that most teachers implemented multiple KB practices at once, the average classroom design reflected 8 KB principles. The most common KB principles translated into practice were: *idea diversity* (12), *KB discourse* (11), *democratizing knowledge* (11), *improvable ideas* (10), and *real ideas, authentic problems* (9). Whereas presentations at the beginning of the school year mostly focused on *real ideas, idea diversity*, and *KB discourse*, presentations toward the end of the school year focused more on *community knowledge, democratizing knowledge*, and *improvable ideas*. This supports the notion that while it is easy to engage students in idea generation, it is challenging to engage them in idea improvement (Scardamalia & Bereiter, 2003). One common practice that teachers consistently used to foster *improvable ideas* was the integration of KB scaffolds into day-to-day class discussions.

<figure><img src="image_61.png" /><figcaption>Figure 3. KB principles used by elementary network teachers.</figcaption></figure>

Taken together, it appears that *KB discourse*, which aims to shift engagement norms of idea generation toward idea improvement, is a fundamental principle to fostering and sustaining a KB community regardless of students' grade or subject area.

It is also interesting to note that the least common KB principle in classroom designs was *rise above*, which reinforces the overarching challenge of seeking conceptual coherence among diverse ideas in the community knowledge. Indeed, one math teacher attempted to design for *rise above*. After studying formulae for area and perimeter of 2D shapes, students were encouraged to develop theories for how to calculate the surface area and volume of 3D shapes. The teacher provided students with a *real, authentic problem* of filling the volume of a sinkhole to see whether students could create formulae for rectangular prisms and cylinders. Students were so captivated with the problem that in addition to finding the formula for a cylinder (including seeking *authoritative sources* to learn about circumference of circles), they were creating formulae to calculate the cost of materials and labour to fix the hole! For another science teacher, *rise above* emerged naturally as students were studying biodiversity in science and climate change in social studies. After spending weeks exploring the relationships between predators, preys, and invasive species in various ecosystems, as well as discussing how humans have an impact the environment, students brought their *symmetric knowledge advances* together to develop the *rise above* theory that humans are an invasive species.

It can be seen that complex, interdisciplinary problems give rise to rich discussions in the classroom. Therefore, it comes as no surprise that deeper KB practices (i.e., involving multiple KB principles) are more likely to occur when teachers implement KB across multiple subject areas. Statistical analyses reveal a positive correlation between the number of subject areas a teacher used for KB and the number of KB principles reflected in their designs (r = 0.76, p > 0.01), as well as the number of KB practices a teacher used and the number of KB principles reflected in their designs (r = 0.74, p > 0.01). This appears to confirm the notion that Knowledge Building requires an enculturation approach to principle-based practice – the more opportunities teachers and students have to engage in sustained, creative work ideas, the deeper they go with KB, eventually unlocking all 12 principles (Scardamalia & Bereiter, 2003; Resendes & Dobbie, 2017).
## KB network surveys

At the end of the fall and winter terms, teachers reflected on their KB practices and completed surveys (27 responses were collected in fall, 25 responses were collected in winter). Using the KB progressions in the KB Gallery resource, teachers rated their progress along four dimensions of KB practice: fostering collective discourse (orange), community norms (yellow), developing ideas (green), and meta-cognition/meta-talk (blue). Figure 4 shows the proportion of teachers in early stages (light colours), developing stages (medium colours), and deepening stages (dark colours) for each dimension of practice. In the fall term, the majority of teachers were in early practice, whereas in the winter term, the majority of teachers were in developing practice. Change scores reveal a 55% decrease in early practices and 68% increase in developing practices between the two terms, suggesting that teachers were engaged in continual development of their KB practices throughout the year.

The survey also provided opportunities for teachers to elaborate on their KB practices in open-ended questions. 20 out of 25 of teachers believed that the key community norm was fostering a sense of safety and community. This includes promoting a growth mindset, openness to different perspectives, respectful communication, celebrating risks, and valuing mistakes. Whereas in the fall term, more teacher-directed practices were reported to scaffold students into KB practices, in the winter term, many teachers were engaging students in co-design of KB practices. For example, in the fall, many teachers created posters and cue cards of KB scaffolds while modelling how to use them during class discussions. In the winter, they listened for students' use of the KB scaffolds while encouraging them to design new ones to advance the community discourse and expand the class's repertoire of KB scaffolds (e.g., "I wonder", "Another way of thinking...", "We should revise this..."). Some teachers were even beginning to recognize curricular connections, suggesting that they were shifting toward deeper practices. Halfway through winter term, one health teacher realized that implementing KB across the curriculum would have been less effortful on her part and more rewarding for the students than carving out their designated weekly KB time. Recall that in the teacher presentations, deeper KB practices involved implementing KB across multiple subject areas.

<figure><img src="image_62.png" /><figcaption>Figure 4. Teacher self-assessments on KB Progressions.</figcaption></figure>

Regarding the most common practices used for fostering community discourse, developing ideas, and meta-cognition/meta-talk, KB scaffolds, KB circles and KB walls were most commonly used in fall, whereas KB scaffolds, KB circles and KF were most commonly used in winter. Given the considerable overlap in responses across the four dimensions, it is likely that the all-in-one nature of KF (i.e., KB scaffolds for collective discourse, endless KB walls for developing ideas, analytic tools for meta-cognition/meta-talk) helped teachers deepen their KB practices. For example, while teachers were encouraging students to use KB scaffolds across both terms, it was not until when students were using the KB scaffolds in KF notes that teachers were able to assess student contributions through the analytic tools. When students were provided with the opportunity to reflect on their contributions through the analytic tools, both the quality and quantity of student engagement was enhanced. Students were excited about the analytic tools and would self-initiate their use to regulate their own contributions. Put differently, the intentional integration of KF into KB practices reflects teachers' growing understanding of the KB principles and their increasing efficacy in balancing the face-to-face and online affordances for advancing community knowledge. Below are a few teachers' reflections on the impact of their KB practices on student learning and well-being:
"Through KB circles students are beginning to build on each other's ideas while discussing our emotions and ways that we can regulate our emotions."

"KF allowed students with anxiety to feel like an important, active member of the community by sharing important ideas that others actually read!"

"Students are more engaged – students have a voice and want to share their ideas and communicate their thinking. The students are gaining new tools for their tool belt... and are able to ask questions and reflect on their peers' process as well as their own learning."

Overall, responses in the end-of-year survey indicate that both teachers and administrators found the network sessions immensely helpful for: deprivatizing classroom practices; troubleshooting and advancing their designs; hearing from different school boards, roles, and perspectives; exposing them to new ideas and strategies; and opening new opportunities for collaboration. Educators at all levels almost unanimously (all except one who was unsure) expressed interest in continuing participation in the multi-district KB network the following year, with 20 out of 25 willing to start a smaller KB network in their schools and/or school boards. Below are some additional insights and reflections from network participants:

"It is always great to hear ideas from other schools and boards, and different perspectives (teacher, admin, students). It is inspiring to learn with educators from other areas. This session gave me the opportunity to reflect on my classroom and program and consider how to further develop student thinking and deepen practice."

"Being able to talk to other teachers and having discussions around how to approach some of my challenges. Learning new ways to use KB in my classroom. It validates my pedagogy and has helped to re-invent myself as an educator."

"I'd like to continue having teachers, students, and administrators share what is happening inside our classrooms and provide time for us to pair up and work together across the regions when working on like ideas."

"Interesting to hear where other schools and other boards are in this journey. Different schools/boards are strong in different areas and it is great to share our information to help the teaching/learning community as a whole develop and grow together."

It can be seen that teachers were taking ownership of their classroom practices. Furthermore, the network sessions helped them see their own practices as improvable, and over time, they were assuming collective responsibility for advancing the design work of their community. In other words, members of the KB network demonstrated intentional efforts to raise their individual efficacy and collective efficacy. As an outcome of this pilot project, 16 teachers and administrators attended the KB Summer Institute to share their advances and explore possibilities for the 2018-2019 school year. This experience helped them reframe their practices around less commonly targeted KB principles over the past year, such as *community knowledge*, *pervasive KB*, and *symmetric knowledge advancement*. They are now linking with international KB communities and building their own networks to deepen their practices and grow capacity in their classrooms, schools, and boards. In the words of one teacher:

"My goal is to continue to have my students... participate in this network universally by having them make effective contributions during online discourse (Pervasive KB) on the existing global issues that prevail today. It's phenomenal observing what solutions/innovations these students come up with that could potentially one day benefit the Public Good!"

## Conclusions and implications

Multi-level networks and partnerships are powerful mechanisms for building capacity as well as spreading Knowledge Building within a system (Laferriere, et al., 2010; Chan, 2011). After the success of the KB Tri-Board Project (Resendes et al., 2016), Knowledge Building has spread across the province to involve more than 11 districts. The purpose of the 2017-2018 KB Network pilot project was to create opportunities and supports for teacher collaboration and design within and across districts in Ontario. Over the span of one school year, teachers
adapted existing KB practices – such as KB scaffolds, KB walls, KB circles, KB walls, and KF communities – into their classrooms and refined those practices with their students.

In working toward translating the 12 KB principles into meaningful classroom designs, teachers were engaged in the conceptual work of shifting the relations between ideas, students, and themselves in the classroom (Teo, 2014; Toth & Ma, 2018). The more KB principles they integrated into their designs, the deeper they went with their KB practices, the more engaged their students. At the same time, the bi-monthly network sessions allowed teachers to deepen their understanding of the KB principles by discussing concrete practices from other teachers' classrooms and reflecting on the adaptability of their classroom designs. Studies on KB in professional learning communities indicate that collaborative design helps teachers become more open and proactive toward their practices (Vokatis & Zhang, 2016), as well as orient towards more innovative practices (Hong, Chai, & Hung, 2015).

This study suggests that the LSA KB network pilot project successfully provided enabling conditions for teachers to engage in collaborative design as means to improve their KB practices. Consistent with past research in professional development (Hoy, 2000), both direct experiences and vicarious experiences were conducive to enhancing teacher efficacy. Recall that teacher efficacy is key to improving teaching and student learning (Takahashi, 2011) while simultaneously enhancing professional commitment (Ware & Kitsantas, 2007). This study suggests that Knowledge Building goes beyond fostering teacher efficacy to igniting both teachers' and students' epistemic agency. While teacher reports indicate that their practices have a positive effect on students' engagement, learning, and well-being, additional analyses are underway to further understand the effects of sustained participation in dynamic multi-level, multi-district KB networks on student achievement outcomes.

## References

Chan, C. K. (2011). Bridging research and practice: Implementing and sustaining knowledge building in Hong Kong classrooms. *International Journal of Computer-Supported Collaborative Learning, 6*(2), 147-186.

Derrington, M. L., & Angelle, P. S. (2013). Teacher leadership and collective efficacy: Connections and links. *International Journal of Teacher Leadership, 4*(1), n1.

Goddard, R. D. (2001). Collective efficacy: A neglected construct in the study of schools and student achievement. *Journal of Educational Psychology, 93*(3), 467.

Goddard, R. D., Hoy, W. K., & Hoy, A. W. (2000). Collective teacher efficacy: Its meaning, measure, and impact on student achievement. *American Educational Research Journal, 37*(2), 479-507.

Hattie, J. (2012). *Visible learning for teachers: Maximizing impact on learning*. Routledge.

Hong, H. Y., Chai, C. S., & Hung, G. T. (2015). Principle-based guidance to foster adaptive teaching practice. In O. Lindwall, P. Hakkinen, T. Koschman, P. Tchounikine, & S. Ludvigsen (Eds.), *Proceedings of the 11<sup>th</sup> International Conference on Computer Supported Collaborative Learning* (pp. 47-54). Gothenburg, Sweden: ISLS.

Hoy, A. W. (2000, April). Changes in teacher efficacy during the early years of teaching. Paper presented at the *AERA Annual Meeting, New Orleans, LA*.

Kurz, T. B., & Knight, S. L. (2004). An exploration of the relationship among teacher efficacy, collective teacher efficacy, and goal consensus. *Learning Environments Research, 7*(2), 111-128.

Laferrière, T., Montane, M., Gros, B., Alvarez, I., Bernaus, M., Breuleux, A., ... &

Lamon, M. (2010). Partnerships for knowledge building: An emerging model. *Canadian Journal of Learning and Technology, 36*(1).

Leithwood (2018). *Leadership Development on a Large Scale: Lessons for Long-Term Success*. Corwin.

Leithwood, K., & Jantzi, D. (2008). Linking leadership to student learning: The contributions of leader efficacy. *Educational Administration Quarterly, 44*(4), 496-528.

Resendes, M., Scardamalia, M., Cordeiro, M., Massey, L., Dobbie, K., & Sirois, L. (2016). A Tri-Level Partnership to Support and Spread Knowledge Building in Ontario. In Looi, C. K., Polman, J. L., Cress, U., and Reimann, P. (Eds.). *Proceedings of the 12<sup>th</sup> International Conference of the Learning Sciences* (pp. 1249-1250). Singapore: ISLS.

Resendes, M. & Dobbie, K. (2017). *Knowledge Building Gallery: Teaching for Deep Understanding and Community Knowledge Creation*. D. Maika, E. Hine, L. Ma, E. Heaver, & L. White (Eds.). Retrieved from https://thelearningexchange.ca/wp-content/uploads/2017/04/Knowledge-Building-Booklet-Accessible-1.pdf

Ross, J. A., & Gray, P. (2006). School leadership and student achievement: The mediating effects of teacher beliefs. *Canadian Journal of Education, 79*(8-22).
Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith, & C. Bereiter (Eds.), *Liberal education in a knowledge society* (pp. 67-98). Berkeley, CA: Distributed by Publishers Group West.

Scardamalia, M., & Bereiter, C. (2003). Beyond brainstorming: Sustained creative work with ideas. *EDUCATION CANADA-TORONTO-, 43*(4), 4-8.

Scardamalia, M., & Bereiter, C. (2014). Knowledge building and knowledge creation: Theory, pedagogy, and technology. In K. Sawyer (Ed.), *Cambridge Handbook of the Learning Sciences* (pp. 397-417). New York: Cambridge University Press.

Schunk, D. H., & DiBenedetto, M. K. (2016). Self-efficacy theory in education. *Handbook of Motivation at School, 2*, 34-54.

Takahashi, S. (2011). Co-constructing efficacy: A “communities of practice” perspective on teachers’ efficacy beliefs. *Teaching and Teacher Education, 27*(4), 732-741.

Teo, C. L. (2014). Conceptual shifts within problem spaces for knowledge-building practice within a teacher community. In *Knowledge Creation in Education* (pp. 205-226). Springer, Singapore.

Toth, P. & Ma, L. (2018). Fostering student voice and epistemic agency through knowledge building. In *Proceedings of the 21st Knowledge Building Summer Institute*; Toronto, Canada: Knowledge Building International.

Tschannen-Moran, M., & Barr, M. (2004). Fostering student learning: The relationship of collective teacher efficacy and student achievement. *Leadership and Policy in Schools, 3*(3), 189-209.

Tschannen-Moran, M., & Hoy, A. W. (2001). Teacher efficacy: Capturing an elusive construct. *Teaching and Teacher Education, 17*(7), 783-805.

Vokatis, B., & Zhang, J. (2016). The Professional Identity of Three Innovative Teachers Engaging in Sustained Knowledge Building Using Technology. *Frontline Learning Research, 4*(1), 58-77.

## Endnotes

(1) l'Association des directions et directions adjointes des écoles franco-ontariennes, Catholic Principals' Council | Ontario, and Ontario Principals' Council

## Acknowledgments

We are grateful to Knowledge Building educators and consultants, Dr. Rob Iannuzzi, Elaine Hine and Denis Maika for their time and support. Their creative ideas and insight have been invaluable to our Knowledge Building.
# Comparing the Effectiveness of Supports for Collaborative Dialogic Sense-Making with Agent-Based Models

Ilana Dubovi, Ben-Gurion University of the Negev, dubovi@post.bgu.ac.il
Victor, R. Lee, Utah State University, victor.lee@usu.edu

**Abstract:** Collaborative dialogue has been classified by the ICAP framework (Chi & Wylie, 2014) as the highest interactive level of cognitive engagement. In this study we evaluate the balance between type of guidance and learner characteristics while collaborative learning with agent-based models takes place. Participants were students from health-care programs and were randomly assigned to one of two conditions: one group learned with agent-based models using the vicarious approach, where pairs viewed and discussed recordings of others using agent-based models; the other group explored agent-based models while discussing in pairs a set of text-based prompts. The results reveal that the vicarious-learning approach was superior to agent-based models' exploration. More detailed evaluation shows that nursing students' performance wasn't affected by the type of guidance, while less experienced in the content, non-nursing students, benefited more from vicarious guidance. Findings suggest that dialoguing while observing the dialogue of others can maximize immediate learning gains.

**Keywords:** agent-based models, vicarious learning, collaboration, ICAP framework, dialogue

## Introduction

According to Chi's ICAP framework (Interactive, Constructive, Active, and Passive), collaborative dialoguing is the highest interactive level of cognitive engagement (Chi et al., 2018; Chi & Wylie, 2014). While differentiating modes of cognitive engagement on the basis of students' overt behaviors, dialogue can be considered truly interactive only when it consists of mutual exchanges of ideas between two (or more) individuals, resulting in new ideas that neither individual knew initially nor could generate alone. Computer-based simulations and models can provide excellent opportunities to generate constructive dialogue with the aim to promote more in-depth conceptual understanding of science content, especially when complex systems are involved (D'Angelo et al., 2014). Of special interest to this study is agent-based modeling (ABM). Joint exploration of agent-based models allows students to test their ideas, negotiate, argue a position and provide justifications, compare, and revise one another's understandings based on the dynamic feedback provided by the running simulation model (Wilensky & Reisman, 2006). Yet, to achieve these potential benefits of learning with agent-based models, there is a need for learning activities that provide appropriate guidance and support interactive dialoguing (see meta-analysis by Lazonder & Harmsen, 2016).

The current study seeks to compare types of support that facilitate constructive joint dialogue by accounting for some learner characteristics within the context of an instructional unit about diabetes mellitus pharmacology for health-care professions education. We compare a vicarious learning approach of observing collaboratively a dialogue between a learner and an instructor with one involving guided exploration and explicit written prompts to have dialogue related to a set of agent-based models. This study is intended to question the efficacy of vicarious learning with agent-based models by comparing it with guided exploration.

## Vicarious learning

Learning from observing, or learning vicariously, was first proposed by social psychology research (Bandura, 1969) to evaluate learning by imitating and modeling someone's else behaviors and actions. Bruner (1986) stated that "most of our encounters with the world, are not direct encounters" (p. 122), which would seem to imply that it is possible to learn through mechanisms other than primary or firsthand experience. Such social learning is effective without the need for the observer to experience feedback directly. Bandura's introduction of the idea – vicarious reinforcement – was based on classic studies in which children were seen to imitate the aggressive behavior modeled by adults as they assaulted Bobo dolls (Bandura, Ross, & Ross, 1963).

Vicarious learning has been also explored in area of learning of cognitive behaviors as asking questions (Rummel & Spada, 2005), doctor-patient communications skills (Stegmann, Pilz, Siebeck, & Fischer, 2012), and acquiring complex cognitive skills (Chi, Roy, & Hausmann, 2008). Initial research on the differences between direct participation in a task and simple observation of it being performed was conducted by Schober and Clark (1989), in which pairs of participants communicated about a task that involved sequencing figures correctly. Simply overhearing the dialogue, rather than participating directly, resulted in poorer performance, leading
Schober and Clark to conclude that the direct participants had an advantage by being able jointly to construct a common ground of comprehension. However, in a replication of Schober and Clark's study (Chi et al., 2008), in which the vicarious leaners were required to be active, by self-explaining aloud while overhearing a tutor-tutee interaction, vicarious leaners did just as well as tutees who interacted with a tutor.

To further explore vicarious learning, Chi and her colleagues tested different conditions involving different vicarious aspects – observing monologue videos alone / observing dialogue videos alone / observing monologue videos collaboratively / observing dialogue videos collaboratively (Chi et al., 2008). The results suggest that vicarious learning through observation doesn't have to be passive but might instead be considered interactive when the conditions of *collaboratively* overhearing and observing *dialogue* between a learner and an instructor are taking place. Dialogue that incorporates mutual exchanges of ideas between individuals will result in new ideas that neither individual knew initially nor could generate alone. Therefore, the ICAP theoretical framework, which categorizes different modes of active learning, defines joint dialoguing as the highest interactive level of cognitive engagement (Chi & Wylie, 2014). The advantage of designing vicarious learning with the dual discourse approach, collaboratively observing a dialogue, is twofold. First, observing collaboratively facilitates interactions between peers that offer an opportunity for exchanges of ideas and argumentation processes which foster learning (Schwarz, Neuman, & Biezuner, 2000). Second, observation of dialogues over monologues makes it possible for vicarious learners to overhear the tutee's questions and struggles and to reflect on their own mental models using the tutor's feedback as well. As a result, learning gains of collaborative observations of dialogues were similar to learning gains following face-to-face human tutoring (Muller, Sharma, & Reimann, 2008). This study is aimed at adding to the existing domain of knowledge on the effectiveness of vicarious learning (Chi et al., 2018) by comparing pairs collaboratively observing videos vicariously with pairs following guided exploration coupled with explicit prompts to dialogue about agent-based models within the antidiabetic pharmacology education context.

## Agent-Based Modeling environment

ABM is a computational modeling paradigm that emphasizes multi-level examination of complex, multi-agent systems. The ABM paradigm encodes the behavior of individual agents in a small set of simple rules so that we can specify and observe the results of these agents' individual actions and interactions. Learning through this approach focuses on entities and their actions (also called the micro level of the system), such as movement, interactions, and global flows (also called the macro level of the system), and allows students to comprehend parallel processes by which emergent phenomena form (Wilensky & Resnick, 1999). Exploration of agent-based models encourages causal emergent thinking in connecting individual behaviors with systemic patterns, thus helping students learn various scientific concepts effectively (i.e., Samon & Levy, 2017).

Yet, to achieve these potential benefits of agent-based models exploration, research has provided conclusive evidence that students need support and guidance (D'Angelo et al., 2014; Kirschner et al., 2006). However, the educational literature presents varied types and levels of guidance, from prompts and cues to specific and direct explanations of how to perform an action (Lazonder & Harmsen, 2016). There is an ongoing debate, the “assistance dilemma”, about what type of guidance is adequate and whom it fits (Koedinger & Aleven, 2007, p. 261). On one side of this argument are those advocating direct instructional guidance as beneficial for novice learners (e.g., Kirschner et al., 2006); on the other side are those suggesting that less heavily guided instructions may be more productive, encouraging concept learning and transfer (e.g., Koedinger & Roll, 2012). Hmelo-Silver and her colleagues challenged this notion by suggesting that instead of contrasting these dichotomous positions on the continuum of guidance level, the questions to be asked are “under what circumstances do these guided inquiry approaches work, what are the kinds of outcomes for which they are effective . . . and what kinds of support and scaffolding are needed for different populations and learning goals?” (Hmelo-Silver, Duncan, & Chinn, 2007, p. 105).

In fact, regarding the exploration of agent-based models, it is still unclear how to balance the instructional guidance. On the one hand, agent-based models are exploratory models that are designed so that students can discover things by experimenting within a microworld; on the other hand, without providing explanations, understanding of complex systems won't become explicit but might remain implicit. Vitale, McBride, and Linn (2016) compared two forms of automated guidance – direct explanations and prompts to motivate learners (knowledge integration guidance) – to support learning of complex systems with agent-based models. They found that directed guidance for agent-based models exploration produced higher immediate learning gains than minimal guidance did. However, recent studies by Jacobson et al. (2017) and Levy et al. (2018, p. 2) on agent-based models' exploration for complex systems learning suggested that minimal guidance, or a “constrained discovery”, can be valuable to conceptual learning.
The current study focuses on science learning, particularly pharmacology learning, among health-care professionals in a higher education setting. Moreover, we aimed to support science learning within health-related education, which has so far occupied only a modest space in the learning sciences. Therefore, we present the Deep Dive into Diabetes (DDD) agent-based learning environment, which was constructed with the NetLogo modeling platform (Wilensky, 1999). The proposed agent-based models, which simulate biochemical processes that represent the relevant anatomy of glucose equilibrium and the mechanisms of drug action, were constructed in a previous study (Dubovi, Dagan, Mazbar, Nassar & Levy, 2018). To explore the effect of guidance level, the DDD learning environment was constructed with two different levels of learning activities that support learning with agent-based models (Figure 1): (1) collaborative observation of a video tutorial, a vicarious approach; and (2) collaborative exploration and experimentation with agent-based models, a guided-exploration approach.

<figure><img src="image_63.png" /><figcaption>**Figure 1.** DDD learning environment: (a) Guided-exploration condition: screenshot, and photo of students learning with the environment; (b) Vicarious condition: example of conversation that students were overhearing while learning with the video tutorial, and photo of students learning with the environment.</figcaption></figure>

# Methods

## Research design

This study employed a quasi-experimental pre- and post-test design with two comparison conditions to explore the effect of guidance level on inquiry learning with agent-based models. We conducted a quantitative analysis of questionnaires of all participants (70) and video analysis of 11 randomly selected dyads of students (22).

## Participants and procedure

The participants were 70 health-care undergraduate students (40 students from a nursing program, 23 from a nutrition program, and 7 from a health education program) who volunteered to participate in this study. Most participants were females (61), and the average age was $23\pm4.85$ years. Participants were randomly assigned to participate in pairs in one of two conditions: a guided-exploration (34) or a vicarious condition (36). Both conditions involved active and continuous dialogue in pairs as participants tried to interpret what was happening in the DDD learning environment. In the first condition, students were required to learn about diabetes through guided exploration of agent-based models (the guided-exploration condition); in the second condition, students learned about diabetes by observing instructor-learner dialogue and by exploring agent-based models (the vicarious condition).

There were no statistically significant differences in demographic characteristics such as grade point average (GPA) ($\chi^2=2.58, p = 0.46$) or prior knowledge about diabetes ($t=1.36, p=0.17$) between the two conditions.
On average, students spent about 50 minutes learning with the vicarious-instruction and 52 minutes learning with the guided-exploration-instruction method.

## Instruction design and materials

While learning with the DDD environment, students in the both groups were asked to use information from an agent-based model to solve problems about diabetes and antidiabetic pharmacology content. The *vicarious condition* group received guidance by observing a video tutorial of an instructor and a learner while exploring the agent-based models. A staff member at a university who was an advanced novice on diabetes and who was familiar with the NetLogo modeling system served as the instructor, and a graduate student who had passing familiarity with diabetes and the modeling system was the learner. This was an unscripted conversation. The study participants were able to overhear the dialogue between learner and instructor that included the instructor's explanations of agent-based models' representations; what to pay attention to while the model is running and how to interpret the model's output; and questions raised by the student which were followed by the instructor's feedback (Figure 1b). Similar to Chi et al.'s study, students could pause, reverse, and skip portions of the video; to enhance participants' active cognitive engagement while observing, we also followed Chi and her colleagues' vicarious learning design by asking learners to solve together several sub-problems which served as landmarks to the video and prompted interactive dialogue between the participants (Chi et al., 2008). The *guided exploration condition* group received less guidance while directly exploring the agent-based models. The scaffolds included information on how to set up the agent-models and on which graphs and monitors to pay attention to. As in the vicarious condition, to facilitate dialogue between pairs, students were asked to talk with one another and to solve sub-problems while exploring the agent-based models (Figure 1a). Their solutions to the problems were entered into a single shared computer. We used learning activities that we developed in a previous study (Dubovi et al., 2018).

## Data collection instruments

### Diabetes knowledge questionnaire

The Diabetes knowledge questionnaire was adapted from the Pharmacology Diabetes Mellitus questionnaire (PDM) developed in a previous study (Dubovi et al., 2018). The questionnaire consists of nine questions (7 multiple-choice, 2 open-ended), and evaluates understanding of biochemical glucose equilibrium, glucose disequilibrium (i.e., diabetes type 1 and diabetes type 2), and medications actions. Analysis of the Diabetes knowledge questionnaire using Cronbach's alpha yielded an internal consistency score of 0.68, which was similar to our previous report (a = 0.71) and can be considered acceptable.

Responses to the questionnaire were coded as either correct or incorrect, and the total score was calculated as the percentage of correct answers. In addition, the items on the Diabetes knowledge questionnaire were scaled by level of difficulty: four items were coded as the most difficult, and five items as the least difficult. Level of difficulty for each item was determined based on the percentage of students who correctly answered it on the post-test. Although students completed the pre- and post-tests questionnaires as individuals, their learning process was nested within their collaboration as a pair. Prior knowledge varied greatly between students as individuals as well as within pairs. To account for these variations, the analysis invoked multi-level modeling. Owing to the pre-test/post-test design used in this study, our data analysis encompassed repeated measures on individuals over time. Consequently, a three-level structure arose: both test times (Level 1) were clustered within students (Level 2), which were nested within dyads (Level 3). Although multi-level models quantified the variance across pairs, the focus of the study was on at the individual student level.

### Video recordings

To assess the learning process, we recorded students' discourse and interactions with the agent-based models using screen-recording software and a separate standing video camera. For the analysis, we randomly chose 11 pairs (22 individual students); five pairs learned with the guided-exploration condition, and six pairs learned with the vicarious condition. We evaluated the frequency of accuracy of students' ideas and explanations as they learned with the DDD environment. To generate students' ideas accuracy, discourses and dialogues were carefully transcribed, iteratively reviewed, and coded in terms of the ideas, explanations, and statements expressed. This approach to the selection of knowledge elements and ideas is comparable to the approach documented in Sherin, Krakowski, and Lee (2012) and Minstrell (1982). From this, transcript excerpts were identified to illustrate some of the dialogues and ideas expressed.
# Results

## Diabetes knowledge questionnaire scores

Multilevel model analysis was conducted to determine the effect of pre-test scores, experimental condition (vicarious vs. guided exploration), and the student's program of study (nursing vs. other health-care field) on post-test outcomes, both independently and through examining the interactions between them. Moreover, multilevel model analysis also considers the random effect of individual and pair characteristics on factors' interactions. As shown in Table 1, after adjusting for differences between individuals and dyads, the overall post-test score for the Diabetes knowledge questionnaire was the sum of the intercept (37.318). A significant interaction of Time × Experimental condition indicates that the vicarious-condition pre-test to post-test learning gains were significantly higher than those for the guided condition (34.22 to 67.47 vs. 40.23 to 61.84).

Examining the level of the Diabetes knowledge questionnaire items independently, being in a nursing program versus another health-care program moderates the effect of experimental condition within the most difficult items (Table 1). More specifically, students from non-nursing programs made significantly higher learning gains when participating in the vicarious condition compared with the guided-exploration condition (8.93 to 60.71 vs. 18.53 to 38.84, respectively; Figure 2), whereas students from the nursing program gained knowledge from learning with both conditions, with no significant differences (Figure 2). This effect was true only for the most difficult questionnaire items; for the least difficult items, the interaction Time × Experimental condition was insignificant, meaning that students showed similar learning gains from learning with the vicarious (nursing: 51.60 to 65.39; non-nursing 57.84 to 71.63) and the guided-exploration conditions (nursing: 52.13 to 65.91; non-nursing 58.37 to 72.16).

<table><caption>Table 1: Three-level nested random-intercepts multilevel model predicting students' Diabetes knowledge questionnaire post-test scores</caption><thead><tr><th></th><th>Overall<br>Questionnaire</th><th>The Most Difficult<br>Items</th><th>The Least Difficult<br>Items</th></tr></thead><tbody><tr><td><strong>Fixed Effects</strong></td><td>Estimate (SE)</td><td>Estimate (SE)</td><td>Estimate (SE)</td></tr><tr><td>Intercept</td><td>35.279 (3.839) ***</td><td>18.535 (5.764)**</td><td>51.762 (4.389)***</td></tr><tr><td>Time: Post vs. Pre</td><td>21.609 (3.187) ***</td><td>20.313 (6.346)**</td><td>13.786 (2.771)*</td></tr><tr><td>Experimental condition: Vicarious vs. Guided</td><td>-6.010 (4.543)</td><td>-9.602 (8.428)</td><td>-0.941 (4.669)</td></tr><tr><td>Program: Nursing vs. Other</td><td>8.671 (3.859)*</td><td>-0.288 (7.817)</td><td>7.264 (4.706)</td></tr><tr><td><strong>Interactions:</strong></td><td></td><td></td><td></td></tr><tr><td>Time × Experimental condition</td><td>11.647 (4.444) **</td><td>31.473 (9.290)***</td><td>-</td></tr><tr><td>Time × Program</td><td>-</td><td>28.298 (8.722)**</td><td>-</td></tr><tr><td>Experimental condition × Program</td><td>-</td><td>2.716 (11.029)</td><td>-</td></tr><tr><td>Time × Experimental condition × Program</td><td>-</td><td>-24.403 (12.304)*</td><td>-</td></tr><tr><td><strong>Random Effects</strong></td><td>Var</td><td>Var</td><td>Var</td></tr><tr><td>Dyads</td><td>79.364</td><td>65.575</td><td>231.849</td></tr><tr><td>Participants within Dyad</td><td>53.560</td><td>79.733</td><td>6.196</td></tr><tr><td>Residual</td><td>172.678</td><td>322.206</td><td>268.727</td></tr></tbody></table>

Note: Sample size is 35 dyads made up of 70 participants with 140 total observations.

Each model contains only significant interactions.

\*p<0.05, \*\*p<0.01, \*\*\*p<0.001.
A close look at the correctness of students’ ideas revealed that students who learned with the guided-exploration condition (five pairs) expressed 41 inaccurate ideas, whereas students who learned with the vicarious condition (six pairs) expressed only seven inaccurate explanations. Most of the inaccurate explanations with the guided-exploration condition resulted from misinterpreting what to pay attention to while exploring the agent-based models.

## Discussion

The most interesting result is the advantage of vicarious learning for immediate learning achievements over the guided-exploration condition. One possible explanation for this effect is that learning with the guided-exploration condition triggered more inaccurate ideas and explanations, even though the number of ideas that students expressed was not affected by the experimental condition. Interestingly, a more precise analysis reveals that there are differential instructional effects across students’ characteristics. Evaluation of the more difficult items of the Diabetes knowledge questionnaire shows that nursing students’ performance wasn’t affected by the type of guidance but instead benefited equally from both the vicarious-learning guidance and the guided-exploration support. However, students from the non-nursing programs, namely nutrition and health-education programs, made higher learning gains within the more difficult items of the Diabetic knowledge questionnaire following learning with the vicarious approach. Hence, non-nursing students benefited more from the vicarious approach than from the guided-exploration one. Although our results show no significant difference in prior diabetes knowledge between the nursing and non-nursing students, a plausible explanation for the difference is related to nursing students’ prior experiences with the pharmacology topics from their clinical practices and curriculum. Whereas students from nutrition and health-education programs are focused on food science and on health promotion and less on pharmacology concerns, nursing training notably emphasizes nurses’ need for accountable and responsible medication management, which is crucial for nursing practice (Khan & Hood, 2018). We propose that this acknowledgment of pharmacology importance as part of a professional identity better prepared nursing students for learning and for making meaning from any support that was available to them, whether vicarious or guided exploration.

As stated in the introduction, the ICAP framework predicts that the more active students are in their learning activities, the better their learning outcomes will be. According to this framework, both conditions of the current study evoked active engagement using both constructive activities (e.g., self-explanations, predictions, and model exploration) and interactive mode involving joint dialogue (Chi & Wylie, 2014). The main practical implication of this study is that learning with tutorials can be scaled up and maximized when visual displays encourage dialogue between instructor and students and when learners solve problems while observing and overhearing this dialogue collaboratively. As we showed, by evoking an active level of engagement using dialoguing, vicarious instructional design to support learning with agent-based models can be at least as efficient as the guided exploration of agent-based models. Vicarious learning provides a unique modeling opportunity for students to learn how to explore agent-based models and enables them to reflect on their own process of learning. Our results show that less-experienced students can especially benefit from the vicarious approach to support learning with agent-based models until they achieve a certain level of expertise.

This study has several limitations, specifically, the current study evaluated only immediate learning effects followed by one short intervention, using a single population. Therefore, the advantage of the vicarious approach over the guided exploration of modeling systems should be further explored. Open exploration of agent-based models is important for concept construction; our findings underline what makes working with agent-based models more or less challenging with different instruction types for different populations.

## References

* Bandura, A. (1969). *Principles of behavior modification*. New York, NY: Holt, Rinehart & Winston.

* Bandura, A., Ross, D., & Ross, S. (1963) Vicarious reinforcement and imitative learning. *Journal of Abnormal and Social Psychology*. 67(6), 601–607.

* Bruner, J. S. (1986). *Actual minds, possible worlds*. Cambridge, MA: Harvard University Press.

* Chi, M. T., Adams, J., Bogusch, E. B., Bruchok, C., Kang, S., Lancaster, M., ... & Wylie, R. (2018). Translating the ICAP theory of cognitive engagement into practice. *Cognitive Science*, doi:10.1111/cogs.12626.

* Chi, M. T., Roy, M., & Hausmann, R. G. (2008). Observing tutorial dialogues collaboratively: Insights about human tutoring effectiveness from vicarious learning. *Cognitive science*, 32(2), 301–341.

* Chi, M. T., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. *Educational Psychologist*, 49(4), 219–243.

* Chi, M. T., Roy, M., & Hausmann, R. G. (2008). Observing tutorial dialogues collaboratively: Insights about human tutoring effectiveness from vicarious learning. *Cognitive Science*, 32(2), 301–341.
D'Angelo, C., Rutstein, D., Harris, C., Bernard, R., Borokhovski, E., & Haertel, G. (2014). *Simulations for STEM learning: Systematic review and meta-analysis*. Menlo Park: SRI International.

Dubovi, I., Dagan, E., Mazbar, O. S., Nassar, L., & Levy, S. T. (2018). Nursing students learning the pharmacology of diabetes mellitus with complexity-based computerized models: A quasi-experimental study. *Nurse education today, 61*, 175-181.

Hmelo-Silver, C. E., Duncan, R. G., & Chinn, C. A. (2007). Scaffolding and achievement in problem-based and inquiry learning: a response to Kirschner, Sweller, and Clark. *Educational Psychologist, 42*(2), 99–107.

Khan, E. U., & Hood, P. A. (2018). Nurses’ perspectives on pharmacology: why, what and at which point of the curricula should education be delivered? *British Journal of Nursing, 27*(10), 546–553.

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. *Educational Psychologist, 41*(2), 75–86.

Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. *Educational Psychology Review, 19*(3), 239–264.

Koedinger, K. R., & Roll, I. (2012). Learning to think: Cognitive mechanisms of knowledge transfer. In K. J. Holyoak & R. G. Morrison (Eds.), *The Oxford handbook of thinking and reasoning* (pp. 789–806).New York, NY: Oxford University Press.

Jacobson, M. J., Markauskaite, L., Portolese, A., Kapur, M., Lai, P. K., Roberts, G., (2017). Designs for learning about climate change as a complex system. *Learning and Instruction*. 52, 1–14.

Lazonder, A. W., & Harmsen, R. (2016). Meta-analysis of inquiry-based learning: Effects of guidance. *Review of Educational Research, 86*(3), 681–718.

Levy, S. T., Peleg, R., Ofeck, E., Tabor, N., Dubovi, I., Bluestein, S., & Ben-Zur, H. (2018). Designing for discovery learning of complexity principles of congestion by driving together in the TrafficJams simulation. *Instructional Science, 46*(1), 105–132.

Minstrell, J. (1982). Facets of students’ knowledge and relevant instruction. In R. Duit, F. M. Goldberg, & H. Niedderer (Eds.), *Proceedings of Research in physics learning: Theoretical issues and empirical studies* (pp. 110–128). Keil, Germany: The Institute for Science Education (IPN).

Muller, D. A., Sharma, M. D., & Reimann, P. (2008). Raising cognitive load with linear multimedia to promote conceptual change. *Science Education, 92*(2), 278–296.

Rummel, N., & Spada, H. (2005). Learning to collaborate: An instructional approach to promoting collaborative problem solving in computer-mediated settings. *Journal of the Learning Sciences, 14*, 201–241.

Samon, S., & Levy, S. T. (2017). Micro–macro compatibility: When does a complex systems approach strongly benefit science learning? *Science Education, 101*(6), 985–1014.

Sherin, B., Krakowski, M., & Lee, V. R. (2012). Some assembly required: How scientific explanations are constructed in clinical interviews. *Journal of Research in Science Teaching, 49*(2), 166–198. doi:10.1002/tea.20455

Schwarz, B. B., Neuman, Y., & Biezuner, S. (2000). Two wrongs may make a right . . . if they argue together! *Cognition and Instruction, 18*(4), 461–494.

Schober, M. F., & Clark, H. H. (1989). Understanding by addressees and observers. *Cognitive Psychology, 21*, 211–232.

Stegmann, K., Pilz, F., Siebeck, M., & Fischer, F. (2012). Vicarious learning during simulations: Is it more effective than hands-on training? *Medical Education, 46*(10), 1001–1008.

Vitale, J. M., McBride, E., & Linn, M. C. (2016). Distinguishing complex ideas about climate change: Knowledge integration vs. specific guidance. *International Journal of Science Education, 38*(9), 1548–1569.

Wilensky, U. (1999). NetLogo. Center for Connected Learning and Computer-Based Modeling. Northwestern University, Evanston, IL.

Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theories—an embodied modeling approach. *Cognition and Instruction, 24*(2), 171–209.

Wilensky, U., & Resnick, M. (1999). Thinking in levels: A dynamic systems approach to making sense of the world. *Journal of Science Education and Technology, 8*(1), 3–19.

## Acknowledgments

This work was supported by AIS - Teaching with Technology Innovation Grant, Utah State University. We are especially grateful to Dr. Sarah Schwartz for her assistance with the statistical analysis.
# Using Voice Assistant Skills in Family Life

Diana Beirl, UCL, UK, dianabeirl01@gmail.com
Yvonne Rogers, UCL, UK, y.rogers@ucl.ac.uk
Nicola Yuill, University of Sussex, nicolay@sussex.ac.uk

**Abstract:** Voice assistants are increasingly prevalent in family life, being used, for example, for listening to music, finding out information, asking jokes and playing games together. However, little research shows how such technology influences dynamic family interactions in the home over time. An in-the-wild study was conducted in six family homes over three weeks. An Alexa, with a number of skills, was set up in each home for the families to use. The findings showed differences in use over time. To begin, family cohesion behavior and family rituals were most prevalent. At the end of the study, the skills were found to motivate distinct family interaction patterns: including more collaboration to manage Alexa and scaffolding of children's interactions with Alexa, given developmental differences in users' grasp of Alexa's capabilities and limitations. We discuss how voice assistants support different interaction patterns and potentially, offer different learning opportunities.

## Introduction

A new technology in many family homes is a voice assistant, such as Amazon's Alexa or Google Home. These devices are often placed in the kitchen or living room, enabling all to interact with it; providing an additional source of entertainment for the family, for example, telling jokes, playing music, keeping track of time and playing games. Recently, Amazon's Alexa has expanded its repertoire by enabling users to download a diversity of 'skills'. These are voice-driven capabilities that are intended to provide a more personalized experience. For example, "Open the Magic Door" is an adventure skill that allows users to choose their path in a story by selecting different options through the narrative. Another one, "Kids court", allows families to settle arguments in an Alexa-run court while learning about the law.

Many of the skills are designed to support multiple users taking part at the same time, offering the potential for families to play together. This kind of social interaction is further encouraged by the affordance of the physical device – that differs from other virtual voice assistants, found on phones and laptops, insofar as their physical presence affords joint ownership and use – similar to other domestic devices, such as the radio, TV and toaster. Little is known, however, about how this combination of device and interactive user experiences are used by families in their own homes.

Playing games and music together can contribute to family bonding and building social and emotional connections (Boer et al., 2014). However, the pervasive uptake of technology in the home is often seen as disrupting such shared engagement, arguably resulting in less face-to-face conversation (Turkle, 2015). But might the introduction of virtual assistants, as shared devices, provide both new opportunities and challenges for the ways families interact in the home? In particular, could the new types of skills, that run on the voice assistants, support these kinds of family interactions, by enabling new repertoires and learning to emerge? If so, how do they manifest themselves? Conversely, could interacting with them lead some family members to dominate while others get frustrated? Could they even promote sibling rivalry and family arguments?

We began to address these questions by conducting an in-the-wild study observing how the voice assistant, Alexa, was used and accommodated in family homes over a period of three weeks. Three different types of skills were downloaded onto it, for the categories of music, storytelling and games. These skills differ in the role that Alexa plays. In providing music, Alexa acts purely to provide content for the user to enjoy either for themselves or as a shared experience. The user usually drives what is played, either independently or in consultation with other people present. In storytelling, Alexa has a greater role in providing potentially immersive content. An audio story is presented with options that lead to different outcomes. hence providing more interaction between Alexa and the users. Games, such as Kids Hub, support a diversity of interaction styles, that have more potential for multiple users interacting with Alexa, as well as parent-child pairs.

The aim of our study was to explore how families learn the new skills and appropriate them into their life. Video observations and interviews were conducted to collect data for how and how often families used the skills. The findings from the study were analysed in terms of different kinds of family interactions exhibited across families. A number of different types of interactions were identified that facilitated family cohesion, bonding and empathy. Some entailed helping children to learn how to play, ask questions and take turns. Others encouraged rituals, routines and rivalry often exhibited in family life. We discuss how families adapt and exploit
the constraints imposed by voice assistants, like Alexa, to play, talk and learn and conversely, how different
Alexa skills shape family interactions.

Background

The home provides an informal learning environment where it is commonplace for parents to buy technological
devices, video games and electronic toys for their children to learn with. Family practices, such as how
technology is used for play in the home, have an influence in developing children’s competencies with
technology (Plowman et al., 2010). Children’s experiences with technology are influenced by specific family
contexts and family culture (Weisner, 2002). But how effective is home technology in supporting learning?
Plowman et al. (2012) conducted a two-year long observation of families, with pre-school children, on use of
technology in the home. They identified four main areas of learning which can be supported by technology: (i)
acquiring operational skills such as learning how to click buttons and learning about cause and effect situations,
(ii) expanding general knowledge such as early literacy and numeracy, (iii) increasing self-efficacy for
curiosity-driven learning and (iv) learning about technology in everyday life.

Based on an ethnographic study of family routines, Davidoff et al., (2005) forewarned how smart home
devices may negatively impact the balance of control in the management of family life, such as leisure planning.
However, studies of the introduction of novel technology into family homes has been found to impact family
behavior in a variety of ways. For example, Ganesh et al. (2014) found that the deployment of augmented
reality technology in a family home setting was able to positively distract pre-school age fussy eaters from their
dislike of green vegetables but also negatively distracted their siblings. The technology worked by removing the
fussy eater's focal attention from the peas on their plate and instead switching their attention to observing and
playing with animated lights projected onto it. Instead of simply refusing to eat the peas – as they normally did –
the augmentation encouraged the young children to view the plate of food in a new way, and in doing so, shift
their focus from eating as a primary activity. However, when parents and siblings also took part in encouraging
the fussy child to eat their peas, the new technology sometimes triggered a negative response, causing sibling
jealousy and competition. Hence, the context had a positive effect on the fussy eater but also impacted on the
social dynamics of the family.

It has also been found that technology designed for the home can engage all family members. Plowman
et al. (2010), for example, observed how parents and older siblings contribute to young children’s engagement
with home technology through guiding and scaffolding their interaction through spoken language, gestures and
expressions. Conversely, school-age children can take more leading roles: a study of parent-child shared reading
on tablets versus paper books reported that school-age children sometimes dominated when reading from
screens compared to paper (Yuill & Martin, 2016). Furthermore, device type affected the nature of family
interaction: using a screen altered the pair’s physical positioning and the interaction quality was less warm than
for paper books. Parents may also introduce a narrative through the technology that may be of particular
motivational interest to their child. For example, Bhömer et al. (2010) designed a photo sharing device intended
as a table centerpiece where personal photos downloaded from Facebook could be shared with the family at
mealtimes. They found how different family members used the device as an opportunistic display of affection to
each other, saying how much they loved particular photographs that popped up. They also found it was used to
highlight and reinforce family ties, through the types of conversation that were triggered.

Short et al. (2017) found greater intergenerational equity of participation with a robot assistant for activities with distinct participant roles. Reeves and Nass (2006) have conducted much research investigating how people approach and treat computers. They will often confide in them more and anthropomorphize them, treating them as if they had human-like qualities. Other research has shown how people respond to technologies as though they were human even if they know they are not (Fong et al, 2003). Additionally, people have been shown to attribute personalities to technology and to apply similar politeness norms from human conversation to such interactions (Nass et al., 1999).

Virtual Assistants add a new dimension to how home technology can play out in family life. Not only
can they take over the control of certain everyday activities, such as reminding users about events, they can also
act as ambient conversational 'partners'. The nature and equity of family interactions is also likely to be
influenced by the structure of different activities provided by virtual assistants. Young children have judged
various virtual assistants to be friendly and trustworthy, and to vary in intelligence (Doucleff, 2017). Druga et
al., (2017) found that younger children (3-4 years old) experienced difficulty interacting with conversational
and chat agents, resulting in them becoming restless. Sometimes, they would try to reword their questions or
speak slower. While people so far have used virtual assistants primarily to seek information, they have also been
shown to contribute to humor in social situations and interruptions in human conversation (Porcheron et al.,
2018). Multi-member households and particularly those with children are more likely to personify voice
assistant devices. However, a study investigating virtual assistants in multi-party conversation demonstrated a number of problematic features which affect the flow of normal social interaction (Purington et al., 2017). This included the need to repeat and refine queries which were not understood by the virtual assistants, and enforced silence by conversation participants so that the virtual assistants could better understand their queries. However, these devices also enabled collaboration, in that any member could interact with the device at any given moment. Family members also tended to reason and reflect about the queries that were made. Porcheron et al. (2018) also examined how voice assistants were used by families in their own homes. They conducted a conversational analysis of some audio recordings that revealed subtle cues and mechanisms used during conversations. They report how a family's interaction with the Amazon Echo is seamlessly interwoven with other ongoing activities, for example, at family mealtimes where parents are at the same time trying to get their child to eat their food. They point out how our conversations with each other and voice-assisted technologies interleave in nuanced ways, rather than being separate conversations between the family or the family and the device, that jump from one to another.

A recent panel discussion on voice assistants (Kaye et al., 2018) identified as important research foci the ways that families interact with voice assistants, and how users understand their capabilities and limitations. Some answers are provided by Sciuto et al. (2018) who analyzed Alexa voice logs for 75 users over a period of about a year, and also interviewed a number of Alexa-owning families with young children under 4. However, to understand the roles that Alexa plays in family interaction, we also need to observe audiovisual dynamic interactions in families with varying compositions, and children across a wider age range, preferably over a period of time and in relation to a range of different voice assistant skills that might engage users differently. The aim of our research was to explore how families approach and interact with different skills downloaded onto Alexa in their own home. Our focus was on how parents and children engage with different skills when they are together, across a range of ages and family types, including only children and those with siblings.

## Method

An in-the-wild study was conducted in six family's homes. The families were given an Alexa to use at home over a period of 3 weeks. They were visited at the beginning and end of the study period when all or most of the family could be present. Ethical approval was obtained from the University Ethics Committee and informed written consent for self and children was gained from parents, as well as assent from the children. They were told that their conversations with Alexa would be recorded for subsequent analysis. All families were given the opportunity to review the Alexa recordings before analysis in case they wanted to remove any items: none chose to remove items.

## Participants

Participants were recruited through flyers, social media, word of mouth and snowballing in south-east UK. The details of the families who participated in the study are shown in Table 1. All had children in the age group of 2-13 years. A broad age range of children was selected to increase the potential for a wider set of interactions to be revealed. Five of the families had used a voice assistant at home and the other was familiar with them.

## Skills

Three skills were pre-loaded onto the voice assistants. These were (i) Music, (ii) Pac-Man stories and (iii) Kids Hub. The families with previous experience were familiar with playing music but did not have experience with the other two skills. The skills were all highly-rated on Amazon and appropriate for the wide age range of participating children. They were chosen for the types of interaction they offered and the degree to which multiple family members would likely participate in using them. (i) Music is essentially providing content, as easily for an individual as for a group. Individual use is supported because music interest likely varies across our wide age range. However, music could also potentially encourage playful interaction in the form of dancing and singing, given the shared audio experience for anyone in the room at the time. (ii) Pac-Man stories is a spoken interactive adventure style game. It similarly provides content accessible for an individual, but has more potential for interaction, because Alexa invites the player to choose alternate paths. The game incorporates sound effects and music, thus supporting the user to become immersed in the story. As with music, this may also have the potential to encourage participation in the form of dancing or engagement from family members not primarily using the skill. (iii) Kids Hub supports a diverse range of more interactive participation, allowing users to choose from a variety of activities including songs with actions, Tongue Twisters, Pictionary and Kiddy Olympics. These activities can be readily used by more than one person and are appropriate for families with single and multiple children.
<table><caption>Table 1: Details of families who participated in the study</caption><thead><tr><th></th><th>Children</th><th>Age</th><th>Parents</th><th>Previous voice assistant</th></tr></thead><tbody><tr><td>Family 1</td><td>Boy</td><td>6</td><td>Mum, dad</td><td>Yes - Google Home</td></tr><tr><td>Family 2</td><td>Boy</td><td>11</td><td>Mum, dad</td><td>Yes - Amazon Alexa</td></tr><tr><td>Family 3</td><td>Boy</td><td>13</td><td>Mum</td><td>Yes - Amazon Alexa</td></tr><tr><td>Family 4</td><td>Boy, girl, girl</td><td>6, 9, 11</td><td>Mum, dad</td><td>Yes - Google Home</td></tr><tr><td>Family 5</td><td>Boy, boy</td><td>7, 10</td><td>Mum, mum</td><td>No</td></tr><tr><td>Family 6</td><td>Girl</td><td>2</td><td>Mum</td><td>Yes - Amazon Alexa</td></tr></tbody></table>

## Procedure

The researcher visited each family home for about an hour and set up the Amazon device in a place chosen by the family - the living room or the kitchen. She explained how to use the three skills. The families were free to explore them. A camera was set up in the corner of the room to record the interactions during her visit. Afterwards, the family members were asked about their reflections of Alexa and how they imagined it would be used in the home. The families were asked to explore the skills for a week. They were then sent a link to the online Alexa Skills catalogue and given the option of downloading additional skills, for the following two weeks. The researcher visited the families again at the end of the study to ask them to reflect on their use of the Alexa skills.

## Data collection and analysis

Video recordings were taken during the first and second visits and conversations and interactions with Alexa transcribed. In addition, the voice recordings from when anyone interacted with Alexa were collected and transcribed. Both were coded separately for each visit by one rater using NVivo software, and all codings were reviewed by a second rater and placed into themes. These themes were extracted and validated as modelled by Saldana (2016), by discussing them with colleagues and checking interpretations with the participants themselves during the second visit.

## Findings

All the families tried out the three skills and several families downloaded other skills. There was much enthusiasm about how they had interacted with Alexa and how it became part of their family rituals. Usage of the skills persisted across the three weeks - as evidenced by the conversations recorded by Alexa. Some families reported using the device during meal times while others used it during shared activities such as cooking or after-school play. The younger children (aged 6) in the families were curious about Alexa, asking it a range of questions, as if it was a friend, for example, "Alexa, what am I doing right now?", "Hey Alexa, what's your second name and your third name?", "Alexa, who is your favorite YouTuber?", "Alexa, is Google and Siri your friends?" One family (family 1) even resorted to asking Alexa during a meal time to resolve a debate: "Once we used it when we were all eating, like we were having our meal and asking it questions...That's what it was, we were discussing something, and we didn't know who was right, so we asked Alexa..."

To examine further the different ways Alexa became integrated into family life we identified and then categorized excerpts from the transcripts in terms of three main themes: family cohesion, family rituals and scaffolding. These were further classified in terms of the specific mechanisms used when interacting with Alexa. Certain themes recurred both within a single observation (i.e. behaviors were observed repetitively in the same family or a theme was mentioned a number of times within the same family) in addition to being a common theme across families. The quantity of incidences together with specific examples and comments made by the families are provided below. The most frequent interactions observed were in the scaffolding category, where different family members helped each other but especially younger children interact with Alexa.

### Family cohesion

Many forms of family cohesion was observed across all families. These were classified in terms of shared laughter, teasing, exchanging gestures and facial expressions and family rivalry.

*Shared laughter.* The interactions with Alexa often resulted in much shared laughter. 62 instances were identified during the three weeks for all families. They typically happened when family members were playing a game and when a mishap occurred, that Alexa was not privy to. For example, when playing the Kids Hub skill, family 3 chose the activity "Pictionary", a charades-style guessing game. In this game, Player 1 has ten seconds
to leave the room and following this Player 2 is given a word which they have to draw. Player 1 then comes back in the room, looks at what the other person has drawn and tries to guess the word. In one game, the son overheard Alexa give his mother the word "dog" to draw. They continued playing the game and the mother mentioned the word by mistake to which the son retorts:

Mother: "This doesn't even look like a dog"

Son: "You just told me!"

At which point they both laugh and the mother continues, "Well you heard it anyway. I'm not good <at drawing> this looks like a cat, maybe even a pig"

Shared laughter was also observed in situations where Alexa didn't work as intended, for example when it was unintentionally activated by a TV ad (family 4). Alexa's misunderstandings of what someone said were also a source of amusement. One child recalled a situation where Alexa misunderstood his mother's request: "Oh, when you were asking how long to cook sausages for. And then it tried to order sausages!" (family 3).

*Teasing*. There were also several instances of teasing (34) recorded, although considerably less than the shared laughter, across five of the families. Typically, family members interacted with Alexa to make fun of or provoke other family members in a playful way. For example, the child in family 2 was playing a music quiz online and losing to an opponent (Tom) in Texas. He kept looking at his dad to see if he knew the answers:

<Alexa plays next song>

<Child laughs and grimaces to his dad to indicate that he doesn't know the song>

Child: "Skip"

Alexa: "The song is Timber by Pitbull, Tom answered correctly"

Father: "Tom needs to get out more."

<Next song comes on and child looks at his dad and laughs>

Father: "don't ask me!"

The parents in family 1 often asked Alexa to play the good morning and the good evening song. When asked about this, the mother laughed and said that she would put the song on to annoy her son in the morning while he was getting ready for school or when it was time to get ready for bed.

*Exchanging gestures and facial expressions*. Families often reacted to interactions with Alexa by exchanging facial expressions, mimicking Alexa or making faces in response to things that it said. There were 41 instances of this coded for families 1,3,4 and 6. An example of exchanging facial expressions was when the mother and son (family 1) were listening to "fun facts":

Alexa: "In the 18th Century it was believed that kissing a donkey could relieve toothache."

<Son and mother make a disgusted face at each other>

An example of mimicking Alexa was when one family switched Alexa from having a British accent to an Australian one. The father then started speaking in an Australian accent which the children found funny. They then also tried to talk in an Australian accent. This led to much banter.

*Family rivalry*. A normal aspect of family life is sibling rivalry - when children think they are getting unequal amounts of attention or responsiveness from their parents, or intergenerational conflict. 45 instances of this were coded for families 2, 3, 4 and 5. Family 2 said they sometimes struggled to take turns to choose music because they did not like each other's choices. In contrast to other personal technologies, where the person holding a device or remote control has sole control, everyone has equal access to control of Alexa. This can mean anyone can override or interrupt what the other has asked Alexa to do as one father commented: "It was like one of us would put music on and then the other one would change it when they couldn't put up with it anymore, so we weren't taking turns." One mother also explained how when her children try to use Alexa at the same time it could lead to arguments. For example, the siblings in Family 4 often competed for Alexa's attention with different requests and then became upset when the mother gave her support to the other one:
Mother: "Alexa play barking"

Youngest child: "I don't want that!"

Mother: "Well you're gonna get it."

Youngest child: *But he got what he wanted!*

Another time a mother (family 5) tried to reason with her elder child about needing to be fair with his younger brother:

Mother: "Did you take it in turns?" <when asking Alexa for music>

Elder child: "Yeah, but then he got to choose."

Mother: "Well probably because you'd been up for an hour already and had loads of choices before he came and had breakfast."

## Family rituals

Family rituals are an integral part of family life. These were much in evidence when interacting with Alexa, with 58 instances coded across all families. Most were triggered during the playing of a song. For example, when a child requested Alexa play the 'Baby Shark' song every day, the mother groaned each time, but then the whole family would start dancing and singing along. Another example, is where the mother of family 1 requested a song on her son's behalf to which the father looked at the son and encouraged him to show off his dance moves:

Mother: "Alexa play the gummy bear song."

<Son looks at his mum>

Father: "Show her your dance moves"

Another child (family 4) commented how their parents now refer them to ask Alexa if they want to find something out rather than looking it up themselves on the internet: "If mum or dad are doing something and I ask them a question instead of going like have you searched it up they would say go and ask Alexa." This shift requires them formulating and talking aloud their query which the parents can be privy to if in the same room and join in or be aware of what their children are trying to find out.

## Scaffolding

A common occurrence was when family members collaborated by encouraging each other when using Alexa. 111 instances of these were observed across all families – making this the most frequently observed category. This happened either when the family worked together as a team or when a more competent family member helped a younger member interact with Alexa.

When playing a story skill with Alexa, such as PacMan, family members sometimes whispered to each other what they should do before asking Alexa – so as not to let it hear or know about their discussion before making their choice. Thumbs up gestures and nodding were also employed during this kind of behind the scenes negotiation process. Sometimes one person would take the role of leader while the rest of the family members would whisper to each other what to answer. For example, family 4 discussed which option to choose before the youngest child eventually shouted out their answer:

Alexa: "You could pick up the fork or you could read the sign."

<All whisper to each other to decide which one.>

Mother: "What do you want to do?"

Youngest child shouts: "Pick up the fork!"

When Alexa did not recognize what a younger member of the family said, other family members often repeated it slowly to encourage the child to try again but to speak more clearly. For example, the youngest child in family 4 wanted to play a funny song but was not able to say the word of the song correctly. His older brother helped him by repeating it and gesturing at Alexa to indicate where to speak. One mother (Family 5) commented on how it was much harder for the younger child to be understood by Alexa. When they struggled with some of the questions she would try to rephrase it to be simpler, so the child could learn how to be more concise. Similarly,
in family 2, the father pointed out how Alexa does not understand his 11-year-old son as well as himself.
Another mother (Family 1) often encouraged her child to interact with Alexa by pointing at it:

Mother to child: *Do you want to hear another?*

<Mother points at the Alexa to encourage child to ask>

Child to mother: “yes”

Another strategy families used to encourage their children to speak to Alexa was to tell them how to ask Alexa questions. Families also encouraged their children to practice speaking to it. Hence, there were many examples of parents helping their children to learn how to speak with Alexa to make themselves understood.

Discussion

The findings demonstrated how families engaged in a diversity of interactions when using the Alexa skills. They developed new family rituals and encouraged each other to talk to and take part in games with Alexa. Many of these interactions could be seen as contributing to social and emotional bonding, leading to further family cohesion. For example, the use of non-verbal behavior while interacting with Alexa provided a means of expressing empathy. They all took to having a new voice in the family home – like having a new family pet. However, the interactions that occurred were not always harmonious and sometimes led to family disputes. In a way, this reflects the richness of family life, where children learn to compete with their siblings but also learn to empathize with them; where families develop new rituals by playing games or performing together and in doing so bond over them but at other times use the opportunity to tease each other.

Lots of shared laughter occurred because of Alexa's mishaps especially when it misunderstood them. For younger children, this could be quite frustrating and at times challenging. Parents and elder children were acutely aware of what they were going through and would often try to help them out, by showing them how to interact with Alexa. Other times, activities with Alexa led to joking, joshing and teasing, building upon pre-existent family discourses about particular interests or sensitivities. Alexa was also incorporated into existing family dynamics and could sometimes encourage or amplify those, for example, when commenting on different tastes in music or different realms of expertise across family members. Different interests across family members could also lead to disagreement, particularly when children wanted to play certain songs or games repeatedly. This happened both between siblings and between parents and children.

Alexa in its current form was unable to recognize different family members by their voice. This meant it could not understand different people talking simultaneously – which families often do in family conversations, and during which they are able to hone in on what one person is saying (cf. the cocktail party effect). To adapt to Alexa not being able to do this, families adapted their behavior. For example, they would all sit together, often on the same sofa, facing Alexa and communicate with it, as a family unit, rather than carry on having a conversation with each other. Alexa became the central focus for their interaction. This differs from Porcheron et al.'s (2018) finding where the conversation with Alexa was interwoven with the parents taking turns to talk to cajole their child to eat their dinner.

Families frequently used back-channel methods of communicating, such as whispers, to avoid getting unintended responses from Alexa. Sometimes this was done in the spirit of pretend play, where families whispered together so that the children would learn how to make joint decisions together when playing a game that others would not be able to hear – in this case Alexa. Once agreed on their answer, an elected person (self or other) in the family would call it out to Alexa. This pattern of interaction was observed when families played games together. However, problems sometimes occurred when there were younger children in the family who found it difficult to follow the rules of playing a game or a quiz, or when siblings felt unequally treated. When such times arose, families changed tack to deal with them, for example, the parents and elder children adopted helper roles to encourage and make suggestions for what the younger child should say to Alexa.

Alexa was also found to encourage families to jointly act out together, namely singing and dancing. Having an external participant made this appear like a performance, as evidenced by parents sometimes referring to Alexa as a potential audience, for example, suggesting that their children show Alexa their dance moves. Listening to music together is something families may already do, for example on journeys in the car. Teenagers may listen to music together and act out the singer's dance routines. With Alexa, it became possible for families to feel comfortable with all the members dancing and singing repeatedly. Even the mother groaning appeared to enjoy being part of the ritual. And, as Druga et al., (2017) point out, family rituals have a positive impact on family cohesion and emotional well-being. In terms of learning, children were inducted into a whole range of skills in how to tailor their communication to a new audience member, how to hold the floor, and how
to use back channels of communication when there are interactants with different needs. Having a new
interactant in the house also gave new possibilities for performance, and engagement with music in particular.
Further work will reveal how the role of voice assistants in the family may evolve over the longer term.

Conclusion

Voice assistants, like Alexa, are making big in-roads into family homes. Our study showed how families readily appropriate them into their family life, playing, singing, performing and even asking Alexa questions to resolve family disputes. Alexa was found to encourage much shared laughter and helping each other but also trigger sibling rivalry, teasing and family arguments. However, both negative and positive aspects of family interactions are an integral part of learning about each other and developing empathy skills. Contrary to Turkle's (2015) concerns about the digital age, interactions with voices assistants such as Alexa, may even help children learn the art and joy of conversation in the presence of humans and machines.

References

Bhömer, M.T., Helmes, J., O'Hara, K. & van den Hoven. E. (2010). 4Photos: a collaborative photo sharing experience. In *Proc. of NordiCHI '10*. ACM Press, 52-61.

Boer, D., & Abubakar, A. (2014). Music listening in families and peer groups: Benefits for young people's social cohesion and emotional well-being across four cultures. *Frontiers in Psychology*. 5.

Davidoff, S., Lee, M.K., Yiu, C., Zimmerman, J. & Dey, A. (2006). Principles of smart home control. In *International conference on ubiquitous computing (UBICOMP)*. Springer, Berlin, Heidelberg. 19-34

Doucleff, M. and Aubrey, A. (2017). Alexa, Are You Safe for My Children? https://www.npr.org/sections/health-shots/2017/10/30/559863326/alexa-are-you-safe-for-my-children.

Druga, S., Williams, R., Breazeal, C. & Resnick, M. (2017). "Hey Google is it OK if I eat you?" In *Proc. of the 2017 Conference on Interaction Design and Children (IDC '17)*. ACM Press. 595-600.

Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003). A survey of socially interactive robots. *Robotics and Autonomous Systems*. 42(3-4), 143-166.

Ganesh, S., Marshall, P., Rogers, Y. & O'Hara, K. (2014). FoodWorks: tackling fussy eating by digitally augmenting children's meals. In Proc. of NordiCHI'14. ACM Press, 147-156.

Kaye, J., Fischer, J., Hong, J., Bentley, F.R., Munteanu, C., Hiniker, A., Tsai, J.Y. & Ammari, T. (2018). Panel: Voice Assistants, UX Design and Research. In *Proc. Conference on Human Factors in Computing Systems (CHI EA '18)*. ACM, New York, 01, 5 pages.

Nass, C., Moon, Y., & Carney, P. (1999). Are people polite to computers? Responses to computer-based interviewing systems. *J Appl Soc Psychol* 29, 5: 1093-1109.

Plowman, L., Stephen, C. & McPake, J. (2010). *Growing up with technology: Young children learning in a digital world*. New York: Routledge.

Plowman, L., Stevenson, O., Stephen, C. & McPake, J. (2012). Preschool children's learning with technology at home. *Computers & Education*, 59 (1), 30-37.

Porcheron, M., Fischer, J.E., Reeves, S. & Sharples, S. (2018). Voice Interfaces in Everyday Life. *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18)*. 640, 12 pages.

Purington, A., Taft, J.G., Sannon, S., Bazarova, N.N. & Taylor, S.H. (2017). "Alexa is my new BFF": Social Roles, User Satisfaction, and Personification of the Amazon Echo. *Proc. of the CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17)*. ACM Press. 2853-2859.

Reeves, B. & Nass, C. (2006). *The media equation: How people treat computers, television, and new media like real people and places*. Stanford, CA: CSLI.

Saldana, J. (2016). *The coding manual for qualitative researchers*. Sage.

Sciuto, A., Saini, A., Forlizzi, J. & Hong.J. (2018). "Hey Alexa, What's Up?: A Mixed-Methods Studies of In-Home Conversational Agent Usage." In *Proceedings of DIS'2018*, 857-868. ACM, NY.

Short, E.S., Swift-Spong, K., Shim, H., Wisniewski, K.M., Zak, D.K., Wu, S. & Matarić, M.J. (2017). Understanding social interactions with socially assistive robotics in intergenerational family groups. *26th IEEE International Symposium on Robot and Human Interactive Communication*. 236-241.

Turkle, S. (2015). *Reclaiming conversation: the power of talk in a digital age*. Penguin.

Wang, B., Taylor, L. & Sun, Q. (2018). *Families that play together stay together: Investigating family bonding through video games*. New Media & Society.

Weisner, T. (2002). Ecocultural understanding of children's developmental pathways. *Hum. Dev.*, 45, 275-281.

Yuill, N. & Martin, A. (2016). Curling up with a good E-Book: Mother-child shared story reading on screen or paper affects embodied interaction and warmth." *Frontiers in psychology* 7 (2016): 1951.
# Measuring the Computational in Computational Participation: Debugging Interactive Stories in Middle School Computer Science

Chris Proctor, Stanford Graduate School of Education, cproctor@stanford.edu

**Abstract:** An equitable implementation of K-12 computer science must support inclusive literacy practices, but it must also develop concrete skills. This study analyzes the extent to which a computer science curriculum based on digital storytelling helped students become more effective at debugging. Prior research has developed digital storytelling as a medium for computational participation, but few studies have reported detailed results on growth in computer science skills. Meanwhile, research on debugging has tended not to address sociocultural factors. This study, conducted over four months of a middle-school computer science course using interactive storytelling, analyzed student reading, writing, and debugging practices based on approximately 1000 story edits and user behavior collected from the platform's logs. The results suggest that literacy-based computer science education using digital storytelling can be productive for developing skills such as debugging.

## Introduction

Driven by the economic opportunities and pervasive societal impacts of computing, computer science (CS) is rapidly becoming a mainstream subject in primary and secondary education. Computational thinking is developing as a set of ideas and practices to be taught across the curriculum. In either case, the interdisciplinary connections to STEM subjects are clear and compelling (Weintrop, et al., 2014), while much less is said about how CS might support the core concerns of the humanities and social sciences. Indeed, when CS is defined narrowly as a collection of facts and skills about solving problems with computers, the subjects may not have much common concern. However, youth today engage in diverse and complex literacy practices with digital media (Ito, et al., 2010), relying to various extents on the computational aspects of these media to engage in computational participation (Burke, O'Byrne, & Kafai, 2016). If K-12 CS were contextualized within these literacies, its central concepts and practices could become an essential part of diverse youth cultures reading, writing, and analyzing digital texts.

Recognizing this opportunity (particularly for youth who do not see their identities and cultures represented in the world of CS), there have been numerous efforts to incorporate digital storytelling into school. Like projects which embed computation into animation (Resnick, et al., 2009) and e-textiles (Buechley, 2006), digital storytelling allows learners to learn programming and encounter powerful ideas from computer science using media which already mediate their literacy practices. Ware & Warschauer (2005) explored the potential for digital storytelling to transcend the divide between school and informal spaces for youth marginalized by race and social class. Kelleher & Pausch (2007) conducted digital storytelling workshops with middle-school girls using a modified version of Alice, finding that the opportunity for self-expression, sharing, and identity development provided motivation for learning to program. Burke & Kafai (2010) drew on parallels between programming and writing to explore how digital storytelling might support growth in each. Proctor and Blikstein (2019) analyzed how the computational elements of interactive stories could function rhetorically and support the development of critical perspectives.

However, some computer scientists argue that sociocultural definitions of K-12 computer science, such as those above, are unworkably vague and impossible to assess (Denning, 2017). This critique demands a response, particularly as it aligns with broader arguments that specific skills are best taught in a context that minimizes cognitive load (Kirschner, Sweller, & Clark, 2006). Must the literacy-based approaches described above involve a tradeoff in terms of how well students learn foundational skills and concepts? Building on prior work developing digital storytelling as a fruitful medium for computational literacies, this study analyzes whether such an environment can also effectively support specific computational practices such as debugging. This study's research questions are:

1. Does writing interactive stories support development of debugging practices?

2. Does reading other stories support development of debugging practices?

## Background
## Text-based interactive storytelling

Interactive storytelling can be distinguished from the broader category of digital storytelling by the use of programming to implement nonlinear single-player games in which the player becomes a character. Text-based interactive stories are particularly effective settings for using computational elements for rhetorical effect in the service of representing and critically analyzing social realities (Proctor & Blikstein, 2019; Proctor & Garcia, 2019). In contrast to primarily-visual storytelling platforms such as Scratch and Alice, writing is singularly important for narrative, representation, and analysis of subjective phenomena in popular culture and in the humanities and social sciences. One example of the transmedia possibilities of text-based interactive storytelling is a high school sociology class in which students wrote interactive stories exploring the use of power in social interactions. The stories' use of programming allowed them to create replayable models of social situations in which readers could explore the consequences of different interactional choices.

The implementation used in this study is a web application called Unfold Studio in which stories' source code is presented side-by-side with a running version. Unfold Studio has social affordances such as the ability to publish stories, browse and read other authors' stories, and a feed showing events related to an author's stories and other authors she follows. Stories are written in a programming language called Ink (Inkle, 2016) and compiled every time an author saves her work. The story excerpt shown in Figure 1 illustrates the syntactical elements analyzed in this study. Chunks of the story are defined as knots (== hotel ==), which are linked together via diverts (->). Knots typically end by presenting the player with choices (+) of what should happen next. Constructs within curly braces allow variables to influence the content and choices shown to the player. When stories contain errors, explanatory error messages are shown in the space that would have been taken up by the running story.

## Debugging

This study analyzes the association between writing interactive stories and performance on a debugging task. McCauley, et al.'s (2008) review of educational literature on debugging defines debugging as part of a response to some kind of breakdown in a programmer's plan for reaching a goal. After testing reveals that a breakdown has occurred, debugging involves "find[ing] out exactly where the error is and how to fix it."(p. 68) The K-12 Computer Science Framework (2016) uses a similar definition and adopts testing and debugging as one of seven core practices in computer science (p. 81). Although debugging has historically not been emphasized in computing education, it occupies a substantial portion of professional programmers' time (Beller, et al., 2017). Bugs may occur at the level of syntax (the program cannot be compiled), semantics (the program crashes at runtime), or logic (the program works, but not as expected). The available data limits this study to an analysis of bugs in syntax.

Debugging is a particularly interesting skill to study in an interactive storytelling context. On the one hand, prior research has found debugging to be a distributed social practice, which suggests a literacy-oriented approach could be effective in developing students' debugging skills. Flood, et al. (2018) found that learning to debug resembles enskilment in which "a newcomer is supported in appreciating and using the affordances of their environment" (p.1). Piorkowski, et al. (2013), found that during debugging, professional programmers spend half their time in information-foraging behaviors such as reading other programs, reading documentation, and searching online. Multiple studies have found comprehension of the program being debugged (and understanding of its goal) to be an important factor in debugging

Welcome to California
+ I am a 14 year old girl.
-> student
+ I am a 50 year old man.
-> teacher

=== teacher ===
As an adult you have special
privileges.
+ Fly to San Fransisco
-> enter

=== student ===
You can show your parent you
can do something on your own!
+ Fly to San Fransisco for a
short independace trip.
-> enter

=== enter ===
You made it to San
Fransisco!!!
+ Go to the hotel
-> hotel

=== hotel ===
{teacher > 0:
The Man at the front
desk says, "Hello Sir, do you
have a reservation for a
room?"
-> room
- else:
The Man at the front
desk says, "Why are you here?
Your dont have a reservation!
Where are your parents? Go
away child!" Oh no! Your
stuck to sleep on the steets!
-> streets
}

<caption>Figure 1. Excerpt from an interactive story written in the Ink programming language.</caption>
success (McCauley, et al., 2008). This invites comparisons to the importance of reading comprehension for fixing grammatical errors in writing (Weaver, 1996).

On the other hand, foundational research on debugging identified misconceptions which could be exacerbated by a literacy-based approach. Bonar and Soloway (1985) found that novices misunderstand programming as writing. In particular, novices tended to inappropriately apply natural language meanings to words such as "while." Pea (1986) saw this as an instance of a more general superbug: novices' misconception that computers reason about programs and interpret them intelligently, instead of following them mechanistically. It is plausible that text-based interactive storytelling, in which authors blend prose with code to implement discourse scenarios, could unproductively mingle the ways humans interpret text and the ways machines interpret code. However, it is also plausible that these misconceptions could actually be ameliorated by casting programming as reading and writing. When there is real authorial meaning motivating the program, and a real audience to whom it is addressed, it could be the case that students better understand the computer's mediating role, rather than misunderstanding it as co-author or interlocutor.

# Methods

## Context

The study took place over four months in a private all-girls middle school in the western United States. The participants, in seventh grade, were enrolled in a computer science class which met twice weekly for a 90-minute block period. 40 students out of a cohort of 67 consented to participate in the study. Students were asked to self-identify with respect to race and gender. 92.5% affirmatively identified as female; 37.5% identified as white, 27.5% as of mixed race, 20% as Asian, 5% as Hispanic, and 10% declined to state. While no data on socioeconomic status were available, the school is located in an affluent area and charges substantial tuition while also offering full or partial scholarships to many of its students.

During the period of the study, the students' computer science class followed the curriculum developed by Proctor & Blikstein (2019). Students were introduced to increasingly complex computational concepts and programming syntax and asked to use them in open-ended story prompts. During class time, students were free to work on their stories, read their peers' stories, and to seek help from peers or the teacher. The study's author worked with the teacher to plan the unit and provided technical support, but was not involved in teaching.

## Data sources

### Stories

There are 327 stories from 39 authors with an average of 92 lines ($\sigma$ = 84) and 431 tokens ($\sigma$ = 400), where tokens include individual words as well as syntactical elements. A snapshot was captured every time a student saved her story. Saving a story was a necessary step in recompiling and re-playing the story. Figure 2 (b) shows that stories generally grew quickly at first, and then entered a period of fluctuating slow growth. 83% of story edits took place within the first 90 minutes (the duration of a block class period) of the story's creation, so most of the activity under investigation in this study took place within the classroom literacy space.

<figure><img src="image_64.png" /><figcaption>Figure 2. Stories written during the study.</figcaption></figure>

This study's primary unit of analysis is the story edit, or differences between each pair of adjacent versions of a story. Comparing line-by-line changes in an edit yields a collection of ops, where each op is a contiguous set of lines inserted, deleted, or replaced from one version to the next. Each op was classified by the syntax elements
involved, such as knots or diverts. Some ops involve changes to code, such as creating new knots, diverts, and variables. Others only involve changes to text, leaving the story's programmatic structure unchanged. Then each edit was classified according to the properties of its ops and whether the prior and latter versions successfully compiled. Edits leaving the story unchanged were filtered out. Story edits inserted an average of 5.3 lines per edit (σ = 11.4) and deleted an average of 1.9 lines per edit (σ = 4.2). Table 1 shows the classification scheme.

<table><caption>Table 1: Classification scheme for states of story edits</caption><thead><tr><td>Pre ok?</td><td>Post ok?</td><td>Ops</td><td>State</td><td>Simple state</td><td>Percent</td></tr></thead><tbody><tr><td>yes</td><td>yes</td><td>&gt;= 5 lines inserted total</td><td>MAJOR INSERT</td><td>OK</td><td>14%</td></tr><tr><td>yes</td><td>yes</td><td>-5 &lt; lines &lt; 5; code changes</td><td>MINOR CODE EDIT</td><td>OK</td><td>27%</td></tr><tr><td>yes</td><td>yes</td><td>-5 &lt; lines &lt; 5; no code changes</td><td>MINOR TEXT EDIT</td><td>OK</td><td>22%</td></tr><tr><td>yes</td><td>yes</td><td>&gt;= 5 lines deleted total</td><td>MAJOR DELETE</td><td>OK</td><td>2%</td></tr><tr><td>no</td><td>yes</td><td>any</td><td>DEBUG SUCCESS</td><td>OK</td><td>10%</td></tr><tr><td>yes</td><td>no</td><td>any</td><td>ERROR</td><td>ERROR</td><td>11%</td></tr><tr><td>no</td><td>no</td><td>any</td><td>DEBUG FAIL</td><td>ERROR</td><td>13%</td></tr></tbody></table>

## Views of other stories

The second research question asks whether reading other stories was helpful in students' debugging. To answer this, backend log data was filtered to collect every instance of a student viewing a story. We then counted the number of a user's views which occurred during the span of each story edit (excluding views of the story being edited). Students viewed an average of 2.4 other stories (σ = 2.6) during editing. This was surprising, as with only a few exceptions, students reported in the post survey that they enjoyed reading each others' stories and frequently cited specific examples of stories they had read. One possible explanation is that this study counts only views that took place during an edit.

## Summative assessment and survey

Two months after the interactive storytelling unit ended, participants were given a summative assessment of their debugging skills and a post-study survey collecting demographic information and measuring their affect and attitudes toward computing (Friend, 2016). Because participants had no prior experience with interactive storytelling, a pre/post test study design was not feasible. We used an adaptation of the Fairy performance assessment (Werner, et al., 2012), which was designed for this situation. Students were given a directed graph (Figure 3(b)) showing the desired functionality of a story (they had previously used directed graphs in planning their stories), annotated with specific issues. They were then given a copy of a broken implementation of the story and asked to fix it.

<figure><img src="image_65.png" /><figcaption>(a) Score distribution (b) Story graph showing desired functionality</figcaption></figure>

Figure 3. In the summative assessment, students were asked to debug a broken story.

These assessments were scored using a rubric. One point was assigned for correctly linking story knots together using diverts (mapping the graph structure to the story structure), and one point was assigned for correcting each issue pointed out by an arrow. Each required the use of a different computational topic students had worked with during the unit: maintaining

state explicitly with variables and implicitly with the players' path through the story, using state to change text output, and using state to control the availability of choices.

## Analysis

### RQ1: Does writing interactive stories support development of debugging practices?
To answer this question, we first needed to describe patterns in students' story editing. We created a matrix counting the number of transitions from each pair of states, across all edits in all stories, and visualized this in a transition diagram. We also created a simplified model grouping together all successful edits and all unsuccessful edits. We then used OLS regression to estimate the association between writing more complex stories and summative score, and the association between probability of successful debugging and summative score. Story complexity was defined as the sum of diverts and choices used across all of an author's stories. This measure was chosen to capture the amount of computational content in stories. The probability of successful debugging was calculated as DEBUG_SUCCESS / (DEBUG_SUCCESS + DEBUG_FAIL) for each author.

## RQ2: Does reading other stories support development of debugging practices?

We theorized that as debugging is a distributed, social, and mediated process, students might be especially likely to view other stories (either written by peers or their own earlier stories) during debugging. As described in the previous section, we collected the number of other story views which occurred during each story edit. We grouped edits into debugging edits (DEBUG_SUCCESS and DEBUG_FAIL) and non-debugging edits (the rest), and conducted a two-tailed, two-sample independent T-test to determine whether a significant difference exists in these two groups' means, with the null hypothesis that there is no difference between group means. We do not assume equal sample size or variance between the groups, so we use Welch's T-test. Additionally, in the post-study survey, we asked students whether they found it helpful to look at other stories.

## Results

### RQ1: Does writing interactive stories support development of debugging practices?

<figure><img src="image_66.png" /><figcaption>**Figure 4.** Transition diagrams showing probabilities of moving between story editing states.</figcaption></figure>

Figure 4 shows the full (a) and simplified (b) transition diagrams between story editing states. (Transition probabilities under 0.15 are omitted for clarity.) Several editing patterns are visible: Authors tend to begin by entering one of two subgraphs: success or error (this was the inspiration for creating the simplified diagram in Figure 4(b)). When edits are successful, authors generally begin with short text editing (for example, writing a story with no code), and then cycle between editing text and code. MINOR_CODE_EDIT is a sink state, suggesting that once stories successfully mature, authors tend to spend their effort tweaking the code rather than adding substantial new textual content.

Of the successful editing states, DEBUG_SUCCESS stood out as most likely to precede an error. This is not surprising, as creating and fixing errors is part of the iterative process of writing challenging programs. When authors fail to successfully debug on encountering an error, they enter the DEBUG_FAIL sink state. The overall impression given by Figure 4(a), and clarified in Figure 4(b), is that of generally productive editing with distinct modes of successful editing and debugging.
Figure 5(a) shows a scatter plot and regression line (with 95% confidence interval) for the association between an author's total output of diverts and choices and score on the summative assessment. After removing the prolific outlier (shown in orange), the association was modeled as score = 1.58 + (diverts+choices) * 0.0035, with $P>|t| = 0.004$ and adjusted $r^2= 0.18$. In contrast to this strong association, there was no association between probability of successful debugging and summative score. This is clearly visible in Figure 5(b).

<figure><img src="image_67.png" /><figcaption>Figure 5. OLS Regression and 95% confidence interval plotted.</figcaption></figure>

## RQ2: Does reading other stories support development of debugging practices?

Figure 6(a) shows the mean and standard deviation number of story views for each editing state. The result of the T-test comparing debugging states and others was $t=-3.66$, $p>|t| = 0.0003$. There was a significant difference between debugging and non-debugging states, but in the opposite direction as was hypothesized. Another unexpected difference was the dramatically higher number of story views during MINOR TEXT EDIT and ERROR states. We wondered whether this was due to these states tending to have longer time durations; Figure 6(b) plots the mean time interval for each editing state. We interpret these results in the next section.

<figure><img src="image_68.png" /><figcaption>Figure 6. Characteristics of different edit states.</figcaption></figure>

## Discussion

This study's results suggest that a literacy-based approach to teaching computer science using interactive storytelling can be an effective context for learning debugging. The results also revealed surprising dynamics between debugging and viewing other stories, which warrant further iterations of design-based research. Participants in this study engaged in substantial debugging (moving in and out of error states) during their story-writing, and authors who used more computational elements in their stories were more likely to score highly on
the summative assessment. However, we did not find an association between successful debugging and higher
performance on the summative task. These findings agree with participants' near-universal consensus that the
literacy-based approach to teaching computer science helped them grow as programmers. Furthermore, when the
post-study survey was administered, students had already spent two months learning Python, and 89% felt that
interactive storytelling with Ink had helped prepare them to learn Python. (In contrast, transitioning from block-
based languages to text-based languages is often difficult for novices (Weintrop & Wilensky, 2016).

One surprising finding was that, rather than using other stories as debugging resources, authors viewed stories less often during debugging. One interpretation of this is might be that debugging is a time to focus on the problem. Intuitively there is some truth to this, however this would also imply that looking at other stories during debugging should be negatively-correlated with debugging success. We did not find this to be the case. Furthermore, the literature on professional debugging (discussed above) finds a substantial reliance on other people and texts. It could be the case that other resources, such as handouts or the Ink language documentation, were more useful than other stories when authors got stuck.

The other unexpected pattern observed in story views across edit states was that authors looked at other stories so frequently while in ERROR and MINOR TEXT EDIT. The prevalence of other story views during ERROR may be partially explained by the fact that ERROR states tended to occupy somewhat more time than other states (see Figure 6(b)). Additionally, it is plausible that authors in ERROR edit states became frustrated and disengaged, and read other stories as a diversion rather than as part of productive debugging. The disproportionately high views of MINOR TEXT EDIT, however, cannot be explained by long duration; 90% of these edits spanned less than three minutes. The post-study survey provides a possible interpretation. All but three participants felt that reading peers' stories was helpful, but when asked to cite specific examples, students almost always described the content of the stories, and not their structural features. This, together with the high number of other views for the MINOR TEXT EDIT state, suggests that students may have been more successful learning from the content of each others' stories than from the computational aspects.

If this interpretation is correct, it poses a design challenge for future development: How might we design the literacy space so that students can learn from computational aspects of other stories as well as their content? One possible approach was tested at the end of the study, and has since been developed further. The students' final assignment was to collaboratively create a story in which the reader explores a world. The class planned out the story on one wall of the classroom, each student wrote a small part of the world, and then they worked together to import partial stories into a whole. Variables (such as energy level or the number of clues collected) maintained state across the different components. This assignment surfaced important computational concepts such as state and interface, and made students dependent on the computational aspects of each others' stories.

The results of this study should not be over-generalized. This study's participants attended an all-girls private school which provides unusual levels of personal attention (possibly allowing them to feel safe participating in a literacy space) and they had already studied computer science for one year using Scratch. This study's quantitative results have yet to be replicated with larger and more heterogeneous participants, and in-person ethnographic analysis of interactions within the literacy space is needed to validate our interpretations. This research is underway.

Conclusion

At the conclusion of CSCL 2017, eight provocations were presented for the future of the field, including the question of whether the community ought focus on basic research and give up trying to make tangible change in the educational system (Wise & Schwartz, 2017). The prospect of widespread adoption of computer science and interdisciplinary computational thinking potentially means that the debate will shift from whether computers should be used in schools, to how. This ought to reinvigorate sociocultural research on how schools can support vibrant human-computer activity systems which attend to questions of culture, identity, power, and privilege, as well as developing students' technical skills. This paper contributes to the proposition that we do not have to choose between those goals.

References

Beller, M., Spruit, N., & Zaidman, A. (n.d.). How developers debug.

Bonar, J., & Soloway, E. (1985). Preprogramming knowledge: A major source of misconceptions in novice programmers. *Human-Computer Interaction, 1*(2), 133–161.

Buechley, L. (2006). A Construction Kit for Electronic Textiles. In *2006 10th IEEE International Symposium on Wearable Computers* (pp. 83–90). Montreux, Switzerland: IEEE.

Burke, Q., & Kafai, Y. B. (2010). Programming & storytelling: opportunities for learning about coding & composition (p. 348). ACM Press.
Burke, Q., O'Byrne, W. I., & Kafai, Y. B. (2016). Computational Participation: Understanding Coding as an Extension of Literacy Instruction. Journal of Adolescent & Adult Literacy, 59(4), 371–375.

Denning, P. J. (2017). Remaining trouble spots with computational thinking. *Communications of the ACM*, 60(6), 33–39.

Flood, V. J., DeLiema, D., Harrer, B. W., & Abrahamson, D. (2018). Enskilment in the Digital Age: The Interactional Work of Learning to Debug, 2.

Friend, M. (2016). *Women's History, Attitudes, and Experiences with Computing*. Dissertation.

Inkle. (2016). *Ink*. London, UK. Retrieved from https://github.com/inkle/ink

Ito, M. (2010). *Hanging out, messing around, and geeking out: Kids living and learning with new media*. Cambridge, MA: MIT Press.

K-12 Comptuer Science Framework. (2016).

Kelleher, C., & Pausch, R. (2007). Using storytelling to motivate programming. *Communications of the ACM*, 50(7), 58.

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching. *Educational Psychologist*, 41(2), 75–86.

McCauley, R., Fitzgerald, S., Lewandowski, G., Murphy, L., Simon, B., Thomas, L., & Zander, C. (2008). Debugging: a review of the literature from an educational perspective. *Computer Science Education, 18*(2), 67–92.

Pea, R. (1986). Language-independent conceptual bugs in novice programming. *Journal of Educational Computing Research*, 21, 25–36.

Piorkowski, D. J., Fleming, S. D., Kwan, I., Burnett, M. M., Scaffidi, C., Bellamy, R. K. E., & Jordahl, J. (2013). The whats and hows of programmers’ foraging diets. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’13* (p. 3063). Paris, France: ACM Press.

Proctor, C., & Blikstein, P. (2019). Unfold Studio: Supporting critical literacies of text & code. *Information and Learning Science*, 2(1).

Proctor, C., & Garcia, A. (2019). Hogg, L., & Stockbridge, K. (Eds.). Student voices in the digital hubbub. *Giving student voice due weight: Possibilities and challenges in USA and New Zealand*.

Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., ... others. (2009). Scratch: programming for all. *Communications of the ACM*, 52(11), 60–67.

Ware, P. D., & Warschauer, M. (2005). Hybrid literacy texts and practices in technology-intensive environments. *International Journal of Educational Research*, 43(7–8), 432–445.

Weaver, C. (1996). *Teaching Grammar in Context*. Portsmouth, NH: Boynton/Cook.

Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., & Wilensky, U. (2016). Defining Computational Thinking for Mathematics and Science Classrooms. *Journal of Science Education and Technology*, 25(1), 127–147.

Weintrop, D., & Wilensky, U. (2016). Bringing Blocks-based Programming into High School Computer Science Classrooms, AERA.

Werner, L., Denner, J., Campe, S., & Kawamoto, D. C. (n.d.). The fairy performance assessment: measuring Cauleycomputational thinking in middle school, 6.

Wise, A. F., & Schwarz, B. B. (2017). Visions of CSCL: eight provocations for the future of the field. *International Journal of Computer-Supported Collaborative Learning*, 12(4), 423–467.
# Does Order Matter?
## Investigating Sequential and Cotemporal Models of Collaboration

Zachari Swiecki, University of Wisconsin—Madison, swiecki@wisc.edu

Zheming Lian, University of Wisconsin—Madison, zlian4@wisc.edu

Andrew R. Ruis, University of Wisconsin—Madison, arruis@wisc.edu

David Williamson Shaffer, University of Wisconsin—Madison and Aalborg University Copenhagen, dws@education.wisc.edu

**Abstract:** Many researchers have argued that models of collaborative processes should account for temporality, but there exist different approaches for doing so. We compared two specific approaches to modeling collaborative processes in a CSCL context: Epistemic Network Analysis, which models events cotemporally (unordered and temporally proximate), and Sequential Pattern Mining, which models events sequentially (ordered and temporally proximate). Our results suggest that in this context cotemporal models constructed with Epistemic Network Analysis outperform sequential models constructed with Sequential Pattern Mining in terms of (a) explanatory power, (b) efficiency, and (c) interpretability.

## Introduction

A central claim of computer-supported collaborative learning (CSCL) research is that collaborative processes influence group performance, and studies have shown that these processes have an important temporal dimension (Reimann, 2009). Researchers thus argue that models of collaborative processes should account for temporality (McGrath & Tschan, 2004), but there exist different approaches for doing so. *Sequential models*, which identify ordered patterns, can be used to investigate whether specific sequences of temporally proximate discourse moves, such as talk or actions, explain variation in group outcomes (Kapur, 2011). *Cotemporal models*, which identify unordered but temporally proximate patterns, can be used to investigate whether discourse moves that co-occur within some period of time explain variation in group outcomes (Siebert-Evenstone et al., 2017).

Prior work has shown that both cotemporal models (Csanadi, Eagan, Shaffer, Kollar, & Fischer, 2019) and sequential models (Kapur, 2011) have advantages over atemporal models such as coding and counting; however, no empirical comparisons have been made between cotemporal models and sequential models in collaborative contexts. Thus, it is unclear whether accounting for temporal proximity, versus both temporal proximity and sequence, is a more effective approach for modeling collaborative processes. Such a comparison will help inform researchers about the conditions under which one class of models outperforms another, potentially impacting assessments of complex thinking and performance. But more importantly, comparing techniques that model temporality in terms of local sequence versus local cotemporality can provide insight into the nature of temporality itself in CSCL contexts.

In this paper, we present an initial attempt to address this issue by comparing two specific modeling approaches: Epistemic Network Analysis (Shaffer, Collier, & Ruis, 2016), which models events cotemporally, and Sequential Pattern Mining (Srikant & Agrawal, 1996), which models events sequentially. We use both models to analyze data collected from air defense warfare (ADW) teams participating in computer-simulated training scenarios. We compare the efficacy of these models at finding differences in group performance in terms of (a) explanatory power, (b) efficiency, and (c) interpretability.

## Theory

Researchers in CSCL argue that collaboration has an important temporal dimension. For example, Kozlowski & Illgen (2006) argue that repeated interactions between individuals create behavioral, cognitive, or motivational states that influence future interactions. Similarly, Clark (1996) argues that as collaborative activities unfold in time, information is added to the *common ground*, the set of shared knowledge and experiences that exist between people when they interact, which in turn influences subsequent actions and interpretations (Dillenbourg, 1999).

Models of collaboration that do not account for temporality thus omit crucial information, limiting their validity (Kapur, 2011). One prevalent response to this critique has been to focus on *sequences* of discourse moves. Sequence is potentially important because, as Reimann (2009) argues, “human learning is inherently cumulative, [and] the sequence in which experiences are encountered affects how one learns and what one learns.” Moreover, in some collaborative settings, discourse moves should be carried out in a particular order. For example, Hutchins’ (1995) study of quartermasters in the U.S. Navy showed that navigation teams needed to follow a particular sequence of actions in order to accurately track the position of their ship. *Sequential models*, such as Sequential
Pattern Mining (SPM), have been used to identify sequences of discourse moves that groups make during collaborative processes in a variety of contexts (Perera et al., 2009).

There is, however, another aspect of the temporality of collaborative processes: *temporal proximity*. Events at any point in time are influenced by prior actions. However, the influence of prior activity does not always span the entire history of group interaction. Halpin and von Davier (2017) argue that the actions of one part of a group make actions by other parts more or less likely in the near future. For example, when one group member asks a question, others are likely to respond soon after. Suthers & Desiato (2012) argue that actions and interactions are interpreted with respect to the *recent temporal context*, or the immediately preceding events. Discourse moves within the recent temporal context may influence one another, directly reference one another, or build upon one another. Thus, while collaborative processes are composed of complex interactions among individuals, the most *relevant* interactions are bounded by temporal proximity.

Many SPM algorithms allow researchers to account for recent temporal context using sliding windows. Such algorithms add the constraint that the identified sequences must occur within a given window of events. However, an alternative approach is to model temporality based on co-occurrence rather than sequence, which focuses on temporal proximity irrespective of order.

In *cotemporal* models, two discourse moves are meaningfully connected if they co-occur within the same recent temporal context (Shaffer, 2017). For example, Epistemic Network Analysis (ENA) can be used to identify the connections groups make during collaboration (Shaffer et al., 2016). ENA identifies these connections by measuring how often particular discourse moves co-occur within the recent temporal context, operationalized as a sliding window that moves through each event—for example, problem step or turn of talk—in the data. ENA represents connections between discourse moves using undirected network models, meaning that connections between moves A and B in the network could mean that A followed B or that B followed A. In this way, ENA is *sensitive* to the order of events in the data—changing the order of events changes which events are present in a given window, and thus changes the results of the model—but the order in which discourse moves occur within any window is not represented in the model. ENA has been used to study CSCL and collaborative problem solving processes in many domains (e.g., Arastoopour, Shaffer, Swiecki, Ruis, & Chesler, 2016; Sullivan et al., 2018).

While it may seem counterintuitive to ignore the local order of discourse moves—after all, we perceive human actions as unfolding linearly in time—there are clearly contexts in which the specific order of these moves is less important than their cotemporality. For example, in complex and ill-formed problem solving, groups might consider issues A, B, and C at one point in the problem solving process; however, it may make little difference whether in that brief span of time they talk about A then B then C, or C then B then A—or any of the possible orderings of those issues. This approach has potential advantages over sequence models. For example, the results of SPM are lists of ordered patterns. A typical analysis identifies frequent patterns using SPM, then clusters those patterns using similarity metrics. Finally, researchers interpret these clusters in terms of the patterns within them (Jovanović, Gašević, Dawson, Pardo, & Mirriahi, 2017). In contrast, ENA produces network models for each unit of analysis, and provides an integrated visualization that helps to interpret the dimensions along which groups of networks differ. Because visualizations can reduce cognitive load in making inferences (Norman, 1993), cotemporal models with integrated visualizations may have interpretive advantages. Moreover, the number of possible permutations of discourse moves rises far more rapidly than their possible pairwise combinations as the number of significant types of moves increases. So in a case where specific ordering is not relevant, sequential models, which distribute sample variance across many variables, may have less explanatory power and may be overfit unless the analyzed dataset is large.

In this paper, we present an attempt to compare cotemporal and sequential approaches to modeling collaboration: ENA, which models events cotemporally, and SPM, which models events sequentially. We use both models to analyze data collected from air defense warfare teams (ADW) as they participated in computer-simulated training scenarios. In these scenarios, ADW teams detect, identify, assess, and take action toward nearby radar contacts. In theory, this process should follow a specific sequence for each contact; however, the demands of the task may lead to deviations from the sequence. Thus, the collaborative problem is neither so well-formed that only specific sequences are of interest nor so ill-formed that sequences are likely to be of little interest. We use this data to compare two hierarchical linear models—one using predictors derived from ENA and one using predictors derived from SPM—to assess whether ENA or SPM provides a more effective model of group performance in terms of (a) explanatory power, (b) efficiency, and (c) interpretability.

## Methods

As part of the Tactical Decision Making Under Stress project, 16 teams composed of six members each participated in four simulated training scenarios to test the impact of a decision-support system on team performance (Johnston, Poirier, & Smith-Jentsch, 1998). During the scenarios, teams performed the *detect-to-
engage sequence. A watch-station provided basic information about identification and behavior of ships and aircraft in the vicinity (referred to as tracks). Teams needed to detect and identify multiple tracks, often simultaneously, assess whether they were threats, and decide how to respond, although the full sequence of actions did not apply to every track. (For example, non-hostile tracks did not require a response.) Teams in the control condition (n = 8) had access to standard watch-stations. Teams in the experimental condition (n = 8) had access to watch-stations enhanced with information about the tactical situation. Each team participated in the same four 30-minute scenarios; scenario order was counterbalanced using a Latin square.

We analyzed two data sources for each team-scenario (that is, each team in each scenario): (1) a transcript of team communications and (2) a teamwork behavior score. Transcripts were segmented into 12,027 turns of talk. Teamwork behavior was assessed in a prior study using the Air Defense Warfare Team Observation Measure (ATOM) (Johnston, Smith-Jentsch, & Cannon-Bowers, 1997), which summarizes four dimensions of teamwork performance—supporting behavior, leadership, information exchange, and communication—into an overall score from 1 (worst) to 55 (best).

## Coding

We analyzed the transcripts using the codes in Table 1, which were developed using a grounded analysis informed by both the existing ADW literature (e.g., Paris et al., 2000) and prior qualitative analyses conducted on similar data (e.g., Morrison et al., 1996). To code the data, we developed automated classifiers for each of the codes in Table 1 using the ncodeR package for the statistical programming language R (Marquart, Swiecki, Eagan, & Shaffer, 2018). We assessed concept validity by requiring that two human raters achieve acceptable values of Cohen's kappa (κ > 0.65) with statistically significant values of Shaffer's rho (ρ < 0.05), and we assessed reliability by requiring that both human raters independently achieve acceptable values of kappa and rho compared to the automated classifier (1). For each code, all pairwise combinations of raters (humans and automated classifier) achieved κ > 0.80 and ρ(0.65) < 0.05.

<table><caption>Table 1: Qualitative codes, definitions, and examples</caption><thead><tr><td>Code</td><td>Definition</td><td>Example</td></tr></thead><tbody><tr><td>Detection</td><td>Talk about radar detection of a track or the identification of a track, (e.g., vessel type).</td><td>IR/EW NEW BEARING, BEARING 078 APQ120 CORRELATES TRACK 7036 POSSIBLE F-4</td></tr><tr><td>Track Behavior</td><td>Talk about kinematic data about a track or a track's location</td><td>AIR/IDS TRACK NUMBER 7021 DROP IN ALTITUDE TO 18 THOUSAND FEET</td></tr><tr><td>Assessment</td><td>Talk about whether a track is friendly or hostile, the threat level of a track, or indicating tracks of interest</td><td>TRACKS OF INTEREST 7013 LEVEL 5 7037 LEVEL 5 7007 LEVEL 4 TRACK 7020 LEVEL 5 AND 7036 LEVEL 5.</td></tr><tr><td>Status Updates</td><td>Talk about procedural information, e.g., track responses, or talk about tactical actions taken by the team</td><td>TAO ID, STILL NO RESPONSE FROM TRACK 37, POSSIBLE PUMA HELO.</td></tr><tr><td>Seeking Information</td><td>Asking questions regarding track behavior, identification, or status.</td><td>TAO CO, WE'VE UPGRADED THEM TO LEVEL 7 RIGHT?</td></tr><tr><td>Recommendation</td><td>Recommending or requesting tactical actions</td><td>AIR/TIC RECOMMEND LEVEL THREE ON TRACK 7016 7022</td></tr><tr><td>Deterrent Orders</td><td>Giving orders meant to warn or deter tracks.</td><td>TIC AIR, CONDUCT LEVEL 2 WARNING ON 7037</td></tr><tr><td>Defensive Orders</td><td>Giving orders to prepare defenses or engage hostile tracks</td><td>TAO/CO COVER 7016 WITH BIRDS</td></tr></tbody></table>

## Epistemic Network Analysis

To conduct a cotemporal analysis, we used the rENA package for the statistical programming language R (Marquart, Swiecki, Collier, et al., 2018). ENA uses a sliding window to construct a network model for each turn of talk in the data. Connections in the network are defined as the co-occurrence between codes in the current turn of talk and codes within the recent temporal context, which we defined as each line plus the four previous lines
based on our qualitative analysis of the data (a window size of 5 turns of talk). The resulting networks are aggregated for all turns of talk for each unit of analysis (team-scenario), such that each team-scenario is represented by a vector whose elements are the number of co-occurrences between each pair of codes for that team-scenario. ENA normalizes the matrix of co-occurrence vectors to account for variation in the amount of talk between teams and performs a dimensional reduction on the matrix via singular value decomposition.

Networks were visualized using two coordinated representations: (1) an *ENA score*, which represents the location of a team-scenario's network in the space (or *ENA space*) created by the dimensional reduction, and (2) a weighted network graph in which the nodes correspond to codes, and the edges are proportional to the relative frequency of connection between two codes. The positions of the network graph nodes are fixed across networks, and those positions are determined by an optimization algorithm that minimizes the difference between the ENA scores and their corresponding network centroids. Thus, ENA scores toward the extremes of a dimension have network graphs with strong connections between nodes located on the extremes. As a result, dimensions in this ENA space distinguish team-scenarios in terms of cotemporality between codes whose nodes are located at the extremes.

## Sequential Pattern Mining

To conduct a sequential analysis, we used the *TraMineR* package for the statistical programming language R (Gabadinho, Ritschard, Müller, & Studer, 2011). SPM identifies frequent patterns using a *support threshold*, where the support for a given pattern is the percentage of sequences that contain at least one instance of the pattern. We defined a sequence as the ordered list of codes across all turns of talk for a given team-scenario. In our data, codes can co-occur within a single turn of talk, so the SPM algorithm treats these codes as occurring simultaneously by defining *event sequences* which may contain patterns of un-ordered as well as ordered events (Ritschard, Bürgin, & Studer, 2013). We used a support threshold of 0.75 to limit the total number of patterns returned by the algorithm, which eases interpretation, and a window size of five turns of talk to match our ENA model (2). To interpret the SPM results, we counted the frequency of each high-support pattern for each team-scenario and applied Principal Components Analysis (PCA) to this data, resulting in a *PCA score* for each team-scenario—that is, the position of each team-scenario on the PCA dimensions (3). To interpret the PCA results, we used the dimension loadings, which show how much each variable—in this case, each frequent pattern—contributes to each dimension.

## Model comparison

We compared cotemporal and sequential models by constructing two Hierarchical Linear Models (HLMs). HLM is a regression technique for data with a nested-structure: in this case, team-scenarios (level-one) were nested into teams (level-two). In both HLMs, teams were random effects, the team-scenario ATOM score was the outcome variable at level-one, and Scenario was a control variable at level-two. For the cotemporal HLM (CT-HLM), ENA scores were explanatory variables at level-one. For the sequential HLM (S-HLM), PCA scores were explanatory variables at level-one.

We assessed model fit using two estimates of the total variance explained for HLMs: one from Snijders and Bosker (2012, TVE1) and the other from LaHuis and colleagues (2014, TVE2) (4). We assessed the efficiency of the models (model fit adjusted for number of parameters) using the Akaike information criterion corrected for small sample sizes. Following Burnham and Anderson (2004), we used a minimum AICc difference of 4 to indicate that the models were distinguishable.

## Results

### Epistemic Network Analysis

The first six ENA dimensions accounted for more variance in the data than any original variable. To reduce the chance of overfitting, we only used ENA scores on the first two dimensions in the CT-HLM. These two dimensions accounted for the highest proportion of the total variance: 51%. Figure 1 shows the average network across all team-scenarios and the ENA scores for each team-scenario in this space.

On the left side of the space are connections to Defensive Orders, Deterrent Orders, Status Updates, and Recommendations, all of which relate to actions taken by teams toward tracks. On the right side of the space are connections to Seeking Information. This suggests that the first dimension distinguishes team-scenarios in terms of whether they focused on *Tactical Actions* versus *Seeking Information*. Toward the top of the space are connections to Detection and Track Behavior; toward the bottom are connections to the remaining codes. Detection and Track Behavior relate to passing information about tracks, while the remaining codes relate to using
information about tracks. This suggests that the second dimension distinguishes team-scenarios in terms of whether they focused on *Track Information* versus *Track Processing*.

<figure><img src="image_69.png" /><figcaption>Figure 1. Average ENA network across all team-scenarios and ENA scores for each team-scenario.</figcaption></figure>

## Sequential Pattern Mining

Our SPM analysis returned 165 patterns. PCA on counts of these patterns returned 23 dimensions that accounted for more variance than any original variable. To maintain consistency with the ENA analysis, we only used PCA scores on the first three dimensions, which accounted for 53% of the variance. As there are 165 original variables, interpretation requires considering 165 loadings for each dimension. However, it is common to use only the loadings with high magnitudes for interpretation. We interpreted each dimension by considering commonalities between the ten patterns that loaded at either extreme. To conserve space, we show only the three patterns at each extreme for each dimension in Table 2.

<table><caption>Table 2: Patterns with extreme loadings on first three principal components</caption><thead><tr><th>PC1</th><th>PC2</th><th>PC3</th></tr></thead><tbody><tr><td colspan="3"><strong>Patterns with Extreme Negative Loadings</strong></td></tr><tr><td>(Track Behavior)-(Detection)-(Track Behavior)</td><td>(Assessment)-(Seeking Information)</td><td>(Track Behavior, Assessment)-(Seeking Information)</td></tr><tr><td>(Track Behavior)-(Detection, Track Behavior)</td><td>(Seeking Information)</td><td>(Deterrent Orders)</td></tr><tr><td>(Track Behavior)-(Track Behavior)-(Detection)</td><td>(Seeking Information)-(Seeking Information)-(Track Behavior)</td><td>(Status Update)-(Assessment)</td></tr><tr><td colspan="3"><strong>Patterns with Extreme Positive Loadings</strong></td></tr><tr><td>(Defensive Orders)</td><td>(Detection, Track Behavior)-(Deterrent Orders)</td><td>(Seeking Information)-(Detection)-(Detection, Track Behavior)</td></tr><tr><td>(Status Update)-(Recommendation)</td><td>(Detection)-(Deterrent Orders)</td><td>(Seeking Information)-(Detection)-(Detection)</td></tr><tr><td>(Status Update)-(Deterrent Orders)</td><td>(Status Update)-(Detection)</td><td>(Detection)-(Detection)</td></tr></tbody></table>
The extreme negative side of the first dimension includes patterns involving Track Behavior and Detection. The extreme positive side includes patterns involving Defensive Orders, Deterrent Orders, Status Updates, and Recommendations. This suggests that the first dimension distinguishes team-scenarios in terms of whether they focused on *Track Information* versus *Tactical Actions*. The extreme negative side of the second dimension includes many patterns involving Seeking Information, while the extreme positive side includes patterns involving Status Updates or ending with Deterrent Orders. This suggests that the second dimension distinguishes team-scenarios in terms of whether they focused on *Seeking Information* versus *Deterring Tracks*. Finally, the extreme negative side of the third dimension includes many patterns involving Assessment. The extreme positive side includes patterns involving Seeking Information, Detection, and Track Behavior. This suggests that the third dimension distinguishes team-scenarios in terms of whether they focused on *Assessing Tracks* versus *Exchanging Information*.

## Model comparison

We compared the CT-HLM (ENA scores as explanatory variables) to the S-HLM (PCA scores as explanatory variables). Coefficients, standard errors (with corresponding *p*-values), as well as measures of model fit and efficiency for both models are shown in Table 3.

<table><caption>Table 3: Cotemporal HLM and sequential HLM, including parameter estimates and model fit</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Intercept</th><th colspan="2">ENA</th><th colspan="3">PCA</th><th colspan="3">Scenarios</th><th colspan="3">Model Fit</th></tr><tr><th>1</th><th>2</th><th>1</th><th>2</th><th>3</th><th>B</th><th>C</th><th>D</th><th>TVE1</th><th>TVE2</th><th>AIC<sub>c</sub></th></tr></thead><tbody><tr><td>CT-HLM</td><td><strong>36.55*</strong><br>(1.06)</td><td><strong>-10.1*</strong><br>(2.77)</td><td>-0.42<br>(3.49)</td><td></td><td></td><td></td><td>2.78<br>(1.58)</td><td>-2.34<br>(1.57)</td><td>2.84<br>(1.49)</td><td>0.259</td><td>0.253</td><td><strong>377.19†</strong></td></tr><tr><td>S-HLM</td><td><strong>36.32*</strong><br>(1.11)</td><td></td><td></td><td>0.12<br>(0.09)</td><td><strong>0.38*</strong><br>(0.18)</td><td>-0.18<br>(0.14)</td><td>2.67<br>(1.61)</td><td>-1.61<br>(1.66)</td><td>3.1<br>(1.55)</td><td>0.238</td><td>0.237</td><td>381.72</td></tr></tbody></table>

( ) indicates standard error; * indicates *p* < 0.05; † indicates significantly lower AIC<sub>c</sub> score (Δ > 4)

For the CT-HLM (cotemporal), the variance estimates for the random effects were 0.67 at level two and 16.70 at level one. Of the explanatory variables, only the coefficient for scores on the first ENA dimension (ENA1) was significant. The negative coefficient (–10.1) indicates that teams with higher teamwork behavior ratings focused more on *tactical actions* toward tracks than *seeking information about tracks*.

For the S-HLM (sequential), the variance estimates for the random effects were 0.06 at level two and 17.79 at level one. Of the explanatory variables, only the coefficient for scores on the second PCA dimension (PC2) was significant. The positive coefficient (0.38) indicates that teams with higher teamwork behavior ratings focused more on *deterring tracks* than *seeking information about tracks*.

For both estimates of total variance explained, the CT-HLM performed better. Moreover, the CT-HLM had an AIC<sub>c</sub> score 4.53 points lower than the S-HLM. Thus, the two models are distinguishable: the cotemporal model performed better than the sequence model in explaining differences in team behavior scores efficiently.

## Discussion

Our results suggest that cotemporal models of collaborative processes constructed with ENA outperform sequential models constructed with SPM on several dimensions. The cotemporal models explained more of the difference between team-scenarios, and while differences in total variance explained were small, model selection for efficiency via AIC<sub>c</sub> showed that the cotemporal-HLM was distinguishable from, and preferable to, the sequential-HLM. A possible explanation is that our ENA model used 28 un-ordered pairs of codes to derive predictors for the regression analysis, while our SPM model used 165 frequent patterns. The dimensional reduction on the ENA variables needed only *two* dimensions to account for a large proportion of the total variance while SPM needed *three* dimensions to account for approximately the same proportion of the total variance. Thus, the HLM model with ENA predictors was more parsimonious and less likely to overfit the data.

Our results also suggest that cotemporal models can have interpretive advantages compared to sequence models. The ENA algorithm combined connection identification, dimensional reduction, and visualization into one technique; once we generated the model it was ready to interpret. SPM, on the other hand, required several steps of post processing. More importantly, however, interpretation using ENA is done via integrated visualizations—network graphs projected into a low-dimensional space. SPM does not include such
visualizations. Because visualizations can reduce cognitive load by replacing cognitive calculations with
perceptual inferences, ENA has an interpretive advantage compared to SPM.

Together, these results suggest that in CSCL and collaborative problem solving more generally, the specific *local order* of discourse moves may be less important than their *local cotemporality*. In the context we examined, teams were expected to follow a specific sequence of steps regarding each track; however, teams had to manage multiple tracks at once and the full sequence of actions did not apply to every track: it mattered more that specific discourse moves occurred together than that they occurred in a specific order.

Our results have several important limitations. First, we examined only one particular context of collaboration. However, the general features of this collaborative task, which contained both well-formed and ill-formed components, suggests that our findings may generalize to similar conditions in CSCL. Our future work will continue to investigate cotemporal and sequence models in similar contexts, as well as contexts that are more well-formed and more ill-formed. Second, it is possible that dimensional reduction on the SPM results via other techniques, such as clustering or factor analysis, could yield sequence models that perform better and are easier to interpret. Our results suggest that a two-cluster solution would be required to compete with ENA in terms of model efficiency, but each cluster would then need to be interpreted using more than 80 patterns. Allowing more clusters could ease interpretation but sacrifice efficiency. Similarly, we could apply a factor analysis to the SPM results using a two-factor solution, but it is unclear whether the performance and interpretability of the resulting sequence model would improve. Thus, we hypothesize results similar to those reported here. Our future work will compare cotemporal models developed using ENA to sequential models developed using clustering or factor analysis on SPM results to test this hypothesis.

Despite these limitations, our comparisons suggest that in CSCL contexts, cotemporal models can outperform sequential models. More importantly however, our results suggest that in some CSCL contexts, local order appears to be less important than local cotemporality. These results have implications for research and assessment in CSCL—in contexts that share both well-formed and ill-formed features, cotemporal models are a potentially more effective means of assessing complex thinking and performance. In turn, these models may better inform pedagogy and learning in such contexts.

## Endnotes

(1) Shaffer's rho is a Monte Carlo rejective statistic that quantifies Type I error for generalizing from a sample of data coded by two raters to the true rate of agreement.

(2) There are no established guidelines for choosing a support threshold. We tested multiple support thresholds above and below 0.75, and the quantitative results of the HLM comparisons were similar in all cases.

(3) Performing a cluster analysis on the high-support patterns is more commonly used to interpret the results of SPM; however, methods are not reliably available for event sequences. One implementation exists (Ritschard et al., 2018), but it has not been validated. PCA has a standard method of interpretation and is agnostic to the type of pattern.

(4) Because these models are non-nested, we were unable to test for significant differences in model fit.

## References

Arastoopour, G., Shaffer, D. W., Swiecki, Z., Ruis, A. R., & Chesler, N. C. (2016). Teaching and assessing engineering design thinking with virtual internships and epistemic network analysis. *International Journal of Engineering Education, 32*(3B), 1492–1501.

Burnham, K. P., & Anderson, D. R. (2004). Multimodel inference: Understanding AIC and BIC in model selection. *Sociological Methods & Research, 33*(2), 261–304.

Clark, H. H. (1996). *Using language*. Cambridge university press.

Csanadi, A., Eagan, B., Shaffer, D. W., Kollar, I., & Fischer, F. (2019). When coding-and-counting is not enough: Using Epistemic Network Analysis (ENA) to analyze verbal data in CSCL research. *International Journal of Computer-Supported Collaborative Learning*.

Dillenbourg, P. (1999). *Collaborative learning: Cognitive and computational approaches. advances in learning and instruction series*. ERIC.

Gabadinho, A., Ritschard, G., Müller, N. S., & Studer, M. (2011). Analyzing and visualizing state sequences in R with TraMineR. *Journal of Statistical Software, 40*(4), 1–37.

Halpin, P. F., & von Davier, A. A. (2017). Modeling Collaboration Using Point Process. In *Innovative Assessment of Collaboration* (pp. 233–247). Springer.

Hutchins, E. (1995). *Cognition in the wild*. Cambridge, MA: MIT Press.

Johnston, J. H., Poirier, J., & Smith-Jentsch, K. A. (1998). Decision making under stress: Creating a research methodology. In J. A. Cannon-Bowers & E. Salas (Eds.), *Making decisions under stress: Implications for individual and team training* (pp. 39–59). Washington, D.C.: American Psychological Association.
Johnston, J. H., Smith-Jentsch, K. A., & Cannon-Bowers, J. A. (1997). Performance measurement tools for enhancing team decision making. In M. T. Brannick, E. Salas, & C. Prince (Eds.), *Team performance assessment and measurement: Theory, method, and application* (pp. 331–327). Hillsdale, NJ: Erlbaum.

Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning Analytics to unveil learning strategies in a flipped classroom. *The Internet and Higher Education, 33*, 74–85.

Kapur, M. (2011). Temporality matters: Advancing a method for analyzing problem-solving processes in a computer-supported collaborative environment. *International Journal of Computer-Supported Collaborative Learning*, 6(1), 39–56.

Kozlowski, S. W., & Ilgen, D. R. (2006). Enhancing the effectiveness of work groups and teams. *Psychological Science in the Public Interest*, 7(3), 77–124.

LaHuis, D. M., Hartman, M. J., Hakoyama, S., & Clark, P. C. (2014). Explained variance measures for multilevel models. *Organizational Research Methods*, 17, 433–451.

Marquart, C. L., Swiecki, Z., Collier, W., Eagan, B., Woodward, R., & Shaffer, D. W. (2018). rENA: Epistemic Network Analysis (Version 0.1.3).

Marquart, C. L., Swiecki, Z., Eagan, B., & Shaffer, D. W. (2018). ncodeR (Version 0.1.2).

McGrath, J. E., & Tschan, F. (2004). *Temporal matters in social psychology: Examining the role of time in the lives of groups and individuals*. Washington: American Psychological Association.

Morrison, J. G., Kelly, R. T., Moore, R. A., & Hutchins, S. G. (1996). Tactical decision making under stress (TADMUS) decision support system, 13.

Norman, D. A. (1993). *Things that make us smart: Defending human attributes in the age of the machine*. New York: Basic Books.

Paris, C., Johnston, J. H., & Reeves, D. (2000). A Schema-Based Approach to Measuring Team Decision Making in a Navy Combat Information Center. In *The Human in Command* (pp. 263–278). Springer, Boston, MA.

Perera, D., Kay, J., Koprinska, I., Yacef, K., & Zaïane, O. R. (2009). Clustering and sequential pattern mining of online collaborative learning data. *IEEE Transactions on Knowledge and Data Engineering*, 21(6), 759–772.

Reimann, P. (2009). Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. *International Journal of Computer-Supported Collaborative Learning*, 4(3), 239–257.

Ritschard, G., Bürgin, R., & Studer, M. (2013). Exploratory mining of life event histories. In J. J. McArdle & G. Ritschard (Eds.), *Contemporary Issues in Exploratory Data Mining in the Behavioral Sciences* (pp. 221–253). New York: Routledge.

Ritschard, G., Studer, M., Buergin, R., Gabadinho, A., Fonta, P.-A., Muller, N., & Rousset, P. (2018). TraMineRextras: TraMineR extension (Version 0.43).

Shaffer, D. W. (2017). *Quantitative ethnography*. Madison, WI: Cathcart Press.

Shaffer, D. W., Collier, W., & Ruis, A. R. (2016). A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. *Journal of Learning Analytics, 3*(3), 9–45.

Siebert-Evenstone, A., Arastoopour Irgens, G., Collier, W., Swiecki, Z., Ruis, A. R., & Williamson Shaffer, D. (2017). In Search of Conversational Grain Size: Modelling Semantic Structure Using Moving Stanza Windows. *Journal of Learning Analytics, 4*(3), 123–139.

Snijders, T. A. B., & Bosker, R. J. (2012). *Multilevel analysis: An introduction to basic and advanced multilevel modeling*. Thousand Oaks, CA: Sage.

Srikant, R., & Agrawal, R. (1996). Mining sequential patterns: Generalizations and performance improvements. *Advances in Database Technology—EDBT*, 1–17.

Sullivan, S. A., Warner-Hillard, C., Eagan, B. R., Thompson, R., Ruis, A. R., Haines, K., ... Jung, H. S. (2017). Using epistemic network analysis to identify targets for educational interventions in trauma team communication. *Surgery*, in press.

Suthers, D. D., & Desiato, C. (2012). Exposing chat features through analysis of uptake between contributions. In *System Science (HICSS), 2012 45th Hawaii International Conference on* (pp. 3368–3377). IEEE.

## Acknowledgments

This work was funded in part by the National Science Foundation (DRL-1661036, DRL-1713110), the U.S. Army Research Laboratory (W911NF-18-2-0039), the Wisconsin Alumni Research Foundation, and the Office of the Vice Chancellor for Research and Graduate Education at the University of Wisconsin-Madison. The opinions, findings, and conclusions do not reflect the views of the funding agencies, cooperating institutions, or other individuals.
# Teacher Monitoring Routines: Understanding Pedagogical Judgments During Students' Collaborative Learning

Nadav Ehrenfeld and Ilana S. Horn (1)

Nadav.Ehrenfeld@vanderbilt.edu, Ilana.Horn@vanderbilt.edu

Peabody College, Vanderbilt University

**Abstract:** In this paper we conceptualize *teacher monitoring routines* – a consequential yet understudied phase of instruction during students’ collaborative work. We examine 5 lessons of experienced and equity-oriented secondary math teachers using a comparative case study design and analysis of their interactional routines. Our goal is to understand how teachers monitor groupwork and how they decide (intentionally or not) when, where, and how to intervene in student talk. We identified various patterns that make clearer the micro-judgments monitoring routines entail for teachers’ (a) ways of moving among students and conversational initiation, (b) conversational entry; (c) focus of the interaction; (d) when and how to exit the interaction; and (e) conversation participation pattern; with an eye toward teachers’ goals and instructional contexts. We end by considering implications for research on teaching and professional education in support of collaborative learning.

## Purpose

As mathematics teachers shift toward what has been called “ambitious instruction” (Lampert et al., 2011), they frequently build on collaborative groupwork to promote students’ sensemaking of mathematical ideas through interaction (Cohen & Lotan 2014; Horn, 2012; Smith & Stein, 2011). While teachers’ judgments during groupwork involve managing tensions between intervention and student autonomy, these micro-judgments remain understudied and ill defined (Stein et al., 2008; Webb et al., 2009). Different scholars who study this phase of lessons suggest different roles for teachers, with some advocating that teachers primarily listen to conversations (Smith & Stein, 2011) or intervene only when a group is “hopelessly off task” (Cohen & Lotan 2014), while others suggest it is more productive for the teacher to be more active and probe students’ thinking (Webb et al., 2009).

Our own interest in studying teacher monitoring routines did not arise as a theoretical gap to fill, but rather as an in vivo problem of practice we encountered as teacher educators. For the past several years, we have partnered with a professional development organization (PDO) to bridge formal professional development with classroom teaching through video-based coaching. As we have spent time in the teachers’ classrooms and discussed their teaching with them, we have noticed two things: (1) There is a surprising variability across teachers’ groupwork monitoring practices; and (2) The “shakiest” moments of lessons tend to occur in the monitoring phase. While tasks, launches, and instructional routines can be easily shared among teachers, monitoring is inherently interpretive and responsive, requiring more of the teachers’ sensemaking in-the-moment. The uncertainties involved in this phase of the lesson, along with the imperative to support cognitive demand and productive engagement for all students, provoked our interest in identifying the sequence of judgments teachers make during this phase of their lessons. To this end, we offer a framework for teachers’ *monitoring routines* as a sequence of moves, which we name *Initiation-Entry-Focus-Exit*, to better understand the choices teachers make. Building on examples from 5 secondary math teachers, this paper aims to probe and conceptualize teacher monitoring routines, and to make clearer the myriad micro-judgments (intentional or not) they entail for teachers.

## Theoretical perspectives

### Interactional routines

In their basic form, interactional routines are recurrent *moves* constituting the patterned ways conversations unfold within a certain social group, as within a classroom. However, while routines might have some recognizable structure, interactional routines are also *emergent*, as their details are contingent on the situation. In this way, interactional routines are both stable and performative, reflecting structure in their consistent aspects as well as the agency of those who use them (Feldman & Pentland, 2003; Horn & Little, 2010; Lavie et al., 2018).

Attuning to the tensions of structure and agency, our conceptualization of monitoring routines builds on two definitions. The first comes from Cazden (2001), who describes interactional routines as holding both *sequential* and *selectional* dimensions. Cazden compares these two dimensions to a Western restaurant menu, likening the sequential dimension to the set of categories (e.g. appetizer, entrée, dessert) that is culturally set and rarely varies, while the selectional dimension is the specific appetizer, entrée or dessert that a person orders. For
monitoring routines, the structural, sequential dimension is captured in our generic framework of initiation, entry, focus, and exit, coupled with students access to participation in the interaction. The agentic, selectional dimensions are then the particular choices teachers make as they initiate, enter, focus, and exit interactions, based on their interpretations and responses to the situation.

For that reason, we find it compatible to use Lavie and colleagues' (2018) definition of interactional routines as a *task-procedure pair*:

> "a routine performed in a given task situation, by a given person, is the task as seen by the performer, together with the procedure she executed to perform the task." (Lavie, Steiner & Sfard, 2018, p. 9).

For example, in the case of groupwork monitoring routines, one teacher might see the *initiation move* as a task in which it is the teacher's responsibility to check-in with the groups. Another teacher might interpret the same task with different reasoning — for instance, as a situation where teacher-initiated conversation might interrupt students' thinking and thus would only approach a group if students had a question. Either way, some relatively stable structure of local classroom culture would be constructed, and students would know whether they need to raise their hands or simply wait for the teacher to come over. As we mentioned before, such teacher sensemaking is necessarily situated in the particularities of their teaching situations, including the teachers' instructional goals and their institutional constraints.

## Situated view of teaching

Teaching is an irreducibly situated act, as teachers navigate complex relational and institutional terrain during instruction. By taking a situated view, we acknowledge teachers' instruction as co-constructed with the particularities of their teaching situations (Greeno, 1998; Horn & Kane, 2015; Lave & Wenger, 1991). As a consequence, teachers' monitoring practices are an interactional accomplishment between teachers and the particular students, content, and settings they are working with. Although routines may stay somewhat stable within individual teachers as they move between classrooms or groups, we also see teachers making micro-judgments about whom they direct their attention to, in what ways, and for how long. For this reason, while abstract and idealized notions of monitoring practices may serve as a useful heuristic guide to practice, teachers necessarily exercise *pedagogical judgment* as they enact any instructional practice in their teaching situations (Horn, in press).

Applying this lens to the teachers in our study, we are especially interested in how their pedagogical judgment shapes the ways they monitor student worktime. By highlighting teachers' micro-judgments as they monitor students' work, we claim that teachers do more than simply implement known models of groupwork; teachers are constantly assessing, interpreting, and adjusting as groupwork unfolds in their particular classroom contexts.

## Research question

Given our goal to probe and conceptualize teacher monitoring routines and considering our theoretical perspective, we ask: *What interactional routines emerge during the focal teachers' conversations with student groups during the monitoring phase of the lessons?* In the rest of the paper, we answer this question, with an eye towards the teacher micro-judgments that underlie these routines.

## Data and methods

### Research context and data

As we mentioned, this study comes from a research practice partnership between our university research team and a PDO. Our shared work centers on bridging formal professional development and classroom instruction, and we have worked to design a video-based formative feedback (VFF) coaching cycle to help elicit and engage secondary math teachers' pedagogical judgment. At the PDO, the secondary mathematics teachers have encountered different models of monitoring groupwork, including the aforementioned *5 Practices* (Smith & Stein, 2011) as well as Complex Instruction (Cohen & Lotan 2014; Horn, 2012). In addition, they have attended workshops on the TRUmath Framework (Schoenfeld, 2014), which highlights issues of student agency and access to rich mathematics. In sum, our participants have had atypically rich opportunities to learn about ambitious instruction in formal professional development.

For this analysis, we looked at 5 lessons (see Table 1) from experienced secondary teachers in the PDO (5 to 18 years of teaching experience, mean = 8 years), in a large urban district in the western U.S. They were
selected because they included segments of 20 minutes or longer of teachers' monitoring students' mathematical groupwork (20 to 45 min segments, mean = 28.5 min). For all 5 lessons, we have two cameras recording the class session. Camera 1, a tablet camera on a robot tripod, captured the whole class, with a focus on the teachers' movements through the classroom. These video records also have student audio tracks captured through 4 separate microphones placed at students' desks. Camera 2, a point of view camera, was mounted on the teacher's head, shoulder, or chest to approximate what they were seeing as they moved through the classroom and interacted with students. In addition to these recordings, our data include lesson artifacts, conversations with the focal teachers before and after instruction, and email exchanges about the activities.

<table><caption>Table 1: information about the focal teachers and the lessons</caption><thead><tr><td>Focal teacher</td><td>Experience (years)</td><td>School</td><td>Class</td><td>Date</td><td>Groupwork (minutes) (%)</td></tr></thead><tbody><tr><td>Veronica Kennedy</td><td>8</td><td>Rees</td><td>8<sup>th</sup> grade math</td><td>Feb 1 2018</td><td>33 of 82 (40%)</td></tr><tr><td>Brad Miller</td><td>6</td><td>Noether</td><td>Algebra 1</td><td>May 7 2018</td><td>20 of 58 (34%)</td></tr><tr><td>Bridgette Campbell</td><td>7</td><td>Johnson</td><td>Algebra 1</td><td>Oct 05 2017</td><td>20 of 85 (23%)</td></tr><tr><td>Lee Bellver</td><td>14</td><td>Falconer</td><td>Algebra 1</td><td>Mar 1 2018</td><td>26 of 100 (26%)</td></tr><tr><td>Lizette McLoughlin</td><td>5</td><td>Fermat</td><td>AP Calculus</td><td>Feb 2 2018</td><td>45 of 55 (82%)</td></tr></tbody></table>

# Data analysis

## Phase 1: Data reduction and re-representation

We edited the video to isolate the monitoring sequences of each lesson, reducing 380 min of classroom footage to 144 min. Then, using Final Cut Pro X, we edited the two camera perspectives together as a picture-in-picture to support coordinated analysis of the teachers' actions and a view of where they are located in the classroom. (See Figure 1.)

<figure><img src="image_70.png" /><figcaption>Figure 1. Two camera perspectives on Lizette's classroom.</figcaption></figure>

## Phase 2: The sequential dimension of teachers' monitoring routines

Using the subset of video data from Phase 1, we used inductive coding (Strauss & Corbin, 1998) to identify the sequential dimension of the *monitoring routine*. We arrived at categories that explained all 5 cases, and they were 5 recurrent *moves* that described: (a) how teachers moved around the room and how conversations were initiated (*initiation*) (b) their opening moves of conversation (*entry*); (c) the main *focus* of the interaction; (d) when and how they *exit* the interaction; (e) whether the teacher interacts with students as a group or as individuals (see Table 2). This constituted the general task of the emergent monitoring routines.

## Phase 3: The selectional dimension of teachers' monitoring routines

Once we arrived at the basic sequence of monitoring routines, we could delve into the interactional details — the particular judgments teachers made in their teaching situations. Using an inductive comparative case study design (Strauss and Corbin, 1998) and methods of interaction analysis (Jordan and Henderson, 1995), we looked closely at the videos to explore patterns of how the 5 teachers took up each of the 5 moves identified in Phase 3. The coding team consisted of two PhD students (including the first author) and a secondary math teacher. During the coding process, the team met regularly, maintained an online “coding dilemmas” document and a codebook with definitions and examples for each code. As coding dilemmas were discussed and resolved among the coding team
and the Principle Investigator (the second author), the definitions and examples were refined. Peer Debriefing was done with the larger research team (2 additional PhD students, a post-doctoral fellow, and an assistant professor) to ensure the ecological validity of the coding scheme (Lincoln & Guba, 1985).

Once the codes were more or less agreed upon, the team used Vosaic software to code the videos. Python code (in Pycharm framework) was used to process the data and to create representations that account for the teachers' time spent with each student group as well as the sequence of their visits (Figures 2,4,5).

<table><caption>Table 2: description of the 4 basic moves and participation pattern</caption><thead><tr><th>Move</th><th>Description</th><th>Codes</th></tr></thead><tbody><tr><td>Initiation</td><td>How the teacher approaches the group?</td><td><ul><li>Student-initiated: Student calls teacher over</li><li>Teacher-initiated: The teacher approaches the group of their own accord</li></ul></td></tr><tr><td>Entry</td><td>Teacher's first statement to the group</td><td><ul><li>Teacher asks for a summary or "what Ss are up to" in terms of math or participation norms</li><li>Teacher answers students' question</li><li>Teacher specifically asks about results or pacing</li><li>Teacher redirects Ss interaction to a new topic</li></ul></td></tr><tr><td>Focus</td><td>The substance of the interaction</td><td><ul><li>Teacher discusses mathematical ideas<ul><li>Probing (how? why?)</li><li>Discussing results (what?)</li><li>Hinting (pointing resources or evaluating students' ideas)</li></ul></li><li>Teacher discusses group dynamics</li><li>Teacher discusses task directions</li><li>Technical issues or brief comment</li><li>Any combination of the above</li></ul></td></tr><tr><td>Exit</td><td>The last directive the teacher gives before moving away</td><td><ul><li>Closed directive of "next steps"</li><li>Open directive of "things to think about"</li></ul></td></tr><tr><td>Participation pattern</td><td>Whom does the teacher interact with?</td><td><ul><li>Teacher discussing with whole group</li><li>Teacher discussing with individuals or subgroup</li></ul></td></tr></tbody></table>

## Findings

As teachers monitor groupwork, we noted five distinct types of moves, generally following the sequence of initiation, entry, focus, and exit, as well as an overarching participation pattern summarized in Table 2. In the following section, we discuss each of these, using two contrasting cases from our data set to illuminate and span a space of possibilities by which these moves can be taken up.

We offer three caveats about the scope of this analysis. First, our intention is not to be exhaustive: We do not claim that these contrasts capture the full breadth of how teachers might use the moves to monitor groupwork. Second, we do intend to claim that these routines capture an essence of any individual teacher. In fact, we have examples in our data set of the same teachers leading different lessons where their monitoring routines look quite different. Finally, we do not seek global, normative statements about monitoring routines. Returning to our situative perspective, we see the monitoring routine's structure as capturing a wide range of instruction across a variety of teaching situations. Instead of highlighting "best practices," we identify these phases to help teachers reflect on and deliberate on the range of micro-judgments they can make during this crucial phase of the lesson.

### Initiation

We used three codes for the initiation phase of each group interaction describing who initiated the teachers' involvement: *student(s)*, *teacher*, or *unclear*. (This third code was not used much.) Within each monitoring case, we saw patterns of *mostly student initiation*, *mostly teacher initiation*, or *mixed initiation*. These top-level patterns capture the general tendency of different teachers to intervene in student groupwork as well as the degree of student autonomy in the class.

To illustrate how initiation patterns vary, we contrast Veronica and Lizette (Figures 2a and 2b). Because these are nonstandard representations, we offer this guide to understanding what they mean: the x-axis represents
time, the y-axis represents each unique group in the class, with -1 representing moments when the teacher did not directly interact with a group. Thus, the rectangles represent discrete interactions, and their lengths represent the amount of time the teacher spent with a particular group. Green rectangles signal teacher-initiated interactions, while blue rectangles signal student-initiated ones. Immediately we see that Veronica's top-level pattern was *mostly teacher initiated*, while Lizette's was *mixed initiation*.

Using these representations, we note a few differences between Veronica and Lizette's initiation routines. First, while Lizette spent 31% of her time *quietly monitoring* (-1 on the y-axis, light blue rectangles), Veronica only spent 2% of hers in the same way. Turning to the underlying pedagogical judgment, we surmise that this may be due to the age difference between the students and, relatedly, their degree of autonomy: Veronica taught 8th grade math, while Lizette taught older students AP Calculus. From our conversations with the teachers, we also see these differences as arising from their conceptions of their role as a teacher during groupwork. For example, when we discussed Veronica's monitoring pattern with her, she shared that she did not think about standing back and watching.

Second, Veronica walked around the room in a fixed pattern from group to group, represented by the sinusoidal curve, whereas Lizette constantly changed the order that she approached student groups. Turning to the underlying pedagogical judgments, we can interpret Veronica's fixed pattern as informed by her judgment that she needs to distribute her attention evenly and help keep the students on task, whereas Lizette's irregular pattern might signal her responsiveness to students' initiation, which may be less predictable.

To reiterate, we do not seek to evaluate these moves as good or bad in an absolute sense. However, we are curious about the consequence of teachers' moves for students' opportunities for mathematical engagement. As we listened to the student audio tracks during Veronica's lesson, for instance, we noted several times that, as a consequence of her fixed pattern of teacher initiation, students' vibrant mathematical talk got interrupted by Veronica's intervention. At the same time, the fixed pattern results in a relatively equal time allocation among the groups, which, as the diagrams show, was not true for Lizette's students. For example, Lizette conversed with Group 2 briefly exactly once and conversed with Group 6 only twice, with the second time lasting much longer. Of course, equal time allocation is not necessarily equitable or even productive for learning. A group that is working well might not need the teacher intervention. But all this points back to the centrality of pedagogical judgment during monitoring: What 'working well' means, for whom, and when, is a matter of interpretation. However, knowing that *not to intervene* is a sound possibility — and one that hadn't occurred to Veronica — might help support the complexity of this (intentional or not) micro-judgment.

<figure><img src="image_71.png" /><figcaption>Figures 2a & 2b. Veronica's (left) & Lizette's (right) initiation routines.</figcaption></figure>

## Entry and focus

Entry and focus capture the heart of the teachers' interactions with student groups. First, the four codes that captured teachers' entry into the conversation answered the question: *how does the teacher verbally begin the interaction with the students?* Our codes were: (1) *listening/asking for a summary*; (2) *asking about results*; (3) *answering student questions*; and (4) *redirecting interaction*. The focal teachers typically started their entry by asking where the group stands (codes 1 and 2). Code 1, listening or asking for a summary, can center on either in terms of their mathematical thinking ("so tell me about what you're doing") or participation patterns ("are they helping you out?").

Next, to capture the focus of the conversations, we asked, *what is the nature of the sensemaking throughout the conversation?* We had six codes for the focus, 3 of which involved math talk and 3 of which involved not-math talk. The three math codes were: *probing students' explanation, results-centered talk*, and *teacher hints/scaffolds*. The three not-math codes were: *norms of participation, assignment centered*, and *technical issues/brief asides*. Of our four moves in the monitoring routine, *focus* was the only one that was often double coded, as teachers commonly focused on more than one thing over the course of an interaction.

To illustrate how entry and focus can vary, we contrast Bridgette and Brad. Bridgette (Figure 3a) was the only teacher who spoke about participation more than she did about math. In comparison, Brad's entry and
focus (Figure 3b), was more typical, since most of his focus was mathematical, with a constant tension between
probing students thinking and scaffolding it.

Like Lizette, Bridgette quietly monitored her classroom more than the other teachers in our sample. However, unlike Lizette (see Figure 2b), Bridgette seldom initiated interactions during the monitoring phase (only 3 out of 16 interactions were teacher-initiated). Bridgette also assigned group roles to students and was very consistent about redirecting student questions back to their peers. In sum, at a top-level description, Bridgette's entry and focus aimed for *equitable participation*.

Brad, on the other hand, like most of our focal teachers, primarily spoke about math with student groups. Looking into the types of math talk, we see a tension between *probing students' thinking* and *hints/scaffolds*. As we mentioned above, Brad's conversations generally started with an entry move (move 2) of asking for a summary of what they have done, followed by a focus (move 3) on *probing* students' questions and thinking, concluding with a strong *hint/scaffold* by either evaluating their thinking or directing students to a next step to continue with. In sum, at a top-level description, Brad's entry and focus aimed for *guided work completion*.

By contrasting Bridgette’s equitable participation approach with Brad’s guided work completion approach, we again abstain from evaluation of their actions. Instead, we emphasize the tensions they were navigating in making the micro-judgments that inform their practice. For instance, Bridgette’s emphasis on equitable participation pressed on student involvement and centered their mathematical thinking. However, in the class session we observed, not all groups completed their task for the day, a valued outcome for many teachers, In contrast, Brad’s students, guided by his responsive hints and scaffolds, mostly completed their work, although its production may not have been as equally distributed across students. How and how much to balance probing and scaffolding, and for whom and when, again is ultimately a matter of teachers’ pedagogical judgment.

<figure><img src="image_72.png" /><figcaption>Figures 3a & 3b. Bridgette's (upper charts) & Brad's (lower charts) entry and focus routines.</figcaption></figure>

Exit

To capture the exit moves, we asked, *how does the teacher end the interaction with students?* with an eye toward how varying exits set students up for continued conversation. For this reason, we distinguished between *open* exits that point toward further exploration, and *closed* exits, which involved clearer directives. For example, an open ending might offer an issue to think about, while a closed exit would tell a next step.

To illustrate exit moves, we contrast Brad and Lizette. Using diagrams constructed similar to those illustrating the initiation move, Figures 4a and 4b show the teachers’ movement around their classroom, the time spent with each group, with yellow rectangles signaling open exits and orange rectangles signaling closed exits. Here, we see that Brad’s exit routines (Figure 4a) provide us with important insights regarding his overall monitoring routine. On the one hand, Brad elicited students thinking and had rich mathematical conversations with the groups (which we omit here for lack of space). On the other hand, all of Brad’s 11 group interactions had a closed ending (e.g., “so the vertex should come in the middle”). In contrast to Brad, Lizette (Figure 4b) had mostly open exit moves, which included less directive hinting, such as pointing to a resource (e.g. “anything we’ve talked about last week that might help here?”), generally leaving the group with something to think about rather than something to do.

Once again, the teachers’ understanding of their role during monitoring shapes these moves. The distinction between open and closed endings are critical here, because it might have important implications for how students’ conversation will unfold after the teacher walks away. In Lavie and colleagues’ (2018) terms, if
teachers interpret their task as supporting students’ collaboration in their absence, they should privilege open exits. If teachers interpret their task as wrapping up the conversation, ensuring work completion, or supporting student pacing, then they should privilege closed endings.

<figure><img src="image_73.png" /><figcaption>Figures 4a & 4b. Brad’s (left) and Lizette’s (right) exit routines.</figcaption></figure>

## Participation pattern

As we were coding the moves of the monitoring routine, we noticed differences in the overarching participation pattern between the teacher and student groups. Specifically, whom the teacher interacts with during intervention and in what ways. To illustrate participation patterns, we turn to the contrasting cases of Bridgette and Lee, two outliers in our sample. Most of the other cases show teachers moving between interacting with all students in a student group or only with some. In many cases, teachers start a conversation with certain students and slowly (and seemingly unintentionally) exclude other students from the discussion through their gaze or body positioning.

Bridgette, as we have already described, was very intentional about equitable participation. Of our 5 focal teachers, Bridgette was the only one who consistently addressed all students in the group (16 of 16 interactions). Other teachers were either less intentional about this participation pattern or clearly facilitated a different one. Lee, for example, walked between groups and made sure to talk to every student; however, he addressed them one at a time. We see this as evidence that Lee’s interpretation of his own task was less about promoting collaboration and more about supporting individual students’ understanding of content.

## Discussion

Teachers’ monitoring practices are consistent enough to be captured as emergent routines involving initiation, entry, focus, and exit, with general overarching participation patterns. We note two things about these routines. First, the different teacher monitoring routines described above have implications for sustaining students’ mathematical engagement. Our claim is not that one mode of monitoring is always appropriate, but rather that monitoring groupwork entails complex interactions, some of which teachers may not be aware of. Second, the differences in monitoring routines seem to reflect the teachers’ understandings of their role during groupwork. By examining and conceptualizing this understudied phase of instruction, we illuminate how different monitoring routines both shape the learning environment and are shaped by it.

Although we see monitoring practices as reflecting teachers’ conceptions of their role, we also stress that they are not teachers’ fixed “teaching style.” For example, while Lizette was quietly monitoring 31% of the time in this class session and mainly used open-ended exits, she was teaching an AP Calculus class. If she, like Veronica, had been teaching a middle school mathematics class, she may not have had a chance to circulate quietly, and she may have exited conversations in a content oriented way to work toward task completion — or to quell the anxieties of the younger students. Other aspects of the teaching situation surely influence monitoring routines, such as the extent to which the tasks are groupworthy (Lotan, 2003) and thus invite open ended exploration, as well as the physical space of the classroom. For instance, some of our teachers used vertical whiteboards on the perimeter of the room for groupwork. In these classrooms, the sinusoidal pattern emerged fairly consistently as teachers circulated around the room’s perimeter. (See Veronica [Figure 2a] and Lee [Figure 5b] for examples.) In contrast, teachers who are walking between desks could circulate quietly without officially being part of some group discussion.

While supports exist to aid teachers in building student-centered classrooms, this analysis highlights key points in this work, uncovering the range of possibilities for monitoring groupwork, and the complexity of the micro-judgments required in monitoring. As we better understand these routines, we can develop clearer ways to support productive equitable mathematical discourse during this crucial phase of lessons.

## Endnotes

(1) The authors contributed equally to this paper.