weeks and consisted of a series of four projects: 1) a paper-card using a simple circuit, 2) a wristband with three
LEDs in parallel, 3) a classroom-wide mural project completed in pairs that incorporated two switches to
computationally create four lighting patterns (the only collaborative project), and 4) a "human sensor" project that
used handmade sensors to create conditions for lighting effects (see Kafai & Fields, 2018). Each project allowed
increasing flexibility in design and personalization in the context of learning challenging new computing concepts.
The e-textile projects provided many opportunities for debugging since problems occurred in the code, the
circuitry, and the physical designs, requiring students to test and isolate problems, often fixing multiple co-
occurring issues that added to the complexity and challenge of the projects (Kafai et al., 2014; Fields et al., 2016;
Jayathirtha et al., 2018). The unit also drew on ECS practices such as pair programming during certain coding
lessons.

The three schools were located in a large metropolitan school district and served a large number of students who came from ethnic/racial groups that are traditionally underrepresented in CS. At the three schools, 54-95% of students were identified as socioeconomically disadvantaged as defined by the state, and were ethnically diverse (40-90% Hispanic or Latino/a, 1-25% White, 4-43% African American, 0-18% Asian, 0-10% Filipino). The classes each included 20-35 students from 9th-12th grade (14-18 years old) and were diverse in terms of and gender ratios (36-66% girls). All teachers received six days of professional development over two years: three days in the first year of the curriculum development focused on design and content learning, and three days in the second year focused on supportive pedagogical practices.

Data collection and analysis

Data for our analysis was drawn from fieldnotes, student artifacts, and interviews across the three classrooms.
Two researchers, including the first author who was also a lead designer of the unit, collected weekly field notes
in each classroom adding up to 39 fieldnotes. In addition to observational fieldnotes, we collected students' design
notebooks, journals, and end-of-unit portfolios where students reflected on challenges and learning during the
unit. Further, we interviewed 12 students from each classroom in focus groups at the end of the unit and
interviewed teachers before, during, and after the unit. Of note, the original focus our data collection was on
students' learning of computational and circuitry concepts, which matched our research agenda. However, we
recognized the central role of unplanned peer-to-peer collaborations in students' learning and perseverance in the
unit after initial analysis of students' progressions of learning with e-textiles. This led us to analyze emergent
collaboration in its own rite.

In analyzing the data, we first sought to identify all examples of emergent peer-to-peer collaboration
across our observations. We defined this to include only student collaborations outside of teacher-directed groups
(i.e., pair programing lessons) or formal partnerships (i.e., the collaborative pair-work in project #3) that related
to student work (i.e., excluding casual banter unrelated to classwork). Drawing from the principles of grounded
theory (Charmaz, 2014), two researchers initially coded a set of three field notes (close to 10% of the total), to
come to agreement on applications of this definition, and coded a second set to verify consistency. Then one
researcher culled all examples of emergent peer-to-peer collaboration in the entire dataset. We then analyzed these
moments holistically (Miles, Huberman & Saldana, 2014), often returning to fieldnotes or related data (i.e.,
portfolios where students reflected on problems), to understand what stimulated those moments of emergent
collaboration, the larger context that surrounded them, and their implications on the students. Next, using a two-
step open coding analysis (i.e., developing initial codes, analyzing the dataset in full, re-visiting and editing the
codes, and re-analyzing the dataset in full), we jointly developed the following categories which we will elaborate
on in the findings section:

* *Stimuli for collaboration:* Bugs, verbal exclamations, help other than debugging, stimulus from a lesson, other.

* *Subject of collaboration:* Problem-solving, feelings of camaraderie, extra pair of hands, other.

* Enabling environmental factors: Physical proximity, overseeing/overhearing, curricular affordances, teacher practices, distributed expertise, adult brokerage, earlier friendship

Overall, our analysis was similar to Sawyer's work (2012) in that there was a focus on individual actions that was
used to identify the events of emergent collaboration, which then guided a deeper sociocultural analysis of the
contextual factors around these events.

Findings

We identified five types of stimuli for emergent peer collaboration: bugs (a problem in a project or task);
exclamations (students saying something loudly, such as something that went well); general help (including
learning a new technique, getting advise on the aesthetics or appearance of a project or completing a necessary
task); following through on a lesson-based task that was not intended for group work, or a general "other". Figure
1 shows the frequency of instances in each of these areas as well as the type of collaboration that occurred: solving
a problem, building camaraderie between peers, working together not on a problem but on something that required
"extra hands" (i.e., holding something), and of course, "other."

<figure><img src="image_185.png" /><figcaption>Figure 1. Graphical visualization of all the instances of emergent collaborations across the themes.</figcaption></figure>

Since the majority of the instances that involved bugs (a little less than half of all emergent
collaborations), we focus the remainder of this paper on investigating this area more deeply. Bug-related events
might involve realizing a breakdown, identifying a bug, fixing, testing or otherwise debugging a project or
celebrating fixing a bug. Collaborations stimulated by bugs included directly helping in problem-solving (red and
orange dots in Figure 1) or providing emotional support (i.e., "camaraderie"—orange and yellow) by fostering a
sense of camaraderie around making mistakes by encouraging, being comical or even heckling. In order to share
the wide variety of technical and emotional supports that students provided to each other, below we explicate
three examples of peer collaboration which illuminate the diversity of supports and immediate contextual factors
that enabled the collaboration. We also include information about the significance of these moments for students
who were involved in these collaborations. At the end we consider the broader contextual factors that enabled
these emergent collaborations, including the tools, spaces, classroom and lesson structures, classroom practices,
and student and teacher roles.

**Observations of emergent collaborations**

Initiating peer-to-peer debugging

One way that bugs became a source of collaboration was when students noticed another student's bug and inserted
themselves into the debugging process without being invited. For instance, this happened with Andrea, a student
who had earlier been identified by her teacher as a 'middle of the class' student who struggled with motivation in
the class. Unusually for her (teacher post-interview, May 24, 2017), Andrea was one of the first to finish her final
project (the human sensor project) and noticed that a friend sitting at the same table, Maria, was struggling with
her project. The collaboration proceeded as follows:

First, Andrea noticed that Maria was having trouble with her project. Andrea shifted her seat to be next to Maria and began to interact with Maria's project. When uploading the code, an error message about the "port" came up, and the girls realized that the USB cable was not working. They switched out the cable and were able to upload the code, but no lighting patterns were triggered when the sensor (two aluminum foil patches) was activated by their touch. Andrea set her computer next to Maria's to check the code. Andrea went line by line through the code with Maria to see if was correct. 'I think it's the 1000 [sensor range]. Mine is set to 900 and it works better; maybe your range is too small,' Andrea said. In this, Andrea identified that the sensor threshold was too high, and the ranges within the conditional statements did not trigger the outputs Maria had programmed. Once they changed the ranges, two of the four patterns in Maria's project worked. Maria then began to debug her own code, identifying some places in her lighting patterns that were not giving the effects she intended. Andrea continued to sit next to Maria for the rest of the time, playing with a fidget spinner and answering questions whenever Maria asked for help (field note, May 18, 2017).

This spontaneous collaboration began with a bug in Maria’s project apparent without any requests for
assistance. In fact, spotting a simple USB cord problem led to a series of bugs including the more challenging
customization of sensor ranges within a set of conditional statements. Notably, both the teacher and researchers
noticed a change in Andrea during the e-textiles unit, moving from hesitation about computing and coding to
being “way more proud and way more into the work” (teacher post-interview, May 24, 2017). This change was
visible in Andrea's confident approach to her friend and tablemate Maria, and in the ways that she both helped identify and fix problems and stood by while Maria continued to debug on her own.

## Soliciting help on peer-to-peer debugging

Another type of emergent peer collaboration involved students explicitly soliciting another's help to debug their projects. Many a times students requested help from peers they considered more expert in a specific area, from threading a needle to checking a circuit diagram to fixing code. This recognition of expertise may have come from a teacher or researcher suggesting getting help from a particular student or, more often, from prior knowledge of a student's expertise from earlier in the class, as it did with Jesse below.

During the fourth project, Jesse had successfully sewn and tested his lights and had sewn his sensors patches on but was “terrified” to start the coding of his sensors, which involved reading the sensors and deciding on a set of ranges that would work with different levels of squeezing the two patches (fieldnote, Mar. 07, 2017). To start on the coding, he got up from his seat and went to the other side of the classroom to seek help from Gencio, a student who had already worked through much of his sensor code (with the help of a researcher) and who was known as a very proficient coder in the class. After getting assistance from Gencio, Jesse came back to his table and almost immediately switched roles from being helped to helping, looking at his neighbor Joyce's code and pointing out an incorrectly (or inconsistently) named variable. Joyce quickly corrected her error, but not before the interaction drew the attention of Felix, sitting at the same table, who teased Joyce for not listening carefully.

The whole sequence of events demonstrates an example of cascading help that was provided from one person to another. First, the researcher helped Gencio and his partner with coding and debugging the sensors on their projects. Later, Jesse approached Gencio for help. Then Jesse pointed out an error to Joyce, and the whole table became involved in Felix's teasing about Joyce's coding bug. Conversations such as these where students developed a sense of camaraderie by heckling, joking and even encouraging each other were observed frequently as demonstrated in the Dante and Juan's story (see next section).

## Peer camaraderie over failure

One other type of collaboration that occurred around debugging included many friendly conversations where peers sympathized about bugs. Often prompted by a student's encounter with some mistake or issue, these conversations became places where students not only shared their individual problems, but also debated their causes and considered possible solutions. Along with promoting a sense of camaraderie and/or sympathy, these moments provided much needed emotional support for one to persevere through moments of frustration and sense of failure. For example, below is a fieldnote record (Jan. 26, 2017) illustrating Dante's experience when debugging his dysfunctional wristband (project #2) led to a spontaneous conversation, inviting his nearby classmate's attention. During this episode, Dante shifted from discouragement to sharing laughter with peers about common mistakes, highlighting the way camaraderie influenced his participation.

Researcher: How's it going?

Dante: I can't sew. Yeah, I keep messing up like 30 times. (Other students giggle.)

Researcher: What did you mess up on? (Others are still snickering.) Everybody else knows what you're messing up on?

Dante: It's 'cause I didn't pull the string all the way, so I had loops in the back...

Juan: Yeah, but his first mistake was getting that color, though.

Dante: Oh yeah. Yeah, my first mistake was getting this [gray] color felt, so I barely see the [gray conductive] string.

Researcher: Ohhh, it's hard to see!... So that was the first problem, the color?

Dante: Yeah, because I didn't know what I was doing so I was just pulling the strings, and before I knew it, I was pulling apart... (Points to frayed areas of felt on his project).

Researcher: Oh, 'cuz you can't tell what's the string and what's the fabric.... (Juan laughs) And Juan, why're you laughin' at him?

Juan: 'Cuz I made the same mistake!

(Juan, sitting across the table from Dante, starts to describe how messy the back of his wristband was, after he thought he did a good job stitching and pulling the thread all the way through. He talks on for a minute.)
Juan: Yeah, and when you look at the back... (Several others laugh, including Dante).

Dante's visible errors—large knotty clumps of thread with fabric the same grey color of the thread—invited attention and laughter from his classmates, providing an opportunity for Juan to share his own similar mistakes. While the laughter of Dante's classmates may initially appear off-putting, the tone of the conversation implies that the laughter was more in sympathy than in jest, effectively lessening the significance of a single mistake. In fact, Dante joined in on the laughter at the end, sharing a sense of camaraderie over common problems. Although Dante never successfully completed his wristband, this did not prevent Dante from finishing his next two (more difficult) projects. In the end, even creating a fully-functional final project which involved a spatially complex, three-dimensional interactive ball created attaching four pieces of felt.

## Designs and supports for emergent collaboration

Analyzing the occurrences of emergent collaboration around bugs (including those above as well as our entire dataset), here we consider the features of the learning environment that supported this phenomenon. Overseeing, overhearing, and close proximity to peers were important features that related to physical aspects of both the projects and the spaces in classrooms. For instance, bugs stood out visibly in the physical *tangibility* of e-textiles. Mistakes such as knotty threads or malfunctioning lights were visible to others within eyesight, effectively serving as “open tools” (Hutchins, 1995) that triggered collaboration and, as seen above with Dante and Juan, commiseration over problems. Further, all three classrooms had *open physical spaces* where students were grouped at tables and could see each other’s work. A number of collaborations happened between students sitting next to each other or across the table from each other (e.g., Andrea and Maria). Muttered frustrations or exclamations also led to conversations and collaborations about problems, even when problems were not physically visible.

More subtle than visible and audible awareness of bugs, the *curricular structures* of the e-textile unit further enabled emergent collaborations. The design constraints for the projects meant that each student had to learn similar concepts (i.e., coding conditionals for human touch sensor patches in project #4) while allowing for personal customization (Kafai & Fields, 2018). This combination of constraints and personalization meant that students could legitimately learn from each other and even copy techniques of code, circuitry, or craft while still making something unique. For instance, looking at a peer's code served as a stepping stone for students like Maria or Jesse to debug their projects. Further, the curriculum suggested regular formal collaborations in certain lessons, including pair programming, a collaborative project (project #3), and whole class discussions. This provided precedence and practice for students to work together, and many students either returned to earlier partnerships or used knowledge about peers' expertise gained from class discussions to seek help.

The social structures promoted in the curriculum combined with prior *social networks* in the classes to facilitate emergent collaborations. Adding to the formal collaborative pairings and discussions, students used their own prior networks of friendship in providing help, seeking assistance, and sympathizing over problems. Many emergent collaborations took place between students who had been long-time friends, as was the case of Andrea and Maria. In addition, some students like Dante stated that they developed new friendships during the unit more than at any other time of the school year, expanding social networks.

Finally, *teachers' practices* and *classroom management* played both indirect and direct roles in emergent collaborations around bugs. For instance, unspoken classroom rules allowed for the *mobility* of students and for *informal conversation* about problems. Though students were expected to be on task, this did not mean that they had to sit silently at their own seat as they might be expected to in other school contexts. In addition, teachers often intervened more directly to support peer-to-peer learning by brokering help between students. Further, as we found in our study of teacher practices in the unit (Fields et al., 2018), teachers frequently legitimized students' diverse and developing expertise, highlighting students' knowledge from their earlier mistakes in classroom reflections and sharing of "tips and tricks". This supported the idea of turning to students, not just teachers, as sources of knowledge. Open physical spaces, tangible artifacts, project constraints, formal collaborations, informal social networks, combined with classroom practices to facilitate emergent peer collaborations around bugs.

## Discussion

In this paper we described unplanned peer-to-peer collaborations that emerged around problems, or bugs, in students' open-ended e-textile designs. Though others have pointed out the collaborations that happen around idea-generation and technique diffusion in makerspaces (Halverson et al., 2018), in our study problem-naming, fixing and helping were a nexus for emergent peer collaboration. Student-to-student collaborations around bugs
resulted in just-in-time assistance with troubleshooting, encouragement, and even humor that helped students' progress and persevere through problems. In doing so students took up roles of leadership and expertise while developing friendships in the class and inserting humor into an often-frustrating design process. Unlike the engineered moments of "productive failure" scenarios (e.g., Kapur, 2008), there is an ample supply of bugs in e-textile designs that arise without structured planning. Together these findings not only refine our understanding of student collaborative experiences in the face of failures but also highlight some of the key contextual factors that turn these moments into productive ones.

In discussing various environmental features that support emergent collaboration, we considered the roles of physical, curricular, social, and teacher attributes. Tangible artifacts and open spaces combined with social norms that allowed movement, conversation, and improvisation afforded opportunities for students to see, hear, and converse with each other over challenges. In settings where making or constructing are the focus and where the teacher is not the only expert in the room, it is important to highlight the more tacit qualities of learning environments and classroom practices that facilitate learning. We do not assume to have captured all environmental aspects that support emergent peer collaboration. Rather we hope that this study provides a starting point for exploring bug-focused collaborations and contextual factors that support emergent collaborations. Much more awaits further study, such as potential relationships between types of problems and the collaborations that emerge around them, the role of different tools and representations in the problem space in inviting collaboration (or not), and further environmental supports (structures, spaces, norms, practices) that facilitate or constrain peer collaboration. More work is also needed on the social-emotional qualities of emergent collaborations, for instance the "camaraderie" identified in this study, which played an important role in making "failure" and bugs more socially acceptable to students like Dante and Juan, facilitating not just problem-solving but emotional robustness and overall perseverance during the course of the unit.

Much work in computer supported collaborative learning has focused on designing direct scaffolds for participation, whether in the form of group roles, group structures, on-screen or in-class scaffolds, or tasks that facilitate collaboration. One of the challenges with such fixed arrangements is that they provide little agency for students to develop their own competencies in dealing with frequent unforeseen challenges that occur while making computational artifacts outside the structured settings. Emergent peer collaboration provides a means to acknowledge and support student agency, expertise, and creativity. Designing environments for emergent collaboration, including in relation to more formal collaborations, may open up even greater opportunities for student learning and development.

## References

* Barron, B. (2003). When Smart Groups Fail. *Journal of the Learning Sciences*, 12(3), 307-359.

* Buechley, L., Peppler, K., Eisenberg, M., & Yasmin, K. (2013). *Textile Messages: Dispatches from the World of E-Textiles and Education*. *New Literacies and Digital Epistemologies. Volume 62*. Peter Lang Publishing Group. 29 Broadway 18th Floor, New York, NY 10006.

* Charmaz, K. (2014). *Constructing grounded theory*. Sage publications, Thousand Oaks, CA.

* Ching, C. C., & Kafai, Y. B. (2008). Peer pedagogy: Student collaboration and reflection in a learning-through-design project. *Teachers College Record*, 110(12), 2601-2632.

* College Board (2016). *AP Computer Science Principles: Course and Exam Description Effective Fall 2016*. College Board: New York, NY. Retrieved from: https://secure-media.collegeboard.org/digitalServices/pdf/ap/ap-description.pdf.

* DeLiema, D. (2017). Co-constructed failure narratives in mathematics tutoring. *Instructional Science*, 45(6), 709-735.

* Fields, D. A., Kafai, Y. B., Nakajima, T. M., Goode, J., & Margolis, J. (2018). Putting making into high school computer science classrooms: Promoting equity in teaching and learning with electronic textiles in *Exploring Computer Science. Equity, Excellence, and Education*, 51(1), 21-35.

* Fields, D. A., Searle, K. A., & Kafai, Y. B (2016). Deconstruction kits for learning: Students' collaborative debugging of electronic textile designs. In *FabLearn '16, Proceedings of the 6th Annual Conference on Creativity and Fabrication in Education*, ACM, New York, NY, 82-85.

* Goode, J., Chapman, G., & Margolis, J. (2012). Beyond Curriculum: The Exploring Computer Science program. *ACM Inroads*, 3(2), 47-53.

* Halverson, E., Litts, B., & Gravel, B. (2018). Forms of Emergent Collaboration in Maker-based learning. In *Proceedings of the International Conference of the Learning Sciences, 2018*, 921-924.

* Hutchins, E. (1995). *Cognition in the wild*. Cambridge, MA: MIT.
Jayathirtha, G., Fields, D., & Kafai, Y. (2018). Computational Concepts, Practices and Collaboration in High School Students' Debugging Electronic Textiles projects. *Proceedings of the International Conference on Computational Thinking Education 2018*. Hong Kong: The Education University of Hong Kong. 27-32.

Kafai, Y. & Fields, D. (2018). Some Reflections on Designing Constructionist Activities for Classrooms. In *Proceedings of the Constructionism Conference 2018*. Vilnius, Lithuania. 601-607.

Kafai, Y., Fields, D., & Searle, K. (2014). Electronic textiles as disruptive designs: Supporting and challenging maker activities in schools. *Harvard Educational Review*, 84(4), 532-556.

Kapur, M. (2008). Productive Failure. *Cognition and Instruction*, 26(3), 379-424.

Kapur, M., & Kinzer, C. K. (2009). Productive failure in CSCL groups. *International Journal of Computer-Supported Collaborative Learning*, 4(1), 21-46.

Ko, A. J., & Myers, B. A. (2004, April). Designing the whyline: a debugging interface for asking questions about program behavior. In *Proceedings of the SIGCHI conference on Human factors in computing systems*, 151-158. ACM.

Litts, B. K., Lui, D. A., Widman, S. A., Walker, J. T., & Kafai, Y. B. (2017). Reflections on Pair E-Crafting: High School Students' Approaches to Collaboration in Electronic Textiles Projects. In *Proceedings of the International Conference of Computer Supported Collaborative Learning 2017*, Philadelphia, PA (2) 569-572.

Miles, M. B., Huberman, A. M., & Saldana, J. (2014). *Qualitative data analysis: A methods sourcebook*. Sage publications, Thousand Oaks, CA.

McCauley, R., Fitzgerald, S., Lewandowski, G., Murphy, L., Simon, B., Thomas, L., & Zander, C. (2008). Debugging: a review of the literature from an educational perspective. *Computer Science Education, 18*(2), 67-92.

Papert, S. (1980). *Mindstorms: Children, computers, and powerful ideas*. Basic Books, Inc.

Patil, A., & Codner, G. (2007). Accreditation of engineering education: review, observations and proposal for global accreditation. *European journal of engineering education*, 32(6), 639-651.

Peppler, K., Halverson, E., & Kafai, Y. B. (Eds.). (2016). *Makeology: Makerspaces as learning environments* (Vol. 1). Routledge.

Sawyer, K. (2012). Extending sociocultural theory to group creativity. *Vocations and Learning*, 5(1), 59-75.

Sheridan, K., Halverson, E. R., Litts, B., Brahms, L., Jacobs-Priebe, L., & Owens, T. (2014). Learning in the making: A comparative case study of three makerspaces. *Harvard Educational Review*, 84(4), 505-531.

Sorva, J., Lönnberg, J., & Malmi, L. (2013). Students' ways of experiencing visual program simulation. *Computer Science Education*, 23(3), 207-238.

Vessey, I. (1985, December). Expertise in Debugging Computer Programs: Situation-Based versus Model-Based Problem Solving. In *International Conference of Information Systems* 1985, 18, 288-302.

Williams, L. A., & Kessler, R. R. (2000). All I really need to know about pair programming I learned in kindergarten. *Communications of the ACM*, 43(5), 108-114.

## Acknowledgments

This work was supported by a grant from the National Science Foundation to Yasmin Kafai and Michael Eisenberg (#1742140). Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of NSF, the University of Pennsylvania, or Utah State University. Special thanks to Tomoko Nakajima for her help with data collection and to Alecia Magnifico, Debora Lui, Justice T. Walker, and Mia Shaw for their valuable feedback.
# Supporting Collaborative Curriculum Customizations Using the Knowledge Integration Framework

Korah J. Wiley, Allison Bradford, and Marcia C. Linn

korah.wiley@berkeley.edu, allison_bradford@berkeley.edu, mclinn@berkeley.edu

University of California, Berkeley

**Abstract:** Teachers are currently facing a major instructional challenge, namely, supporting students to meet the three dimensional learning goals of the Next Generation Science Standards without adequate curriculum materials to do so. In this paper, we report the design and outcomes of professional development activities that support teachers and researchers to collaboratively customize Web-based Inquiry Science Environment (WISE) units to help students develop coherent science knowledge. The WISE units and professional development activities were developed using the Knowledge Integration (KI) Framework. We show that the KI framework functioned as an effective scaffold to support teachers in modifying their teaching practice to make curriculum customizations that are evidence-based and aligned with a theory of learning. We discuss how our study results informed the design of an online curriculum customization and implementation interface and offer the principles of the KI framework as design principles for the development of other collaborative professional development endeavors.

**Keywords:** curriculum customization, professional development workshop, teacher-researcher collaborative learning, NGSS, knowledge integration

## Introduction

We report on the design and impact of a professional development workshop that positions teachers and researchers as collaborative partners in customizing Web-based Inquiry Science Environment (WISE) units to align with the Next Generation Science Standards (NGSS, 2013). To the workshop, teachers brought their teaching expertise and used their experience implementing the units and logged student data to improve the unit. Researchers brought their expertise in designing the units following the Knowledge Integration (KI) pedagogical framework and used the framework to guide the professional development activities. We hypothesized that, if aligned with curriculum activities, the four steps of the KI process (i.e. eliciting ideas, adding ideas, distinguishing ideas, and reflecting on ideas) would serve as instructional strategies and scaffold teachers' application of the underlying constructivist theory of learning. We further hypothesized that, if aligned with the workshop activities, the four KI steps would give teachers and researchers an experiential knowledge of the process, facilitate the integration of their respective knowledge, and foster the collaborative customization of WISE units for NGSS-alignment and variegated classroom implementation. We discuss how the results from this study informed the design of an online interface on the WISE platform to support teachers to customize and implement NGSS-aligned curricula in ways that helps students develop coherent, three-dimensional science knowledge.

## Background

Since its publication in 2013, the NGSS (NGSS, 2013) has been widely adopted by U.S. states, often without accompanying curriculum materials. During initial implementation of WISE units, many teachers reported interest in better aligning their instruction with NGSS, motivating this study. Given the new, three-dimensional learning goals of the NGSS, developing curricula de novo that can support students to meet these ambitious standards is a formidable task, even for the most adept teacher. In their instructional comparison study, Penuel and Gallagher (2009) showed that when teachers assume the role of curriculum customizer rather than designer, they produce higher quality curriculum, as measured by its ability to support inquiry and promote student science understanding. Building on these findings, we designed a knowledge integration workshop to build teachers' capacity to customize existing curricula rather than design new curricula.

A review of professional development supporting teachers to use technology-enhanced science inquiry curricula found a significant effect for programs that followed the KI framework (Gerard et al, 2011). Successful professional development efforts elicited teachers' ideas, added new ideas to their existing repertoire, and encouraged teachers to distinguish among their ideas, reflect upon their experiences, and develop an
integrated, coherent view of instruction (Linn & Eylon, 2011). The KI framework draws on extensive research supporting a socio-constructivist view of learning.

A key tenet of socio-constructivist learning and knowledge integration is sustained collegial collaboration. In successful professional development efforts, this collaboration involves teachers from multiple schools along with university mentors (Penuel et al., 2007). As teachers collaborate with other education community members in activities such as customization, they negotiate meaning (Lave, 1996), which is a critical aspect of learning.

## Research questions

Research suggests the value of focusing on the customization of existing curriculum materials. However, supporting teachers to incorporate the NGSS into their teaching practices calls for new models of professional development. The KI framework as a professional development model offers promise, especially for the customization of WISE units, since they were designed using the KI framework. Thus, we investigated the following research questions:

1. Can the KI framework guide the design of both curriculum customization and professional development?

2. Which professional development activities:

* Enable teachers to use pedagogical principles to align existing curriculum materials with NGSS?

* Support teachers and researchers to collaborate to develop curriculum that enables students to meet the three-dimensional learning goals of the NGSS?

## Methods and materials

We designed and tested professional development activities consisting of a 1.5 day workshop to review and customize WISE units, in-class support during implementation of the customized unit, and post-implementation interviews. The professional development activities followed the KI pedagogy of eliciting ideas about teaching the unit, adding new ideas to customize the unit, distinguishing among ideas during implementation of the customized unit, and reflecting on the experience.

## Participants

The workshop participants included 21 middle school science teachers and 15 researchers. The teachers came from 8 schools across 6 districts and the researchers came from 2 universities. All teachers had implemented at least one WISE curriculum unit prior to attending the workshop.

## WISE curriculum units

WISE units are developed using the KI Framework. They elicit student ideas from their own experiences; engage students in gathering new ideas using embedded models and simulations and hands-on activities; encourage students to distinguish among these ideas by building models, testing alternative views, or critiquing ideas of others; and request reflections in reports or presentations. The WISE units used by teachers in this study are: Photosynthesis, Global Climate Change, Plate Tectonics, and Self-Propelled Vehicles. Each unit features embedded assessments, logs student responses, and captures student activities (e.g. click-stream data). Teachers and researchers can access all logged, de-identified student data via the data export interface in the Grading Tool.

## Customization materials and activities

Before the workshop, in preparation for the customization process, we analyzed the WISE units to identify the NGSS performance expectations addressed in each. Then, we restructured the units into lesson series such that each lesson in the unit targeted a single NGSS performance expectation. To support coherent knowledge building for the target performance expectation, we also structured each lesson in the unit to engage students in each step of the KI process (i.e. one lesson corresponds to engagement in a complete KI cycle). The lessons were designed such that they could be taught in sequence as a unit or as independent lessons.

In addition, we created a diagrammed version of each unit consisting of a 5" x 7" notecard for each lesson with each activity in the lesson briefly described on 3" x 3" sticky notes, color-coded by each KI step (pink: Elicit Ideas; orange: Add Ideas; green: Distinguish Ideas; blue: Reflect On/Revise Ideas, see Figure 2C). On the 5" x 7" notecards was the following lesson information: unit title, lesson title, recommended grade
levels, targeted performance expectation, and a brief description of the learning goals. These tangible, diagrammed versions of the WISE units were the objects of customization and were designed to pilot a prototype version of a unit customization interface on the WISE platform.

After the workshop, the researchers incorporated the customizations of the diagrammed version of each unit into a complete digital version for subsequent post-workshop classroom implementation.

## Data sources and data analysis

To address our research questions, we gathered data corresponding to each of the KI-aligned professional development activities. During the 1.5 day workshop, we documented the customized units for comparison to their previously implemented counterparts. Throughout the workshop, the researchers captured audio recordings and photographs of the workshop activities, and collected teachers' written responses to the following prompts:

* What are some things you learned or have taken away from engaging in this customization process and sharing with other teachers?

* Was this customization process and reflecting on the KI cycle helpful for you in thinking about how to achieve your NGSS and other curricular goals?

* Do you think you could use this customization process for another unit you'd like to run?

* Please share any other reflections or feedback you have from the workshop.

To analyze the customization of the WISE units, we counted the number and type of interleaved KI-aligned, non-WISE activities that teachers added to the diagrammed units. Our analysis of teachers' written reflection consisted of identifying themes related to the customization and collaboration process. For this paper, we analyze the implementation of the customized WISE Photosynthesis unit as carried out by two, 7th grade teachers, Mr. Vega and Mr. Harrison. We took field notes during classroom observations and conducted post-implementation interviews to discuss their customization decisions and their overall implementation experiences. We evaluated our observation notes and interview data in terms of their implementation of the workshop customizations and the type of customizations they made during implementation.

## Results

### Knowledge-integrating and experiential activities

The activities of the 1.5 day workshop aligned with the four steps of the KI process (see Table 1) and positioned teachers as experts on their teaching practice and researchers as experts on their curriculum designs. To begin the workshop, the researchers facilitated a whole group discussion to elicit ideas about the function of each dimension of the NGSS performance expectations in terms of lesson development (e.g. disciplinary core ideas provide the lesson content). During this discussion, the researchers introduced the KI framework and invited teachers to share activities they used or could KI process. This activity was designed to highlight the similarities between how the teachers and researchers conceptualize and support the knowledge building process. Each group activity of the workshop (whole group and small group) was designed to highlight the expertise of both teachers and researchers and to create opportunities for expertise sharing.

<table><caption>Table 1: A summary of the professional development activities, their alignment with the KI process, and sample teacher reflections.</caption><thead><tr><td>Professional Development Activity</td><td>Knowledge Integration Alignment</td><td>Sample Teacher Reflections</td></tr></thead><tbody><tr><td>KI Framework for NGSS Lesson Development</td><td>Eliciting & Adding Ideas</td><td>"I enjoyed looking at the KI. It is very similar to how we plan our lessons and it was cool to see we were aligned in our thinking."</td></tr><tr><td>Identifying KI-supporting Activities</td><td>Eliciting & Adding Ideas</td><td>"[KI] allowed [me] to better think about the learning process. Breaking it down into components and deciding what kind of activities and scaffolding will go into that part of the project"</td></tr><tr><td>WISE Unit Analysis Using</td><td>Adding & Distinguishing</td><td>"This is something I will use in the future with other lessons"</td></tr></tbody></table>
<table><thead><tr><td>Student Data</td><td>Ideas</td><td>to make sure it is really meeting all areas[, if] students [are] really understanding a lesson.”</td></tr></thead><tbody><tr><td>Interleaving KI-supporting Activities into WISE Units (Customizing)</td><td>Adding & Distinguishing Ideas</td><td>“The customization process and reflecting on the KI cycle are definitely helpful for me. I understand better how each unit is built and how I can interact with it. I can pull different components together to fit the needs of the curriculum or the time constraints”</td></tr><tr><td>Reviewing Unit Customizations</td><td>Distinguishing & Reflecting On Ideas</td><td>“It was really interesting to hear other teachers’ reflections on their units. It gave me the chance to think about how I might want to use some similar ideas in the units I plan to run in my classroom.”</td></tr><tr><td>Actualizing Customizations Using the WISE Platform</td><td>Adding Ideas</td><td>“I added new labs to the WISE project, learning how to use the authoring tool was great to be able to add our new ideas into the project.”</td></tr><tr><td>Implementation of Customized Unit</td><td>Distinguishing & Reflecting On Ideas</td><td>“I deployed instruction in similar fashion to the Knowledge Integration process through a unit designed and planned by [another curriculum provided] for 6th graders learning from instruction that is 3-D and that conforms to the NGSS. I generally follow the same process throughout each unit of instruction.”</td></tr><tr><td>Post-Implementation Interviews</td><td>Reflecting On Ideas</td><td>“The [KI] labels on each step [of the customized WISE unit] were useful to me in identifying the stage of learning the students were in; to which, I targeted my instruction or interactions with small groups. I loved how the unit was divided into distinct but incremental lesson[s] for that exact reason, they demarcated conceptual bubbles that can exist on their own but are still a part of something larger, the bigger picture, the overarching idea.”</td></tr></tbody></table>

The sharing and integration of expertise was evident in the customization activity. For the unit customization activity, teachers were assigned to a small group corresponding to a unit they recently implemented and with researchers who had developed the units. Teachers were asked to think of and write on colored 3 x 3 sticky notes as many activities as they could think of for each step of the KI process (Pink: elicit ideas; Orange: add ideas; Green: distinguish ideas; Blue: reflect on/revise ideas). Teachers were invited to share their KI activity ideas with other workshop participants by placing the sticky notes on a long table centrally located in the meeting room (see Figure 1). The goal of this activity was to further elicit and add ideas regarding topic-specific activities that were accessible to teachers and would support the KI process.

<figure><img src="image_186.png" /><figcaption>Figure 1. Shared interleaved teacher activities.</figcaption></figure>

To further add ideas regarding unit customization, the researchers provided teachers with evidence of their students' learning. Specifically, researchers gave the teachers a random sampling of 30 of their students' responses to a post test assessment item that targeted at least one dimension of an NGSS performance expectation addressed in the unit. Teachers were asked to evaluate the responses to identify areas of strength and weakness in their students' understanding.

The researchers then invited the teachers to customize the unit by interleaving the activities they previously wrote on the color-coded sticky notes with those on the lesson notecards. The goal of this activity
was to customize the unit in ways that would better support students in developing an integrated understanding of the targeted NGSS performance expectation, using evidence of previous student learning as a point of reference. Teachers were given full license to eliminate, substitute, or add any KI-supporting activity by rearranging the sticky notes on the lesson notecards. In this way, teachers could distinguish their ideas about how to customize the unit in ways that would support their students in the KI process and function within their classroom constraints and resources (see Figure 2A, 2B).

<figure><img src="image_187.png" /><figcaption>**Figure 2.** WISE unit customization process: Identifying non-WISE, KI-aligned activities (a); Identifying relevant activities to customize the unit (b); Integrating their activities into the diagrammed WISE unit (c).</figcaption></figure>

Teachers worked collaboratively within and across their small groups to exchange ideas and activities to customize their unit. Of all the customizations that teachers made across all the units, 33% (31/94) would elicit students’ ideas, 33% (31/94) would add to students’ ideas, 10% (9/94) would support students in distinguishing their ideas, and 24% (23/94) would help students reflect on or revise their ideas (see Figure 3).

<figure><img src="image_188.png" /><figcaption>**Figure 3.** Percentage of unit customizations aligned to each Knowledge Integration step.</figcaption></figure>

In another group workshop activity, teachers were asked to evaluate whether their customized unit would support their students in developing an integrated understanding of the disciplinary core ideas, cross cutting concepts, and science or engineering practices targeted in the unit lessons. After teachers made their final customizations, they were invited to share their customized units with other teachers who were interested in implementing the unit. During this unit exchange activity, teachers were able to get feedback from other workshop participants about their customization decisions.

In the final group workshop activity, the researchers demonstrated and helped teachers use the unit authoring tools currently built-in to the WISE platform. During these small workgroup sessions, teachers chose 1-2 customizations to reify in the digital version of the unit on the WISE platform.

To conclude the workshop, teachers were asked to provide their reflections on their workshop experiences (see Table 1). Of the 21 teacher participants, 15 (71%) provided responses to the reflection questions. As for whether the KI framework was a productive lens through which to evaluate and customize a unit, 80% of teachers (12/15) answered in the affirmative. In response to the question, “Do you think you could use this process again to customize another unit you’d like to run?”, 87% of teachers (13/15) answered in the affirmative. Additionally, many teachers (87%, 13/15) shared that the lesson series format of the WISE units provide great affordances for customization using their own activities.
# Reflection on and implementation of a customized unit: The cases of Mr. Vega and Mr. Harrison

To continue exchanging expertise after the workshop and to extend the collaborative customization efforts, the first author provided in-class support during implementation of the customized unit. The following are the results of the implementation of the customized Photosynthesis WISE unit (https://wise.berkeley.edu/project/24548) in the fall term after the summer workshop by two teachers, Mr Vega and Mr. Harrison. Mr. Harrison worked in the small group that customized the Photosynthesis unit during the summer workshop, however Mr. Vega worked in the small group that customized the Global Climate Change unit. Over the course of implementation of the customized Photosynthesis unit, the first author noticed how both teachers interleaved non-WISE activities, most of which were designed to provide students with additional ideas to supplement their limited prior knowledge about certain topics, like chemical reactions. This observation supports the analysis of the unit customizations made during the workshop where 33% of the total customizations aligned with the Adding Ideas step of the KI process (see Figure 3). In both cases, the teachers used a molecular modeling kit to provide their students with ideas about atoms, molecules, and the nature of chemical reactions related to photosynthesis. Additionally, Mr. Harrison incorporated a multi-modal activity on the conservation of matter during the photosynthesis reaction as a transition from Lesson 1 (Plant Growth Needs) to Lesson 2 (Photosynthesis and Cellular Respiration Reactions). During the post-implementation interview he commented that he used the WISE concept mapping activities to support his students in distinguishing their ideas about energy and matter cycling, the targeted cross-cutting concept in the unit. Mr. Harrison’s use of WISE activities over his own activities to help students distinguish their ideas correlates with the result from the customization activity, namely the low levels (10%) of customization activities aligning with the distinguishing step (see Figure 3). This finding also parallels Mr. Vega’s reflection comment that he plans to “[s]trengthen the role WISE plays during the Distinguishing…phase”.

## Mr. Harrison: A deeper look

During the workshop, Mr. Harrison expressed reticence regarding the customization partnership. His past experiences with the units led him to feel like they were “adapted to fit the need of each researcher”, additionally he could not see how the digital version of the WISE unit could be restructured into related but independent lessons aligned to specific NGSS performance expectations. However, when asked how he thought the customized, digital Photosynthesis unit aligned with the NGSS he remarked, “It's definitely aligned, it fits right in with our [other curriculum materials] because we go from Photosynthesis right into ecosystems” (which is Lesson 4 of the customized Photosynthesis unit). This comment highlights the way that Mr. Harrison interleaved the WISE Photosynthesis unit with his existing curriculum to better support his students in meeting the targeted NGSS performance expectations. Specifically, Mr. Harrison commented that his existing curriculum, although nominally aligned with the NGSS, did not actually provide his students with sufficient opportunities to develop a robust and coherent understanding of the targeted ideas, concepts, and practices. He, therefore, used the customized WISE unit to provide his students with these opportunities. Mr. Harrison further discussed the difficulty and discomfort he experienced when implementing curriculum that he did not develop and therefore greatly appreciated the ability to customize the WISE unit with his own activities to make the unit “his own”. When the first author asked Mr. Harrison how the researchers could further collaborate to address his curriculum customization challenges, he offered the idea of incorporating specific WISE lessons and activities into the Google Classroom that he used to plan and organize his instruction. He stated that doing so would solve the notoriously difficult problem of trying to re-synchronize students who progress at different speeds through the WISE activities. (We elaborate upon this idea in the Conclusion section.)

## Mr. Vega: A deeper look

Mr. Vega also engaged in extensive customization during implementation to assume greater “authorship” of the unit. During the implementation, he commented to the first author, “I was motivated to Edit Content to the unit...Take a look”, referring to his use of the unit authoring tools on the WISE platform, a feature historically used only by researchers. In these edits, Mr. Vega customized the prompts for the concept maps to align with the topics and terminology used in his non-WISE activities. In this way, he created greater continuity between WISE and his other curriculum materials. Mr. Vega’s customization of the Photosynthesis unit during implementation is of particular note since during the workshop customization activity he was not part of the Photosynthesis small group. Therefore, Mr. Vega’s customization of the Photosynthesis unit demonstrates that he was able to transfer and apply the knowledge he gained during the workshop to another unit, actions that substantiated his “YES” to the reflection question of whether he could customize another unit. Beyond making edits to provide greater continuity between the WISE unit and his existing curriculum materials, Mr. Vega also
applied his understanding of the KI process to all his instruction. During the post-implementation interview he stated that he used the steps of the KI process to keep track of the stage of learning in which his students were engaged and thus provide them with targeted support.

# Discussion

## Design principles for Knowledge Integration customization

In this paper, we described professional development activities that were aligned with the KI framework and showed how the framework helped teachers customize their existing curriculum to better support their students in meeting the three dimensional learning goals of the NGSS. The workshop reveals three design principles that we recommend for teacher-researcher professional development: support teachers and researchers to learn from each other; make thinking about customization visible; and support sustainable customization practices. In the sections below, we highlight the value of each design principle in supporting collaborative learning amongst researchers and teachers. We conclude the section with a discussion of how the insights gained during the customization activity inform the design of an online curriculum customization interface.

## Supporting teachers and researchers to learn from each other

The workshop facilitated the integration of teachers' and researchers' knowledge and expertise. Throughout the workshop activities, researchers used the KI framework as a mediating tool to share their insights into the learning process, and teachers shared their insights into the instructional constraints and resources of their teaching contexts. Transitioning from whole to small group activities allowed workshop participants to learn from each other and consider new strategies and activities to help students meet the learning goals of the NGSS. Teachers expressed that having opportunities throughout the workshop to learn from researchers and other teachers within and outside their school was invaluable, as they do not regularly have such opportunities. The final customized WISE units reflected the integration of the ideas and expertise of researchers and teachers. Thus, the workshop embedded research-based pedagogical insights into teachers' curriculum customization practices and expanded researchers' understanding of teachers' instructional constraints and resources.

## Making thinking about customization visible

To customize the WISE units, we designed workshop activities that made the customization process visible. The diagrammed WISE units made the researchers' thinking visible to teachers by highlighting the units' salient features and the researchers' design rationale. The diagrammed units facilitated productive conversation about the feasibility of unit implementation and theories of learning, specifically the KI framework. Writing their ideas for lesson activities on sticky notes color-coded according to the steps of the KI process provided teachers tangible artifacts with which to organize their current curriculum materials in ways that would support constructivist-grounded pedagogy. Having the content of the sticky notes at the grain-size of lesson activity supported experimentation with lesson structure and sequence. Teachers could place and replace the notes on the lesson notecards. Thus, the customization activity helped participants make their thinking visible by moving their ideas from conceptualization to paper, making their ideas available for evaluation and revision by themselves and others. The activities also helped make learning the KI framework accessible to teachers, thereby providing teachers and researchers with a common pedagogical lens with which to evaluate and customize curriculum.

## Supporting sustainable customization practices

After the workshop, teachers and researchers partnered to implement the customized WISE units. The two cases presented in this paper, highlight the power of the professional development activities to promote sustainable curriculum customization practices. In one case, the teacher (Mr. Vega) implemented a customized Photosynthesis WISE unit which he did not work on during the workshop. During implementation he applied the KI framework to effectively interleave his existing curriculum activities into the digital WISE Photosynthesis unit to support his primarily English Language Learner students in developing a coherent understanding of the targeted performance expectations. Similarly, the other teacher (Mr. Harrison) customized the WISE Photosynthesis unit to supplement his existing curriculum. He recognized that his existing curriculum did not support students to distinguish their ideas, a critical step in the KI process. He used WISE activities to do so. Both teachers noted that the structure of the WISE units, namely a series of related yet independent lessons targeted to specific NGSS performance expectations that engage students in each step of the KI process, made customization feasible and effective. Rather than viewing the WISE units as a product of research efforts
that needed to remain unedited, both teachers viewed the units as their own and acted accordingly. The
workshop developed teacher agency around curriculum customization thereby allowing the customized WISE
unit to be the product of a partnership between education experts, experts of theory and experts of practice.

## Conclusion

The outcomes and associated principles presented in this paper illustrate the promise of the KI professional development activities to support the alignment of existing curriculum with NGSS. Analysis of the activities suggest three design principles that echo the KI tenets: learning from others, making thinking visible, making learning accessible, and promoting autonomous learning (Linn & Eylon, 2011). The principles were developed in the context of STEM curriculum customization; however, they should also be tested in other contexts, such as developing culturally-relevant curricula. The findings also illustrate the value of logging student activities and using student work to support customization. Incorporating learning analytics into the WISE units could potentially allow teachers to use evidence of student learning in real-time to make curriculum customization that support the KI process. This is an application of these findings that we are actively investigating.

## Make customization accessible

The diagrammed version of the WISE units and the workshop activities provided a prototype for an online
customization and implementation interface. This interface makes the customization process accessible to
teachers. Teachers commented that the diagrammed WISE unit provided the ideal amount and type of
information they needed to gain a working knowledge of a unit. They also valued the intuitiveness of
customizing units with activity-level sticky notes color-coded for the KI process. Using these experiences, the
researchers and WISE technology team designed a customization and implementation interface on the WISE
platform. When teachers view a WISE unit from the interface they will see a pop-out window with the same
information that was on the notecards. Upon selecting the unit for implementation, the unit will be displayed at
the activity level with each activity tagged for the KI step that it targets. Teachers can add specific activities
from other WISE units or from their own resources. In the future, we anticipate integrating with Google
Classroom so that teachers can seamlessly combine their existing curriculum with WISE lessons or activities.
Additional analysis of the final customized unit will help ensure that teachers have a customized curriculum that
engages their students in compete KI cycles. The interface will allow teachers and researchers to continually
benefit from each other's respective expertise and sustain their curriculum customization partnership.

## Next steps

Based on the successful implementation of the customized WISE Photosynthesis unit, we will study the
implementation of all the customized WISE units. These findings will allow us to refine the online
customization and implementation interface and the workshop design.

## References

Gerard, L. F., Varma, K., Corliss, S. B., & Linn, M. C. (2011). Professional Development for Technology-Enhanced Inquiry Science. *Review of Educational Research, 81*(3), 408-448.
https://doi.org/10.3102/0034654311415121

Lave, J. (1996). Teaching, as Learning, in Practice. *Mind, Culture, and Activity, 3*(3), 149–164.
https://doi.org/10.1207/s15327884mca0303_2

Linn, M. C., & Eylon, B.-S. (2011). Science learning and instruction: Taking advantage of technology to promote knowledge integration. New York: Routledge.

NGSS Lead States. 2013. Next generation science standards: For states, by states. (2013).

Penuel, W. R., Fishman, B., Yamaguchi, R., & Gallagher, L. P. (2007). What makes professional development effective? Strategies that foster curriculum implementation. *American Education Research Journal, 44*, 921–958.

Penuel, W. R., & Gallagher, L. P. (2009). Preparing teachers to design instruction for deep understanding in middle-school earth science. *Journal of the Learning Sciences, 18*, 461–508.
# The Affordable Touchy Feely Classroom: Textbooks Embedded With Manipulable Vectors and Lesson Plans Augment Imagination, Extend Teaching-Learning Practices

Durgaprasad Karnam, Harshit Agrawal, Priyanka Borar, and Sanjay Chandrasekharan
karnamdpdurga@gmail.com, harshitagrawal.iitr@gmail.com, priyankaborar@gmail.com,
sanjay@hbcse.tifr.res.in

Learning Sciences Research Group, Homi Bhabha Centre for Science Education, TIFR, Mumbai-India

**Abstract:** Computational media for mathematics and science education have been around for forty years, but teaching and learning practices are still centered surrounding the textbooks in the developing world, mainly due to their affordability. Teachers thus organise their thinking, workflow and classroom interactions using textbooks. However, text media limit students' ability to imagine the dynamics embedded in mathematics and science formalisms, such as vectors. We outline an affordable design that augments textbooks, to: allow all students to imagine such dynamics; help teachers reorganize their thinking and workflow; make all classrooms highly collaborative. The design consists of a 'touchy-feely vector' system (smartphone-based manipulation of vectors), a 'virtual lesson plan' (manipulable simulations that extend textbook figures), and a 'mixed-media' textbook (QR codes pasted next to figures, linked to manipulable simulations). We discuss how this integrated design changed learning and teaching practices, based on an ethnographic study of experimental and control classrooms.

## Introduction

Interactive media for making concepts and formal structures in mathematics and science easier to understand (Balacheff & Kaput, 1996) have been around for close to forty years (Papert, 1980, 1994). But these media have not changed classroom practices significantly, as teaching and learning processes and teacher training are still centred around textbooks, particularly in the developing world. This is because textbooks are cheap and widely available, and they thus play an artifactual role in teaching practice, helping organise teachers' thinking, workflow and classroom interaction. The textbook is thus the dominant media in education, but text and figures are limited in conveying the dynamic nature of mathematics and science content, particularly formal structures such as vectors, which are critical for developing model-based reasoning, a key skill needed for practising science (Hestenes, 2010; Magnani, Nersessian, & Thagard, 2012; Nersessian, 1999). Teachers struggle to enact or 'act out' in the classroom the dynamic nature of such formal structures, and students have a hard time imagining the dynamic nature based on the teacher's gestures, words and enaction and dropout physically or cognitively from the classroom learning.

## Our design TFV-2

To address this problem for teaching/learning vectors (Barniol & Zavala, 2014; Dorier & Sierpinska, 2002; Hillel, 2000), we developed a multi-pronged design (including cognitive, practice and access factors) to make the dynamic nature of formal structures (geometrical ones in vectors) available to all students, particularly in the developing world.

**Cognitive:** A simulation system was developed, which allows by touching and manipulating them on any smartphone (Touchy Feely Vectors-2: TFV-2: bit.ly/tfv-2) using embodied interactions, for creating (by simple double-consecutive taps), adding (drag one vector over other and double-consecutive taps), and resolving (pinching away gesture) (Fig.1). This makes the formal entities tangible, dynamics visible, and the modes coherent. An earlier version of TFV-2 was tested to be promising in helping students understand and imagine the dynamics better (Karnam, Agrawal, Mishra, & Chandrasekharan, 2016; Karnam, Agrawal, & Chandrasekharan, 2018).

**Practice:** Teacher adoption and usage of digital resources in the classroom has not been smooth (Bingimlas, 2009). In our experience too, teachers were hesitant to use it in their teaching. To address the integration with teachers' current textbook-based practice, particularly their thinking about vectors and the sequencing of the lessons, we decided to design tasks that could be planned for organised classroom activities. As an exercise, a 'virtual lesson plan' (VLP) was created, in collaboration with teachers, where this manipulable system can be used to extend their existing teaching lesson plans.
Access: These VLPs were embedded in the textbooks through QR codes (Uluyol & Agca, 2012) augmenting the figures and tasks in the textbook, as well as teacher thinking and practices based on them. This makes the introduction of new artefacts into the classrooms smooth and less abrupt as it builds on the already existing artefacts (textbooks), without replacing them.

<figure><img src="image_189.png" /><figcaption>Figure 1. Gesture for creating vector, and for deleting a vector; An active vector, changing magnitude, direction (row-1); Pinching away gesture for resolution and gesture for reversing the resolution of vector (row-2).</figcaption></figure>

We report a study involving classroom teaching using TFV-2 and the changes in teaching and learning practices, based on data from an ethnographic analysis of experimental and control classrooms.

## The classroom study

To test whether this design actually meets the design requirements, we did a classroom study (grade 11), with three experimental classrooms where the integrated system was deployed, and three control classrooms where the standard textbook teaching method was used. We worked with 5 physics teachers (T1-T5), in 3 junior colleges (C1-C3) (pre-university level high schools) in Mumbai, India. All the teachers were trained postgraduates in Physics and had about 20 years of experience teaching at this level. Each school had multiple classrooms (divisions/sections) of grade 11. We observed two grade 11 classrooms (Control group - CG, and Experimental group - EG) in each college (a total of 6 classrooms). In CG, the teachers taught using the conventional method. In EG, the teacher used tablets with the TFV-2 deployed.

<table><caption>Table 1: Colleges and classrooms, and corresponding teacher codes. NCERT - national, MH - a provincial (Maharashtra) textbook. * T3 & T5 taught in C2 and C3 due to inter-school transfers (could not be controlled)</caption><thead><tr><th>College (type, Curriculum)</th><th>Classroom</th><th>Teacher</th><th>No. of students</th></tr></thead><tbody><tr><td rowspan="2">C1 (Private, MH)</td><td>EG1</td><td>T1</td><td>73</td></tr><tr><td>CG1</td><td>T2</td><td>70-75</td></tr><tr><td rowspan="2">C2 (Public, MH)</td><td>EG2</td><td>T3</td><td>~49</td></tr><tr><td>CG2</td><td>T3 & T5*</td><td>~41</td></tr><tr><td rowspan="2">C3 (Public, NCERT)</td><td>EG3</td><td>T4</td><td>45</td></tr><tr><td>CG3</td><td>T5 & T3*</td><td>45</td></tr></tbody></table>

## Methodology and data collection

See figure-2. A brief introduction to the research study and the TFV-2 system was given to the respective college managements and the teachers to get their consent for the study. To get some initial familiarity with the system, the teachers were requested to do some exploration of a few tasks (not mandatory) on an earlier (mouse-control) version of the vector system. All the teachers except T5 attended a Lesson Planning Workshop (LPW) for ~3 hours. In the workshop, the teachers based on their already existing lesson plans, together discussed and arrived at a virtual lesson plan with tasks on TFV-2, to be used in their teaching. We video recorded all the classroom sessions (both in CG and EG for 5-8 teaching sessions of 35/40 minutes each in different colleges). In CG classrooms, the teacher was given no inputs by the researchers. In EG classrooms, as per the VLP, a 1-2 page worksheet with pre-determined tasks (developed during the LPW) was prepared, depending on the topics the teacher planned to teach in the class (one worksheet per student). About 25-30 6" Android tablets (SWIPE)
with the necessary files and QR reading applications pre-installed were taken to the classrooms. The classroom was split into groups of 2-3 students (depending on the total strength) and each group was given one tablet. The tasks are available over the web and can be accessed directly. But the colleges didn't have reliable internet connectivity, so the necessary files were copied to each tablet, and they were accessed locally using QR codes generated for the file paths. In the EG session, the teacher led the class, discussing the day's topics. The students in the groups interacted with the tablet-based system, as directed by the teachers and facilitated by the researchers (fig.3). They also completed related worksheets. The researchers' roles were primarily logistics related to the tablets, technical support (for the teacher as well as the students), facilitating activity based on TFV, and recording the classroom proceedings. After about 1-3 weeks, students in all the 6 classrooms were given a questionnaire, which included questions that sought to captured students conceptual understanding and reasoning approaches in the context of vectors.

<figure><img src="image_190.png" /><figcaption>Figure 2. The flow of the study and the sample. Figure 3. Students scanning the QR code in an EG Classroom.</figcaption></figure>

## Analysis and findings

Given the space constraints, we confine to reporting here the analysis of qualitative changes, both in students' learning (particularly imagination) and teachers' teaching practices in the classroom setting. The main data sources for the current analysis were video recordings, which were analysed to understand the differences in: classroom dynamics (CG and EG), interactions between teachers and the students, and interactions with various artefacts (textbook, blackboard, notebooks and worksheets, the tablet). For this, the video recordings were analyzed using two broad lenses – differences in learning, particularly imagination, and differences in teacher/teaching related practices. This is not a rigid and complete framework (like situated frames of instrumental genesis (Artigue, 2002) or socio-mathematical norms (Yackel & Cobb, 1996) etc) yet, but, we expect that iteratively useful holistic frames could emerge to analyse the new mixed media based classroom artefacts perhaps enhancing the existing frames incorporating embodied indicators of imagination like gestures.

## Learning practices

The main differences in the learning process, particularly related to imagination, between the CG and EG classrooms were in terms of the use of gestures, trajectories, collaboration and flow of learning.

### Gestures: Active and meaningful engagement with content

<figure><img src="image_191.png" /><figcaption>Figure 4. Student in CG using gestures to remotely access the content on the board (left). Student in EG using gestures to explain to the peer in the context of an activity on TFV (right).</figcaption></figure>

As one of our key design objectives was to help students imagine the vector in geometric terms, we examined the gestures students made in the two classrooms, as these indicate geometric thinking (Kim, Roth, & Thom, 2011). In the CG classroom, students were passive spectators most of the time, except for a few episodes where they took note of what was on the blackboard. Activity, when rarely present, was limited to nodding in agreement or tracing pens over the diagrams seen on the board (fig. 4 left). Gestures of this kind suggest a very limited engagement with the geometric content of vectors. The teacher's enactment of the textbook figures, which is the sole source for triggering the imagination of dynamic operations, is not resonated by the students. In the EG classrooms, more active and meaningful engagement with the content was observed. This was reflected in
more sustained gestures, along with discussions between students, which indicate conceptually deeper engagement with the geometric operations (fig. 4 right). These gestures were usually situated in conversation, where one student was actively trying to explain his imagination to another. The gestures indicate the possibility of students resonating with the teacher's writings (equations) or drawings (diagrams) and meaningfully engagement with content.

## Multiple trajectories for learning

In the CG classroom, the learning opportunities were only when the teachers enacted a concept, and most of the activity in the classroom involved taking notes, which followed the teacher's description of the topic and solutions to problems. In the EG classroom, the exploratory and open-ended nature of tasks led to a surprising number of problem-solving trajectories. The diversity of approaches indicate the high imagination and learning potential provided by the TFV-2. To illustrate, in one of the tasks, students had to create two vectors, whose resultant is a given vector with magnitude 60 and a direction of 40°. The following approaches were seen.

* Trial and error (the most common approach, where students manipulate the magnitude and direction of vectors randomly). Some students found some patterns of change as they kept interacting.

* Estimating the rectangular components along x and y-axes and creating them as the two vectors. However, these students struggled when asked to create another set of vectors. Here the students explicitly applied the interconnections between resolution into rectangular components and addition.

* Creating two vectors with magnitude 30 at 20° from the x-axis and then adding to find resultant of magnitude 60 at 20° instead of at 40°. This combination provides the opportunity for students to realise that magnitudes and directions (represented as angles) cannot be algebraically added directly.

* Creating the vector of magnitude 60 at 40° from the x-axis, and arguing that the other vector is a zero vector. This particular group of students tried out this task for a long time using trial and error method, and eventually applied the idea of zero vector quietly.

## Collaboration

In the CG classroom, there is very limited scope for students to engage in discussions with peers. The only conversations recorded between students involved asking for an extra pen or pencil, or to show some part of his/her notes for clarification, or giggles with playing (disinterested in the content of the lesson).

In the EG classroom, the very nature of the lesson plan involved making groups of 2 or 3 students, who worked together on the tablet. This inherently made the class collaborative. In addition, there were many episodes (similar to fig.5) where students naturally started interacting across groups. The nature of collaboration was grounded in the tasks they were performing on the tablets. Collaboration is not necessarily an indicator of an effective classroom, as there could be a lot of social and cognitive loafing (O'Donnell & O'Kelly, 1994), which could have happened in the EG classroom as well. However, as the students were involved in the collaborative activity using the TFV-2, the teacher could monitor student progress, to start discussions.

<figure><img src="image_192.png" /><figcaption>Figure 5. Students in EGs collaborating within their designated groups as well as beyond their groups.</figcaption></figure>

## Control on the flow of learning

In CG classrooms, most of the time was spent on the teacher delivering content in the lecturing mode. The teacher thus had full control over the flow of knowledge and learning in the classroom. There was very little scope for students to intervene in the flow and build their own knowledge. Further, the teacher decided when students could take notes, when to listen to him/her, which book was to be taken, who should respond to questions etc. This authority stance is central to the conventional classroom in developing countries, and this is required to some extent for smooth classroom management. However, this power structure often makes students intimidated, and they thus seldom interact with the teacher. Students tend to overcome this fear using chorus responses. The teacher wielded further control by appreciating (saying words like very good, interesting) or dismissing (ignoring or not attending to) student responses.

EG students actively participated in the enactment process along with the teacher, and this transferred agency to the students, who had more power in directing the flow of learning than in a normal classroom. This was reflected in the free interactions among students, the change in the nature of teacher-student interactions,
and also students' emotional connections with the tasks. The students were active participants with the teacher, and their interaction with the content was active and meaningful (see fig.6) as can be seen in the nature of gestures used. Based on these interactions, along with questions and discussions with the teacher, they controlled the flow of learning, and hence constructed their own knowledge. The teacher was no longer the sole controller of the flow of knowledge. Students generated knowledge during their interactions with their peers mediated by artfacts like TFV-2, and knowledge flowed horizontally through collaborative work. Moreover, as suggested by the teachers, if tasks could be designed for students to do at home (thus flipping the classroom), the flow of learning could become even more student-centred.

<figure><img src="image_193.png" /><figcaption>**Figure 6.** Students in a CG taking notes in their notebooks (1,2) Students in EG interacting with the tablet (3,4) and taking note of it in the worksheet (5,6).</figcaption></figure>

## Other observations

The CG classrooms were very calm, and there were never any emotionally charged moments. In contrast, in the EG classroom had a charged feel, and there were numerous moments of excitement and disappointment. There were multiple instances where EG students asked for similar systems for other topics in the curriculum. Students were very excited to see the tablets on the first day. By day 3 and 4, they started engaging with the TFV-2 in a more serious way and were not just excited due to the presence of the tablet.

## Teacher/Teaching practices

The introduction of the TFV-2 into the classrooms was not fully and easily embraced by the teachers, though there were changes that indicated possible integration eventually. Here we discuss some of the central differences (and lack of changes) captured in the video data. We start with a discussion of practice elements resistant to change.

### Resistant teacher practice

Three out of 5 EG teachers (T1,T2,T5) used the same diagram for teaching parallelogram law of vector addition, where the angle is less acute, even though the vectors used in triangle law, shown simultaneously on the board, were quite different (See fig.7). This drawing closely follows the textbook one and indicates the deep and subtle influence of the textbooks, where the teachers are conditioned by the textbook representation to such an extent that the diagrams seem into their practice without explicit awareness. While the TFV-2 has changed the way teachers present the content (as discussed in later subsections), there were such subtle aspects that were still driven by the textbook representations. We present this result first to indicate how deeply the textbook is integrated into teaching practice. This conditioning by textbooks extended to the way content was presented. In the EG classroom, where teachers had the possibility of widening the scope of discussions (like emphasizing and discussing the patterns of changes in the algebraic expressions through geometric manipulations, or the nature of the x and y components being interconnected by the circle). Such topics were not initiated by the teacher, and the scope of the class was still within the limits of the topics presented in the textbook. This reaffirms the conditioning role played by textbooks in setting classroom practices.

<figure><img src="image_194.png" /><figcaption>**Figure 7.** Teachers making parallelograms very similar to the one given in their textbook (right) with almost 90° angle while teaching parallelogram law of vector addition.</figcaption></figure>

In the CG classroom, the teaching narrative usually took the form of a lecture, where the teacher introduced the definition or the statement of a law, explained it using some examples, and then drew diagrams. The teacher used some gestures to enact the diagram. The students took written notes based on these practice elements. When some extra time was available before the end of the session, the teachers solved some numerical problems. The flow of the lesson was very linear and monotonic. In the EG classroom, the narrative was similar in the beginning (introduction to the definitions or statements, making diagrams). It then changed to describe the tasks the students needed to do using the system. But once students started interacting with TFV-2, this was
altered. Interestingly, there was a tendency for both CG and EG teachers to follow a set question and answer template for discussion (especially when stating the definitions or the laws of additions), as this helps students in standard assessment. In later discussions, T1, who has taught in an EG class, noted that despite the teachers being given the flexibility to use new modes of teaching, there is a lot of emphasis on examination results and time limitations. This emphasis comes from the way the education system in India is structured around exams, which restricts the possible narratives in the classroom.

## Practice elements that changed

A central change was the nature of teacher talk in EG classrooms, from mere description of static diagrams (like one arrow and another arrow forming two sides of a triangle, and the third side being the resultant, in triangle law of vector addition), to a more dynamic enaction, where teachers described the addition process. For instance, the teacher used the gestures in the TFV-2 to explain the process of adding two vectors saying "*now you know how to take vectors, just bring this vector (2nd vector) and attach here (head of 1st vector), then you will have to press your thumb (at the tail of 1st vector) and other finger here (head of 2nd vector), you will get the resultant*". Teachers in both CG and EG tended to use many gestures and bodily actions when explaining content using diagrams. However, there were differences in the gestures used by CG and EG teachers. In CG, teachers tended to use iconic gestures (like drawing an arrow in the air or using an opened palm to represent a vector) which are linked to diagrams on the board. Some were metaphorical (Roth, 2001) gestures (such as hand movements to show the splitting of vectors during resolution). In the EG classroom, teachers used similar gestures as a CG teacher during lecturing. However, they also incorporated gestures used in the TFV-2. For example, T4 and T3 used gestures influenced by the system when explaining triangle law of vector addition (see quote above). These gestures signify a dynamatisation of the vector operations, based on interactions in TFV-2

<figure><img src="image_195.png" /><figcaption>**Figure 8.** (Left) Teacher in CG using iconic gestures to show a vector, which is still a diagram. (Right) Teacher in EG using a picking action, reflecting that the vector is a touchable entity.</figcaption></figure>

In the CG classroom, the teacher stayed near the blackboard (as seen in many figures). As the TFV-2 allowed all EG students to enact vector operations, the teacher did not need to demonstrate the enaction in the class, and she thus had a lot of time available to move around in the class and engage in discussions with the students. The role of the TFV-2 in generating this behaviour is clear from the contrasting behaviour of T3 (only teacher present in both EG and CG) in the CG and EG classrooms. The discussions involved clarifying conceptual doubts and questions. This allowed the teacher to also get real-time feedback about students' understanding of vector concepts, and initiate discussion with individual groups or the entire classroom.

<figure><img src="image_196.png" /><figcaption>**Figure 9.** (Left and Center) Teachers engaging in discussion with student groups in EG. (Right) Teacher showing a group's work on the tablet to the entire classroom (a potential space for initiating a discussion).</figcaption></figure>

<figure><img src="image_197.png" /><figcaption>**Figure 10.** Teachers in CG calling students to the board to answer or solve some questions.</figcaption></figure>

Usually, teachers ask questions to check for understanding (Rosenshine, 1983), or when students are not attentive. In the CG classroom, the monologue of the teacher was often interspersed by questions, where the teacher explained a concept or a problem and asked one of the students to stand or come to the board and solve or answer a similar problem (Fig.10). There were some such episodes in the CG classroom, where the teacher called a student to perform subtraction of two vectors, or derivation of expression of parallelogram law by simple substitution, or to solve a numerical problem similar to the one the teacher had just solved. Irrespective
of the purpose of asking the question, they were close-ended questions with a unique answer.

In contrast, in the EG classroom, the teacher spoke to the entire classroom only in the beginning. The rest of the time was spent answering questions from students. The nature of the questions was open-ended (with no single answer) and exploratory, following the exploratory nature of TFV tasks. Besides the above changes, a few other episodes indicate significant changes in teacher thinking and practice, induced by the TFV-2.

As teachers started discussing possible lesson plans during LPW, they immediately suggested that we could design tasks for students to do at home (as they get more time to engage with the TFV-2) and then discuss their findings and responses in the classroom next day, which was aiming to flip-classrooms. After one of the lessons, T3 said - "besides helping the students in visualising and understanding the concepts, it (TVF-2) has clarified and made imagination easier for topics for me too". This shows how systems like TFV-2 could also help teachers understand complex concepts better, and also help develop pedagogical content knowledge (Shulman, 1986), particularly given the acute shortage of good teacher trainers and programs in the developing world. The teachers highly appreciated the TFV-2 when it was first shown to them. As discussed, some of the teachers were not comfortable using it in their teaching. However, after a few classes based on teaching using the system, the teachers became more comfortable and confident. Teachers have repeatedly requested extensions to the TFV-2, particularly to include 3D vectors, and also products of vectors, as well as waves. This is because it is difficult to draw and explain these using the blackboard. A physics teacher who was not part of the study sat through an entire classroom session, to see how the system was being used by students. Also, T5, who could not be part of the LPW, and who taught only in the CG classroom, repeatedly requested deployment of the system in the CG classroom. These indicate a shift to a motivation to use technology from an initial hesitation in teachers.

## Discussion

The TFV-2 provides a new model to address the problem of augmenting science and mathematics teaching and learning practices, particularly in the developing world. The focus on smoothly extending textbooks and related practices, rather than replacing them, makes our approach different from other similar approaches to integrating multiple external representations (Bodemer, Ploetzner, Feuerlein, & Spada, 2004), such as GeoGebra. Also, the high scaling possibilities provided by the teacher-driven QR-based augmentation of textbooks offer a new approach to address key resource limitations in developing world classrooms, in terms of both technology access and training costs. The QR-based approach also provides a way for educational research institutions to curate good learning resources (such as videos and problem banks), and include these periodically as annotations to current textbooks, without waiting for the lengthy and bureaucratic process of textbook revision.

We could see a transformation of the classroom space for the students and situations and practices for their learning emerged in them smoothly. Among teachers, we see they could adapt the technological intervention smoothly, but the changes in their practices were far subtler, as we see indicators of them still anchored to textbooks. It is here, a key insight emerges about the central role textbooks play in directing teacher practice, and how building on the textbook could lead to rapid changes in teaching practices, particularly in terms of the way teachers direct/control the flow of knowledge in the classroom. The augmentation of textbooks could also help in swiftly upgrading teacher thinking and pedagogical knowledge of science and mathematics concepts, which is a critical issue in the developing world. Such an upgrade of pedagogical content knowledge (Van Driel, Verloop, & de Vos, 1998) could be better facilitated by QR augmentations that support flipping the classroom, as this feature would raise the number of potential questions from students, and in turn, motivate teachers to come prepared to answer these questions.

The narrow scope of our vector application, which seeks to just seed imagination of geometry, is different from most other interactive systems for learning vectors (such as PHET), which are scoped more widely. This scoping makes such systems unsuitable while trying to extend teachers' current textbook-based practices, which tend to be very modular. The touchable and manipulable vector also potentially creates a new and embodied way of understanding vectors and related concepts for learners, as has been argued in the case of touchable numbers (Sinclair & de Freitas, 2014). The embodied interaction with vectors also makes our approach different from augmented reality approaches, which focus more on the visual component.

More broadly, our mixed media design stance -- based on an integration of cognitive, practice, and access factors -- treats textbooks as a very useful resource for advancing technology-based solutions for education, particularly in the developing world. This approach seeks to create a bridge from current practice to future practice, where digital platforms such as NetLogo could support the learning of cognitive skills not supported by textbooks, such as computational thinking (Dickes, Sengupta, Farris, & Basu, 2016).

However, we take a more careful design approach towards this shift, analysing the positive role textbooks could play, through their ability to promote working memory based operations such as mental rotation and modelling, particularly by *limiting* dynamic interaction. We believe this reflective stance, and the specific
design instantiation of this stance outlined here provides a way to reevaluate key assumptions underlying educational technologies for learning science and mathematics.

## References

Artigue, M. (2002). Learning Mathematics in a CAS Environment: The Genesis of a Reflection about Instrumentation and the Dialectics between Technical and Conceptual Work. *International Journal of Computers for Mathematical Learning*, 7(3), 245.

Balacheff, N., & Kaput, J. J. (1996). Computer-Based Learning Environments in Mathematics. In *International Handbook of Mathematics Education* (pp. 469–501). Springer Netherlands.

Barniol, P., & Zavala, G. (2014). Test of understanding of vectors: A reliable multiple-choice vector concept test. *Physical Review Special Topics - Physics Education Research, 10*(1), 010121.

Bingimlas, K. A. (2009). Barriers to the successful integration of ICT in teaching and learning environments: A review of the literature. *Eurasia Journal of Mathematics, Science & Technology Education, 5*(3).

Bodemer, D., Ploetzner, R., Feuerlein, I., & Spada, H. (2004). The active integration of information during learning with dynamic and interactive visualisations. *Learning and Instruction, 14*(3), 325–341.

Dickes, A. C., Sengupta, P., Farris, A. V., & Basu, S. (2016). Development of Mechanistic Reasoning and Multilevel Explanations of Ecology in Third Grade Using Agent-Based Models. *Science Education, 100*(4), 734–776.

Dorier, J.L., & Sierpinska, A. (2002). Research into the Teaching and Learning of Linear Algebra. In D. Holton, M. Artigue, U. Kirchgräber, J. Hillel, M. Niss, & A. Schoenfeld (Eds.), *The Teaching and Learning of Mathematics at University Level: An ICMI Study* (pp. 255–273). Dordrecht: Springer Netherlands.

Hestenes, D. (2010). Modeling Theory for Math and Science Education. In R. Lesh, P. L. Galbraith, C. R. Haines, & A. Hurford (Eds.), *Modeling Students' Mathematical Modeling Competencies* (pp. 13–41). Springer US.

Hillel, J. (2000). Modes of Description and the Problem of Representation in Linear Algebra. In J.-L. Dorier (Ed.), *On the Teaching of Linear Algebra* (pp. 191–207). Dordrecht: Springer Netherlands.

Karnam, D., Agrawal, H., Mishra, D., & Chandrasekharan, S. (2016). Interactive vectors for model-based reasoning. In Chen, W. et al. (Eds.), *The Workshop Proceedings of the 24th International Conference on Computers in Education, India: Asia-Pacific Society for Computers in Education*, 401–406.

Karnam, D., Agrawal, H., & Chandrasekharan, S. (2018). “Touchy Feely Vectors” changes students’ understanding and modes of reasoning. In Yang, J. C. et al. (Eds.), *The 26th International Conference on Computers in Education. Philippines: Asia-Pacific Society for Computers in Education*.

Kim, M., Roth, W. M., & Thom, J. (2011). Children’s gestures and the embodied knowledge of geometry. *Journal of Science and Mathematics Education, 9*(1).

Magnani, L., Nersessian, N., & Thagard, P. (2012). *Model-based reasoning in scientific discovery*. Springer Science & Business Media.

Nersessian, N. J. (1999). Model-Based Reasoning in Conceptual Change. In *Model-Based Reasoning in Scientific Discovery* (pp. 5–22). Springer US.

O'Donnell, A. M., & O'Kelly, J. (1994). Learning from peers: Beyond the rhetoric of positive results. *Educational Psychology Review, 6*(4), 321–349.

Papert, S. (1980). *Mindstorms: Children, Computers, and Powerful Ideas*. NY, USA: Basic Books, Inc.

Papert, S. (1994). *The children's machine: Rethinking school in the age of the computer*. Basic books.

Rosenshine, B. (1983). Teaching functions in instructional programs. *The Elementary School Journal, 83*(4), 335–351.

Roth, W. M. (2001). Gestures: Their Role in Teaching and Learning. *Review of Educational Research, 71*(3), 365–392.

Shulman, L. S. (1986). Those Who Understand: Knowledge Growth in Teaching. *Educational Researcher, 15*(2), 4–14.

Sinclair, N., & de Freitas, E. (2014). The haptic nature of gesture: Rethinking gesture with new multitouch digital technologies. * Gesture, 14*(3), 351–374.

Uluyol, C., & Agca, R. K. (2012). Integrating mobile multimedia into textbooks: 2D barcodes. *Computers & Education, 59*(4), 1192–1198.

Van Driel, J. H., Verloop, N., & de Vos, W. (1998). Developing science teachers’ pedagogical content knowledge. *Journal of Research in Science Teaching, 35*(6), 673–695.

Yackel, E., & Cobb, P. (1996). Sociomathematical Norms, Argumentation, and Autonomy in Mathematics. *Journal for Research in Mathematics Education, 27*(4), 458–477.

Young, E. &. (2016). *Future of Digital Content Consumption in India*. EY.
# Evaluating an Adaptive Equity-Oriented Pedagogy on Student Collaboration Outcomes Through Randomized Controlled Trials

Andrew Estrada Phuong, University of California Berkeley, andrew.e.phuong@gmail.com

Judy Nguyen, Stanford University, judynguyen@stanford.edu

**Abstract:** This study evaluates an adaptive equity-oriented pedagogy (AEP) through a randomized controlled trial. AEP employs evidence-based teaching practices (e.g., formative assessment, universal design for learning) to address college students' diverse learning needs (i.e., their strengths, interests, and areas for growth). AEP provides collaborative computer-supported project-based opportunities that enable students to extend the application of course concepts to embedded contexts that engage them as learners. To compare AEP and active-learning control conditions, this study utilizes identical measures: validated surveys, interviews, observation notes, anonymous course-feedback forms, and formative and summative assessments. Multivariate regression analyses suggest that students learning through AEP outperformed control conditions by a full letter grade and scored on average 14.20 percentage-points higher on final assessments, when controlling for disability status, gender, and pre-test achievement. AEP has a large standardized effect size on the final (d=2.40) and provides a framework to improve student success through embedded and extended learning opportunities.

## Adaptive equity-oriented pedagogy (AEP), research questions, and rationale

This study evaluates an adaptive equity-oriented pedagogy (AEP) that seeks to improve diverse students' success through computer-based formative assessment and computer-supported collaborative project-based learning. Applying AEP, instructors administer weekly computer-based student assessment and surveys to diagnose students' learning needs (i.e., their strengths, interests, and areas for growth). Using student data and analytics, instructors iteratively adjust how they provide feedback, model key skills needed for summative assessments, and implement deliberate practice activities. Students pursue their interests by applying course concepts to a novel context through collaborative project-based learning.

AEP seeks to engage learners in an embedded space by making the content and context relevant so that students can extend their learning into the real-world as a scientist. These embedded learning opportunities are beneficial since instructors address skills that are relevant to students' lived experiences and contexts outside of the classroom. To achieve this goal, students' learning is embedded in the real world since they can apply concepts to a range of contexts, which include researching longitudinal data on students' mental health, business management practices, and cancer treatments. Through project-based learning, students apply design-based research to collaboratively examine and address shared problems. Students work in teams to develop competing hypotheses based on existing literature, their experiences, and the concepts presented in class. They then gather data to test these hypotheses through statistical software as they collaborate online to collect data, build databases, generate code, produce visualizations, analyze data, and interpret findings. Students work together to extend course concepts in embedded contexts to make sense of real-world data. Collaboration in these contexts is critical, since students offer diverse perspectives on ways to refine research design and analyses. AEP facilitates this process because it identifies barriers to student collaboration. For example, instructors use formative assessment and surveys to diagnose students' learning needs with respect to collaboration.

Through these strategies, AEP enables instructors to overcome barriers that otherwise inhibit collaborative learning in university classrooms. One such barrier is the conflict that arises in group projects when members do not complete tasks on time. Reasons for incomplete tasks include lack of proper planning, miscommunication, students not knowing how to apply concepts to large tasks, and students working on tasks that do not engage their interests and strengths. The AEP model helps students develop a structure for collaborative work. For instance, students form teams based on common interests and are asked to collaboratively create a task chart via Google sheets or other mediums to assign roles during group projects. To create a system of accountability, students use this chart to designate individuals who work collaboratively on tasks that have deadlines. To foster trust and motivation, group members assign tasks and roles that leverage their members' strengths and interests. Groups also co-construct norms to navigate conflict and ensure accountability. In addition, instructors continuously model collaborative strategies and leverage student data to provide additional support that groups need. Although these approaches may be common in K-12, they are fairly uncommon in U.S. higher education (Phuong et al., 2017). By equipping students with the skills to collaboratively address problems using technology, AEP aims to increase students' sense of belonging in STEM, since they are developing skills that
society values and needs to thrive in a digital age. These experiences provide opportunities for students to practice
academic discourse and to connect it to their interests and sociocultural identity (Gee, 1996).

To formally study whether AEP can have strong impacts in a university statistics course, we conducted a mixed-methods randomized controlled trial (RCT). We examine underrepresented minority (URM) student groups that are often overlooked in research on higher education pedagogy. Our research questions below examine an AEP treatment course's impact relative to an active-learning control course and on subgroups (i.e., disability status and gender), since these URM groups have experienced low achievement, persistence, and retention in STEM (McGregor et al., 2016; Stout & Wright, 2016). This study controls for students' pre-test achievement on the final assessment in treatment and control conditions, since prior achievement significantly influences subsequent achievement (Hattie, 2009). Based on these interests, we explore the following research questions:

1. Is there a mean difference on final collaborative assessments between students in the AEP treatment condition and students in the control condition, after controlling for disability status (i.e., students with and without disabilities), pre-test achievement, and gender (i.e., gender non-conforming, female, male)?

2. Is the impact of AEP on student assessment scores consistent across disability and gender groups?

Our research questions evaluating AEP are significant since universities experience difficulties recruiting and retaining STEM students, especially URM (U.S. Department of Education, 2017). Research shows that 90% of students leaving STEM cited poor teaching as a primary concern (Seymour & Hewitt, 1997). Specifically, university statistics courses, a foundational requirement for STEM fields, are typically computation-heavy and lecture-based (Allen et al., 2012). With limited opportunities for real-world application, students can struggle seeing the relevance of learning statistics (Allen et al., 2012). Furthermore, URM students (e.g., students with disabilities, gender non-conforming students) with limited mathematical preparation often experience math-phobia and poor academic achievement (Tishkovskaya & Lancaster, 2012) which can affect their confidence, scientific identity, persistence, and retention (Peters, 2014; Stout & Wright, 2016). Since teaching can directly impact student success (Condon et al., 2016), studying pedagogy to improve students' academic achievement, especially for URM, in STEM is critical.

Potential significance

Increasing recruitment and retention in STEM, especially for URM, is important because this population of students can offer diverse perspectives on problems and solutions. Increasing diversity can enable future leaders to be more responsive to their constituents and environment. Additionally, many industries in today's workforce require competency with computer-based technologies and collaboration in team settings. However, many higher education STEM courses are not adequately equipping students, especially URM, with technology-based competencies and interpersonal, collaborative skills (Gasiewski et al., 2012). This is important since students in a scientific community need to learn to engage with others, recognizing community guidelines and norms. Many of these norms are often not explicit to students, especially for URM and first-generation students who did not have access to academic-oriented scientific communities. AEP seeks to make these guidelines and norms explicit by modeling these norms and providing students with opportunities to collaboratively practice and build on these norms with a community of learners. To draw students into a scientific community, the AEP model assumes that like scientists, students often have an interest and purpose to pursue a research topic. Based on this premise, the AEP model seeks to understand learners' sociocultural experiences and socio-academic interests. Through surveys and formative assessments, AEP enables instructors to understand their students' sociocultural environment and context—their lived experiences and interests that inspire them to pursue STEM. For example, some students are drawn to medicine because their parents suffered from medical conditions and could not address their illnesses due to financial constraints. Hence, AEP empowers instructors to identify what brings students to academic STEM spaces by providing frameworks to understand learners and their sociocultural histories.

As students bring themselves into their projects, we study how students offer diverse strengths and perspectives to practice “thinking scientifically” as they design research questions, make and test predictions, solve problems, and critique their own and others’ reasonings (Deslauriers et al., 2011). Students also collaborate synchronously and asynchronously on their computers through Google docs where they work together to share visualizations and to analyze and interpret data. By cultivating environments that mirror a professional scientific community, students have opportunities to collaboratively practice and engage with scientific discourse, where they can embody communicating, behaving, thinking critically, and solving problems like a scientist (Gee, 1996).

Our research represents a significant contribution. First, it is useful for practitioners since it provides a research-based pedagogical framework and tools that instructors can use to increase equity in student outcomes, especially for URM. Second, this study's methodological approach advances higher education pedagogy scholarship practices; this is the first study to our knowledge that uses a randomized controlled trial (RCT) with
the same university instructors where the control condition employs active learning in a university STEM course. In many higher education STEM studies, the control condition differs in multiple respects than the treatment making the source of different outcomes difficult to isolate. For instance, the control is often lecture-based and employs instructors with different characteristics than those in the treatment. In this study, the treatment and control conditions are more precisely contrasted allowing us to focus more precisely on AEP effects of interest.

# Theoretical approaches

## Conceptual framework of adaptive equity-oriented pedagogy (AEP)

In this section, we describe AEP's key elements and the learning theories that underpin them. Drawing on McCallum's (2013) Assessment-Instruction (A-I) conceptual framework, AEP focuses on collecting ongoing diagnostic assessment data to guide classroom instruction, decisions, and lesson planning. These assessments gather information on students' backgrounds, interests, and experiences to understand the cognitive and noncognitive factors that impact student engagement in light of their sociocultural history. Applying AEP, instructors use weekly data on students' learning needs to continuously adjust how they

1. foreground how course concepts are relevant to students' goals and shared contexts

2. provide brief warm-up active learning exercises where students can engage in productive struggle with concepts and make meaning individually or collaboratively

3. model software skills, expert thinking on concepts, and strategies that students need to excel on summative assessments; during this time, instructors build on and respond to students' critical thinking strategies from previous assessments and/or productive struggle experiences

4. provide written steps and strategies (e.g., task analysis) of how to approach problems that align with rubrics and the rigor of summative assessments

5. include class time for students to practice these skills and strategies to collaboratively analyze and interpret data on computers; during these deliberate practice activities, students articulate to each other which concept-driven step-by-step strategies are useful for their team final project and why

6. provide low-stakes feedback during class to address misconceptions and close gaps in understanding

7. offer in-class and project-based opportunities for students to incorporate feedback and reinforce learning AEP seeks to support URM students' success by building on McCallum's (2013) A-I model and Vygotsky's (1978) sociocultural theory of learning, in which learning is situated as a cultural and social process. To avoid the lack of social, collaborative interaction associated with more traditional lecture-based pedagogies, AEP draws on formative assessment and universal design for learning (UDL) to address a wide range of learning needs. Throughout this process, AEP provides an adaptive active-learning framework that helps instructors use formative assessment to iteratively adjust their UDL and collaborative learning practices.

## Formative assessment

AEP uses formative assessment (Black & William, 1998) by engaging Vygotsky's (1978) sociocultural theory and the zone of proximal development (ZPD), where an instructor assesses a student's actual developmental level (i.e., a student's level without assistance) to help them reach their potential developmental level (i.e., a student's level with assistance). Using formative assessment data, the instructor modifies evidence-based practices (e.g., modeling key skills, collaborative deliberate practice opportunities, feedback) to help students reach that potential. Instructors assess and provide feedback to all students via formative assessment instead of assuming that students enter the class with the appropriate resources and skills to meet course goals. Furthermore, formative assessment aligned with the final exam's rigor clarifies expectations for students unfamiliar with the dominant cultural capital and academic discourse assumed in many college classrooms (Bourdieu, 1973; Phuong et al., 2017). Formative assessment promotes equity in student outcomes since struggling students are more likely to persist in a field when they develop stronger conceptual foundations, reflect on their growth, have a collaborative community, and are confident in their work (Ambrose et al., 2010; Phuong et al., 2017).

AEP's in-class formative assessment, modeling, collaborative practice, and feedback loops promote equity in student outcomes since they support novice learners and underrepresented students who may struggle with challenging concepts. According to Schwartz et al. (2015), novices "find it harder to engage in deliberate practice on their own, because novices often do not have the experience to know what skills they should be working on or how to go about it" (p. 295). Incorrect types of practice can reinforce misconceptions. AEP's in-class formative assessment, modeling, practice, and feedback loops allow instructors to review challenging concepts. AEP also leverages collaborative learning via peer-to-peer support since peers who more recently
learned concepts often empathize with barriers to understanding concepts. Often, these peers can break down course material in digestible ways for struggling and advanced students. This form of reciprocal teaching enables students to work with their peers and draw on each other's expertise to move farther together in their ZPD.

In AEP, modeling key skills and strategies provide opportunities for instructors to offer various worked examples that show students “what to do and why” when they approach problems (Schwartz et al., 2015, p. 295). Instructors applying AEP provide step-by-step explanations for solving problems in writing so that students can understand different ways to read prompts, identify underlying concepts in the problem, explain why they are doing each step, articulate how each step connects to key concepts, and understand when these steps can and cannot be generalized to other situations. These step-by-step strategies further reduce cognitive load because the steps help chunk students’ mental processes and deliberate practice opportunities (Sweller, 1994). This approach can make learning new concepts less daunting for novice learners. These strategies and collaborative practice opportunities can help improve students’ success and self-efficacy with mastering challenging concepts, especially for URM students who had less access to college-level coursework. AEP seeks to improve students’ conceptual understandings and self-efficacy so they can actively contribute to their project teams and their peer’s learning during computer-supported, collaborative activities.

These formative assessment, collaborative practice, and feedback loops can also clarify expectations for success and foster students’ metacognition and self-regulation processes (Nicol & Macfarlane-Dick, 2007). For example, students reflect on and monitor whether they understand concepts when they take formative assessments and practice scientific thinking. This process can help students identify their strengths, interests, and areas for growth, which can help them set academic goals and develop strategies to advocate for their learning. For URM students who feel underprepared for college STEM courses, providing weekly ungraded assessments and supportive collaborative activities creates a psychologically safe space. Creating a psychologically safe space is crucial because many URM students have experienced high incidences of microaggressions and low levels of STEM achievement, sense of belonging, persistence, and retention rates (Stout & Wright, 2016). Offering greater opportunities for URM students is necessary because many URM families often do not have equitable access to extracurriculars, resources, and rigorous curricula compared to more privileged families (Martin et al., 2016).

## Universal design for learning

To support diverse learners, AEP also incorporates UDL to optimize learning by engaging affective, recognition, and strategic brain networks (Rose et al., 2002). UDL enables students to learn, demonstrate, and reinforce knowledge in various ways by providing multiple means of engagement, representation, and action and expression (Rose et al., 2002). For example, AEP instructors synthesize lectures, simulations, technology, computer-based activities, and project-based learning. Building on Lee's (2005) culturally-responsive-teaching framework, instructors apply UDL strategies to draw on students' funds of knowledge—their background, interests, aspirations (Moll et al., 1992)—to increase their sense of belonging and engagement with developing academic skills. To increase student engagement with academic content, instructors: 1) use student survey data to explicitly articulate how students' interests and aspirations are relevant to course material during interactive lectures and active learning activities, 2) visualize challenging concepts with written annotations, and 3) ask students to apply course content to their own contexts. Consequently, students can see how their backgrounds, interests, and identities are relevant to and are strengths in STEM. In addition, students can exercise agency as they apply concepts to collaboratively address social challenges that relate to their lives and communities using technology.

In sum, AEP's formative assessment and survey strategies provide instructors with data to understand students' interests, barriers to learning, and any microaggressions they face in STEM. AEP addresses cultural and social misalignments by equipping instructors with a framework to clarify expectations for success, create a space that validates students' backgrounds, and provide opportunities for students to code switch and connect their interests with academic discourse. AEP supports struggling students since it empowers learners to make meaning through productive struggle and assists students who are still acquiring the skills to complete conceptual tasks. Seeking to avoid assimilationist approaches to equity, AEP's end goal is not to ask students to think like the instructor and to solely learn a form of routine expertise via didactic modes of teaching (Schwartz et al., 2015). Instead, AEP encourages students to develop a greater sense of adaptive expertise, where students examine problems and apply relevant concepts with and without direct instruction (as described above). Throughout this process, students practice and synthesize different ways of scientific thinking gained from their community, instructors, peer-to-peer collaboration, and their own insights.

AEP is responsive to ongoing learning and is culturally responsive because it does not perceive students' pre-test scores and backgrounds as deficits dictating an endpoint. Instead, AEP sees students' backgrounds as sources for innovation, and AEP therefore empowers students to leverage their existing social and cultural competencies to apply scientific concepts to both novel problems and contexts of their choice. For example, AEP's
instructional and project-based approach enables students to co-construct and make meaning of STEM concepts to utilize innovative strategies that build on their peers' and instructors' critical thought process. Drawing on multiple frameworks, AEP focuses on equipping students with developing the mindset, skills, sense of community, and confidence to tackle meaningful problems where there exists no single solution.

## Methods

127 undergraduate student participants from a R-1 US university took the same course, but were randomly assigned into treatment or control sections. The course required students to complete a pre-test on foundational course content and a final assessment. These assessments and the following identical measures were collected in both conditions: validated surveys, demographic information, interviews, observation notes, anonymous course-feedback forms, and formative and summative assessments. Four observers took notes during every class in both conditions to determine if 1) the treatment course applied AEP and 2) the control course did not adjust instruction based on data. The treatment and the control conditions had the same instructors who employed exemplary active-learning strategies (e.g., modeling key skills, deliberate practice, dynamic lecturing, dialogue, case studies). However, instructors adjusted their teaching practices for students in the treatment group based on weekly student data. For the control condition, these same instructors did not adjust their teaching based on weekly student data, since they did not see these data. To ensure that the treatment did not impact pedagogical practice in the control condition, the control condition was taught before the treatment condition in each week of instruction. Instructors were provided with data from the treatment condition only after they had taught the control condition. In the control condition, students received scores on assessments but instructors did not.

We conducted multiple regression at the 5% significance level. We fitted one regression model for research question 1: the final assessment scores (scale 0-100) were regressed on pre-test (scale 0-100), dichotomous variables for treatment group and disability status, and a categorical variable for gender (female and male with gender non-conforming as the reference category). For question 1, the coefficient of treatment is an estimate of the difference in mean final achievement between treatment and control students after controlling for disability status, gender, and pre-test. For question 2, we looked for an interaction between treatment group and disability status within the regression model. The coefficient for this interaction term represents the mean difference in the treatment effect between disabled and non-disabled students, after controlling for pre-test and gender. We also looked for an interaction between treatment group and gender within the regression model. This produced two interaction terms. The coefficient for one interaction term represents the mean difference in the treatment effect between gender non-conforming and female gender groups, controlling for pre-test and disability status. The coefficient for the other interaction term represents the mean difference in the treatment effect between gender non-conforming and male gender groups, controlling for pre-test and disability status.

We also used Saldaña's (2009) methods of qualitative coding, categorizing, and identifying patterns. We inductively coded data by identifying themes and assigning thematic codes. We then created overarching thematic codes based on patterns, which became our analytical focus. To establish inter-rater reliability, we created a codebook with definitions of engagement and achievement with examples to guide our analyses. We triangulated data sources and multiple viewpoints (e.g., instructors, students, and researchers) to further analyze data.

## Findings

### Descriptive statistics

The treatment and control groups have similar frequencies on disability status and gender explanatory variables (See table 1). The control's mean pre-test score (M=29.42; SD=12.84) is comparable to the treatment's (M=29.70; SD=12.24). Additionally, the treatment's mean final score (M= 97.82; SD=1.75) is higher than the control's (M=83.66; SD=8.10).

<table><caption>Table 1: Contingency table of disability status and gender categories of control and treatment groups</caption><thead><tr><th rowspan="2">Demographic Variable</th><th>Control (n=65)</th><th>Treatment (n=62)</th></tr><tr><th>N (%)</th><th>N (%)</th></tr></thead><tbody><tr><td>Disability Status</td><td></td><td></td></tr><tr><td>Non-Disabled Students</td><td>42 (64.62%)</td><td>41 (66.13%)</td></tr><tr><td>Disabled Students</td><td>23 (35.38%)</td><td>21 (33.87%)</td></tr></tbody></table>
Gender

<table><tr><td>Gender Non-Conforming</td><td>16 (24.62%)</td><td>15 (24.19%)</td></tr><tr><td>Female</td><td>26 (40.00%)</td><td>25 (40.32%)</td></tr><tr><td>Male</td><td>23 (35.38%)</td><td>22 (35.48%)</td></tr></table>

Regression analyses

We conducted a regression analysis with robust standard errors to correct for heteroskedasticity, in order to examine the treatment effect on collaborative final project outcomes, controlling for disability status, pre-test achievement, and gender. See table 2 for regression results. We performed appropriate regression diagnostics to validate the use of this approach. There was no evidence of collinearity between predictors (mean VIF =1.24).

<table><caption>Table 2: Multiple regression with robust standard errors for the effect of treatment group, pre-test, disability, and gender</caption><thead><tr><th>Variable</th><th>Est. Coeff. (Robust Std. Err)</th><th>95% Confidence Int.l</th><th>p-value</th></tr></thead><tbody><tr><td>Treatment</td><td>14.20 (.99)</td><td>12.23</td><td>16.17</td><td>&lt;0.001</td></tr><tr><td>Pre-test</td><td>0.03 (0.04)</td><td>-0.06</td><td>0.12</td><td>0.50</td></tr><tr><td>Disability</td><td>2.65 (1.16)</td><td>0.35</td><td>4.94</td><td>0.02</td></tr><tr><td>Gender (Reference: Non-Conforming)</td><td></td><td></td><td></td><td></td></tr><tr><td>Female</td><td>-2.73 (1.40)</td><td>-5.50</td><td>0.03</td><td>0.05</td></tr><tr><td>Male</td><td>-2.28 (1.34)</td><td>-4.93</td><td>0.37</td><td>0.09</td></tr><tr><td>Intercept</td><td>86.38 (2.58)</td><td>81.28</td><td>91.48</td><td>&lt;0.001</td></tr></tbody></table>

*Research question 1.* The treatment group on average is estimated to perform 14.20 percentage points higher (SE = 0.99, 95% confidence interval from 12.23 to 16.17) on the final than the control, after controlling for disability status, gender, and pre-test achievement. This difference was statistically significant (t =14.27, df=121, *p*<0.001). This model's *R*² is 0.63, suggesting that 63% of the variation in the final assessment scores can be explained by treatment group, disability, gender, and pre-test.

*Research question 2.* The effect of AEP persists across disability status and gender groups. No significant interaction exists between treatment group and disability status at the 5% level. Additionally, no significant interaction exists among treatment group and gender at the 5% level. Therefore, students in each of these groups appear to benefit equally from AEP.

Qualitative findings

At the beginning of the semester, survey data showed that treatment and control students had anxiety about collaborative learning. Many of them were unconfident in their skills and they often felt uncomfortable interacting with peers because they did not know each other or their peers' strengths. Students also felt that project requirements were overwhelming since they had multiple parts and asked for deep conceptual application. One student stated, "Working with others on a project that has no clear answer feels much more difficult than a multiple-choice exam." Students also added that using statistical software felt daunting and they had fears about their team not completing project tasks. To alleviate student anxiety, instructors applied AEP strategies to address interpersonal and conceptual shortcomings that students voiced on formative assessments and surveys. In surveys, treatment students indicated that it was helpful when instructors provided team activities and peer-to-peer validation exercises to address these shortcomings and foster a sense of community.

Coding of survey data suggests that AEP provided a structure that helped improve students' collaborative learning skills and experiences. Treatment students indicated that they learned effectively from applying statistical competencies via peer-to-peer activities to succeed in a collaborative scientific community. These students noted that group members contributed effectively to the team since peers and instructors ensured that everyone understood how to apply course concepts and use software. These treatment students highlighted how instructors
supported groups by providing additional examples of how other organizations dealt with interpersonal challenges. Moreover, treatment students appreciated completing point-people, task, and deadline sheets, because they could plan and break down projects into manageable tasks that could be completed each week within an hour. These tasks had deadlines with enough time for the team to provide feedback to each other and to allow extensions that did not significantly slow down the team. A majority of students also found it less stressful when they could submit project pieces in small chunks to a folder or other medium (e.g., Canvas page, Google drive, project Dropbox folder), because they could earn points for these smaller and intermediate submissions. Treatment students also highlighted how instructors used assessment data to provide resources and page numbers from readings to address group project needs. Moreover, the teaching assistants monitored who was completing tasks on time. They provided suggestions to group members on how they could ask questions to understand and address the barriers that team members had with completing tasks on time. The teaching assistants regularly encouraged team members to play to their strengths and to offer each other collaborative support with completing tasks. Students also mentioned that ungraded individual reflections mitigated conflict and contributed to more productive collaboration, since students self-acknowledged and addressed their areas for growth. This study shows that formative assessment data in higher education is not only useful for addressing students' conceptual difficulties, but is also beneficial for addressing barriers to collaborative learning.

Based on surveys and interviews, treatment students reported excelling on the collaborative final project because 1) they received weekly feedback from peers and instructors and 2) instructors adjusted teaching in response to student performance. When asked to indicate effectiveness of various teaching practices, treatment and control students appreciated how steps for challenging problems were broken down, linked to definitions of concepts, provided in writing, and modeled for them with reference to a rubric. Treatment students reported being more prepared for collaborative in-class activities when instructors provided class-discussion questions and lesson-plan outlines before class. Treatment students also consistently emphasized how instructors addressed misconceptions from weekly assessments, reviewed challenging concepts, and validated and built on the different ways students arrived to accurate solutions. Additionally, these students reported improved learning when instructors offered written cues and strategies to help students analyze question prompts. These students said this approach helped them apply different skills and thought processes when completing assessments. Moreover, treatment students stated that the collaborative learning process was helpful because they were held accountable to a peer and had to explain and answer questions about the rationale behind their thought processes. Treatment students appreciated using various concept-driven step-by-step strategies to identify their problem and goal, select the strategies to address their goal, and then develop a rubric for assessing their work and their peers' work. These students mentioned that this process helped them engage in purposeful application and avoid mindless application of code and strategies when analyzing data.

By contrast, data from the control condition show consequences when instructors were not aware of barriers to student collaboration. Many control students mentioned experiencing difficulties with collaborative assessments since they needed more review of dense concepts and software skills to contribute effectively to their teams. Moreover, several control students indicated that they were not satisfied with group projects because collaboration was strained. For example, some control students felt that they could not voice their concerns to their instructors, fearing repercussions for their teammates. Many control students indicated that they decided to work on tasks individually to avoid conflict. Fortunately, a majority of students in both conditions found synchronous and asynchronous computer collaboration to be helpful for sharing code, editing visualizations, and analyzing data together, especially when they used Google doc functions. For example, these students used collaborative editing features and clicked a user's icon in the Google doc, which would bring them to the location within the document where their counterpart was working. Students in both conditions reported learning effectively when they collaboratively worked on clicker-esque exercises because they could explain concepts to each other and question each other's thinking.

## Implications and conclusion

AEP's large standardized effect size on the final (d=2.40) is consistent with previous studies that examined the impacts of formative assessment and active learning (Deslauriers et al., 2011; Froyd, 2008; Phuong et al., 2017). AEP provides a framework on how to leverage the benefits of extended and embedded collaborative learning. Students in the treatment reported how learning to diagnose and break down problems from instructors and peers helped them better understand concepts since they could see the application of these concepts in collaborative tasks and via computer-based visualizations. Highlighting the benefits of embedded learning, treatment students emphasized that the material felt more relevant when they were asked to collaboratively discuss different step-by-step modeled strategies that were and were not useful for the context of their team project and why. Alluding to the advantages of extended learning, treatment students also said the concepts and statistical tests made more
sense and were worth learning when they were applicable to contexts and data that were meaningful. These findings suggest that AEP can help instructors address their students' learning needs (i.e., their interests, strengths, and areas for growth) to improve computer-supported collaborative learning experiences in real-world contexts.

This research is novel since no study to our knowledge has used an RCT with the same university instructors to compare an equity-oriented framework with a control condition that employed exemplary active-learning strategies. This control condition avoided the pedagogical limitations cited in multiple studies across different higher education contexts for introductory STEM courses. Pervasive across several colleges, including the university in this study, these limitations include curved grading, competitive and un-collaborative peer environments, overpacked lecture-based curricula, limited faculty interaction, and artificially difficult exams that are disconnected from both classroom instruction and the real world (Bettinger 2010; Barr, Gonzalez, & Wanat 2008; Crisp, Nora, & Taggart 2009; Eagan et al. 2011; Seymour & Hewitt 1997). At the R-1 institution we studied, statistics instructors in introductory courses and electives primarily teach through lecture and typically do not adjust teaching based on weekly formative assessments aligned with the final's rigor. They often have problem sets and classroom exercises that are lower in rigor than the final and midterms. For the active learning control condition in this study, instructors modeled key skills and provided deliberate practice and active learning activities to students. However, like many higher education STEM faculty who have not been formally trained in teaching, the control instructors did not adjust instruction based on weekly assessments. Nevertheless, control students, on average, performed about a third to half a letter grade higher than department norms for introductory statistics courses. By comparison, the treatment group, on average, scored over a full letter grade higher than these department norms. Therefore, there are significant implications for AEP in introductory college statistics and data science courses that are foundational to STEM disciplines. These courses can be gateway courses, which impact students' attitudes, confidence, sense of belonging, persistence, retention, and identity in STEM. Applying AEP can benefit universities and STEM-related departments that want to create collaborative spaces that retain URM.

Limitations to this study include institutional context and that the disability variable combines physical, invisible, and mental health disabilities. Another possible limitation is that the control was taught before the treatment for each week of instruction. However, in previous studies, the standardized effect size (d=2.36) was similar when the AEP treatment condition was taught before the active learning control condition (Phuong et al. 2017). Future research directions include controlling for additional variables (e.g., race, class) and finding ways to support faculty with implementing AEP to improve student learning. It would be particularly interesting to examine whether students in the treatment group would perform better in subsequent courses. We would also like to examine if the treatment would impact other academic (e.g., persistence, retention) or psychosocial outcomes (e.g., sense of self-efficacy, sense of belonging, stereotype threat) in STEM courses.

Faculty development centers and equity units can provide programs, guides, and incentives that support instructors with applying elements of AEP. AEP's elements (e.g., formative assessment) can be useful across disciplines, academic departments, universities, and schools committed to narrowing academic achievement gaps, especially for URM. By increasing URM students' success in higher education, the AEP model strives to diversify perspectives on STEM and respond to systemic inequities, such as retention issues. These students' success, which can be enhanced through AEP, can inspire others in their communities to pursue STEM-related majors and careers.

## References

* Allen, R. A., Folkhard, A., Abram, B., & Lancaster, G. A. (2010). Statistics for the biological and environmental sciences: improving service teaching for postgraduates. *Journal of Statistical Education*.

* Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrollment physics class. *Science*, 332(6031), 862-864.

* Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. *Proceedings of the National Academy of Sciences*, 111(23), 8410-8415.

* Hattie, J. (2009). Visible learning: A synthesis of meta-analyses in education. Routledge.

* McGregor, K. K., Langenfeld, N., Van Horne, S., Oleson, J., Anson, M., & Jacobson, W. (2016). The University Experiences of Students with Learning Disabilities. *Learning Disabilities Research and Practice*, 31(2), 90-102.

* Phuong, A. E., Nguyen, J. & Marie, D. (2017). Evaluating an adaptive equity-oriented pedagogy: A study of its impacts in higher education. *The Journal of Effective Teaching*. 17(2), 5-44.

* Schwartz, D. L., Tsang, J. M., & Blair, K. P. (2016). *The ABCs of how we learn: 26 scientifically proven approaches, how they work, and when to use them*. WW Norton & Company.

* Seymour, E., & Hewitt, N. (1997). Talking About Leaving: Why Undergraduates Leave the Sciences. *Westview Press*.
# The Effectiveness of Publicly vs. Privately Assigned Group Leaders Among Learners in Rural Villages in Tanzania

Judith Uchidiuno, Evelyn Yarzebinski, Emilio Vargas-Vite, Ken Koedinger, and Amy Ogan
jio@andrew.cmu.edu, eey2@andrew.cmu.edu, evargasv@andrew.cmu.edu, koedinger@andrew.cmu.edu,
aeo@andrew.cmu.edu
Carnegie Mellon University

**Abstract:** Research studies show that teachers increase the success of education technologies in rural settings by supporting students via technology support, domain-relevant explanations, enforcing discipline, and maintaining student engagement. However, a teacher's presence hinders student collaboration in some cultural contexts, and some students may not have a teacher or knowledgeable adult who can provide this support. We conducted an experiment with K-1 students (N=36) in a rural Tanzanian village, where we trained students to provide technology support for their peers under different experimental conditions. We found that with basic technical training and social awareness of the assigned leaders, students can indeed provide peer support in the absence of a teacher, and additionally enable collaboration. We challenge the popularly held notion that natural leaders will emerge and support students' technology and learning needs without adequate training, and discuss the implications of our findings in the deployment of technologies in similar socio-cultural contexts.

## Introduction

The number of out-of-school children in rural and urban regions of Sub-Saharan Africa is increasing at rates that traditional schooling infrastructures cannot accommodate. Current estimates by UNESCO report that there are over 63 million out-of-school children, with over 50% of those children living in Sub-Saharan Africa (1). Those estimates have been increasing over the last few years despite government commitments to address the issue. Of the children enrolled in school, less than 20% score above minimum proficiency on math and reading assessments (2). Popular initiatives to address this challenge involve deploying educational technologies to students in and out of schools, particularly as hiring the large quantity of necessary qualified teachers is difficult (Buchele & Owusu-Aning, 2007). However, such wide-scale initiatives have mostly been unsuccessful due to a lack of understanding of the local context of the end users, as well as a lack of technical infrastructure and support for these initiatives (Therias, Bird, & Marshall, 2015; Warschauer & Ames, 2010).

Despite the failure of these large-scale, one-size-fits-all initiatives, several studies have shown that when deployed using a bottom-up approach, where the social, cultural, and infrastructural needs of the end users are prioritized, educational technologies can indeed lead to positive learning gains (Hollow & Masperi, 2009; Therias et al., 2015). These studies emphasize that providing support for students, including technical and domain knowledge support, is critical to the success of education technologies in under-resourced communities. Other types of support needed by students include help navigating learning applications, scaffolding their lack of basic digital literacy, and maintaining engagement while learning with technology (Uchidiuno et al., 2018).

A number of these large-scale education technology initiatives have been developed with the expectation that natural leaders will emerge among peers and provide this much needed support, without the need for a knowledgeable adult (Mitra et al., 2005; Papert, 2006). Some prior CSCL research has investigated the emergence of leaders among collaborating groups (Sun, Jackson, Burns, & Anderson, 2017), including peer-nomination and self-perceived leadership qualities (Xie, Sun, & Lu, 2015). In our study, we observed groups of K-1 students in a rural village in Tanzania using an Android tablet-based literacy and numeracy curriculum without the support of a teacher. As our own recent work has indicated that emergent leadership is rare in this context, in this study we build on prior work by exploring the effects of assigning leaders within a group of learners, under two conditions that differentially bestow authority on these leaders: a publicly assigned group leader or a privately assigned group leader (in which the leader is assigned, but not publicized to the rest of the group). Based on scores from an administered pretest, we trained selected leaders on how to navigate the tablet application and perform basic troubleshooting. We investigate students' help-seeking and help-giving behaviors in these different conditions and provide insights on how to better foster peer support for educational technology in the absence of a teacher or knowledgeable adult.

## Literature review
The classroom is an organized system of social interactions with institutional and socio-cultural norms influencing students and teachers behaviors (Ädel, 2011). These norms influence help-seeking and help-giving behaviors, and may affect the efficacy of educational interventions if unaccounted for. For example, in certain contexts, teachers encourage children to collaborate freely with one another (Halloluwa, Vyas, Usoof, & Hewagamage, 2018; Mann, Hinrichs, Read, & Quigley, 2016), while in others, collaboration in the presence of a teacher is regarded as cheating (Uchidiuno et al., 2018). Peers have been found to support each other by engaging in discourse that can bring about positive learning gains, including “modelling, assisting, directing, tutoring, negotiating, affirming, and contradicting each other” (Stone & Christie, 1996). Without training in meaningful help-giving behaviors, students are likely to provide just the answers to their peers’ questions rather than elaborated, domain knowledge explanations (Roscoe & Chi, 2008). The quality of this discourse depends on students’ prior knowledge and self-efficacy, as well as their personal relationships with one another (Parr & Townsend, 2002). Prior studies also provide evidence that the quality of help peers provide for one another depends on the closeness of their friendships (Graesser & Person, 1994; Webb & Mastergeorge, 2003).

The support that students offer one another becomes even more important when educational technologies are deployed in unsupervised settings, where children have to rely on each other to navigate and learn from such systems. Some research studies have reported positive outcomes from such scenarios, reporting an increase in student curiosity (Mitra & Dangwal, 2010), and even marginal learning gains (Breazeal, Morris, Gottwald, Galyean, & Wolf, 2016), but do not provide any insights on the social interactions that lead to such outcomes. For instance, Kumar et al. provide rich insights from conducting an unsupervised learning research study using a mobile phone in rural parts of India (Kumar et al., 2010). They found that beyond infrastructural issues such as inconsistent electricity, factors such as gender, caste, and time of day significantly affected mobile usage of learning content. Children learned on average three new words per week for 9 weeks. The authors report that children learned to support their peers from watching the experimenters navigate the devices but provide few details on how this transfer of knowledge occurred.

While these studies expected (and in some cases did observe) natural leaders to emerge amongst the group, other research studies have provided insights on how peers interact when they are explicitly assigned to offer support in learning contexts. Fantuzzo et al. examined the efficacy of reciprocal tutoring, where students take turns providing support for their peers, and fixed-role tutor assignments (Fantuzzo, King, & Heller, 1992). Reciprocal tutoring was found to have more benefits over fixed-role assignments, however, is impractical for contexts such as our target demographic where most students are learning using a technology medium that is foreign to them. Group leaders tend to provide domain knowledge support in primarily two ways - providing answers only or stating facts (knowledge-telling) and giving reflective and elaborated explanations (knowledge-building). These types of support differ not only in the quality of feedback a help-seeker receives, but also benefit the help-giver differently - peer-tutors who provide knowledge-building support score higher on posttests as they organize their knowledge and monitor their understanding better (Roscoe & Chi, 2007). When students are not trained to provide constructive and knowledge-building feedback to their peers, they may likely provide knowledge-telling feedback, but studies show that deeper questions from help-seekers can elicit more elaborate, knowledge-building responses (Duran & Monereo, 2005).

Research such as (Kumar et al., 2010) has provided limited evidence that when children are equipped with technical skills, they can naturally emerge as leaders and provide support for their peers, without having an official assignment as a group leader. Other studies show that behaviors differ when leaders are assigned rather than emergent (Wickham & Walther, 2009) We investigate whether this finding applies in a different cultural context by equipping children with basic technical knowledge, and asking them to support groups where they are officially assigned as the group leader. We deployed an Android tablet-based learning system with *unsupervised* groups consisting of a child with domain knowledge and technical competency, their closest friends, and randomly assigned peers. We vary the experiment conditions by publicly assigning a leader in some groups, and not in others. We use these observations to answer the following research questions:

* To what extent do knowledgeable children take on the role of a leader within a group of learners, when either privately or publicly assigned the authority to do so?

* What kinds of support (knowledge-telling vs knowledge-building) do assigned leaders offer across social factors such as gender and close friendships in this cultural context?

## Methodology

This study was conducted in partnership with a Swahili-speaking, rural village in a Northwestern region of Tanzania. Members of our research team have conducted research in the region over the last three years. The village was limited in physical and technical infrastructure with inadequate power or clean water. Three mobile
network providers serviced the village. Throughout our stay (and in previous visits), we observed families with mostly feature phones without internet connectivity, however, a few school teachers owned basic Android smartphones. We did not observe any tablets in the village homes, schools, or public settings throughout our stay. Consent (in Swahili) and approvals were obtained from students, parents/guardians, the school administration, and the village council with help from a native Swahili speaker.

The Swahili learning application focused on the following areas: literacy (letter and phonemic awareness, writing, stories curated from the African Storybook Project - http://www.africanstorybook.org/), and math (number identification, number writing, addition and subtraction). Most system interactions involved tapping on the screen, although some require tracing, writing, or speaking to engage, with a speech recognition engine validated in various African contexts (Mills-Tettey et al., 2009), and video tutorials with continuous finger placement scaffolds to support children's' digital literacy. The Swahili audio prompts and instruction were recorded by a Kenyan Professor of Swahili, who has taught Swahili from kindergarten to university levels. The Swahili video application tutorials were recorded by a Tanzanian instructor of Swahili, who grew up in a region about 10 hours away from the village where the study was conducted. All communications with the children were translated by a native Swahili speaker who lived in the same village and was well-known to the children. More details about the learning application can be found in (Uchidiuno et al., 2018).

To ensure that we observed a sufficiently broad sample across experimental conditions, we structured our observations into 24 discrete sessions during which six groups of six children each engaged with the tablet software. Each observation lasted about 1 hour. In these sessions we observed a total of 36 unique children; this included 18 girls and 18 boys from grades K-1, ages 5 to 10 depending on year of entering school. Before the sessions began, we gave all the children a paper pre-test covering letter and number recognition, letter and number writing, simple word problems, and listening comprehension. These pre-tests were administered by Swahili speakers who read the questions one by one to each child, and recorded their responses. We noted down the students who scored the highest, and distributed them equally among the groups. Following the pretest, we ran a baseline session with each group, allowing students' natural interaction patterns to emerge. We selected six children who performed well on the pre-test and quickly learned to navigate the tablet application without adult assistance. We selected the highest-scoring children as leaders just in case their peers needed domain knowledge support in addition to technical support, and assigned them as leaders for each group in the experiment scenarios. We started forming the groups by asking each leader to select two of their best friends from the class - each group was comprised of a leader, the leader's selection of their two best friends, and a random assignment of three other children from the class, balancing each group by gender. After forming groups, we informed the leaders privately that they performed the best in the pre-test and with navigating the system, and were now responsible for supporting their group. We conducted a training session with all six group leaders to reinforce practices of navigating each application without assistance, as well as performing basic troubleshooting tasks e.g. helping a peer return to the learning application if they exit accidentally. After the training, all leaders could navigate the tablet and application without assistance.

<table><caption>Table 1: Group Information with Summary of Pre-Test Scores, the leader's Score, and Experiment Condition</caption><thead><tr><td>Group #</td><td>Age Range</td><td>Score Mean</td><td>Score SD</td><td>Score Range</td><td>Leader's Score</td><td>Leader Condition</td></tr></thead><tbody><tr><td>1</td><td>5-10</td><td>34%</td><td>23%</td><td>8-72%</td><td>72%</td><td>Privately Assigned Leader</td></tr><tr><td>2</td><td>5-7</td><td>46%</td><td>14%</td><td>42-60%</td><td>60%</td><td>Privately Assigned Leader</td></tr><tr><td>3</td><td>5-7</td><td>46%</td><td>12%</td><td>32-62%</td><td>62%</td><td>Privately Assigned Leader</td></tr><tr><td>4</td><td>6-9</td><td>72%</td><td>14%</td><td>52-90%</td><td>90%</td><td>Publicly Assigned Leader</td></tr><tr><td>5</td><td>6-8</td><td>79%</td><td>11%</td><td>65-93%</td><td>90%</td><td>Publicly Assigned Leader</td></tr><tr><td>6</td><td>6-8</td><td>71%</td><td>13%</td><td>55-88%</td><td>88%</td><td>Publicly Assigned Leader</td></tr></tbody></table>

Before the start of the sessions, we reminded the leader of their job to help their peers, as no adults would be around. For 3 groups, we shared this privately with the leader so other group members did not know the group had a leader (privately assigned leader condition), however, in the other 3 groups, we made this announcement publicly, informing the group that the leader would answer all questions they had (publicly assigned leader condition). All groups completed 4 sessions each - 1 initial baseline session without a leader assigned and 3 sessions with an assigned leader. Following the baseline, we conducted two periods of classroom observation to better contextualize each leader's natural peer interactions. leaders are summarized below as "L#"; the number corresponds with their group:

* L1: 6 y/o girl - Quiet. Answers all questions correctly, does not interact much with her peers.
* L2: 7 y/o boy - Energetic, sometimes disruptive. Tries to answer all questions asked by teacher before others can, so the teacher sits him in the back, and never calls on him to answer questions. Once when the teacher left, he ran to the front and pretended to teach the classroom.

* L3: 6 y/o girl - Usually answers correctly when called on. Observed mouthing an answer to a boy called to answer question when he did not know the answer. Usually sits with her friends in the front, who all wait for her to write an answer on a worksheet, and then then copy her for in-class activities.

* L4: 9 y/o boy - Quiet in class. Does not speak except when directly asked a question, does not disrupt.

* L5: 8 y/o girl - Quiet but engages in occasional banter with friends even when the teacher is present.

* L6: 7 y/o girl - Quiet but also engages in occasional banter with her friends.

No researcher or adult was present in the room with the students during the study sessions. Instead, these unsupervised sessions were video recorded from multiple angles to capture natural student interactions. The team analyzed the data from all group sessions, recording qualitative observations of the student interactions with the tablet and each other. Next, the team reviewed the videos, notes, and observation logs and identified emergent themes related to the types of help-seeking and help-giving behaviors surrounding each group leader following a grounded theory approach (Corbin & Strauss, 2008). Six themes of student behaviors emerged: 1) student distracting the group, 2) leader addresses student distraction, 3) student asks leader for help, 4) student asks leader to help a peer, 5) student (not leader) helps another non-adjacent student, and 6) leader helps a non-adjacent peer. Next, we split the sessions equally among 2 team members for categorization according to the 6 themes. A third team member reviewed and coded all 24 sessions to validate the categories created by the other team members, breaking each session into 30 second intervals and adopting a partial-interval recording method (Hintze, Volpe, & Shapiro, 2002). Finally, we triangulated all observations related to these themes with logs captured by researchers in the field, debrief recordings, follow-up interviews with teachers, and photographs captured on site to ensure that all evidence were mutually supportive. After all sessions were categorized, all members of the team each reviewed the findings for all 24 sessions, discussed all areas of disagreement, and re-categorized findings as agreed upon by the entire team. We worked with native Swahili speakers from Tanzania to help translate interactions, as well as provide insights on the cultural underpinnings of those interactions. This study design and data analysis methodology has been used and validated in previous CSCL research studies e.g. (Pauw et al., 2015).

# Findings

## Baseline student behavior

As soon as the sessions started and the adults left the room, the children smiled at each other, looked around briefly, and then started interacting without prompting. These interactions included sharing new activity types with their adjacent peers, sharing a funny activity, celebrating their accomplishments, singing nursery rhymes together, and even repeating spoken tablet instructions in unison. We observed lots of play (and real) fighting in the room as they encroached on each other's tablets, and children intentionally distracting each other from staying on task. If an adult re-entered the room for any reason (forgot an item, child in the session called for their help, end of session, etc), the children stopped all interactions and focused on their tablets.

Without an assigned leader, students generally provided help when all of the following conditions were true: 1) they noticed a peer struggling or the peer asked them for help, 2) the peer is sitting adjacent to them, 3) they are not engrossed in their own work, and 4) they knew how to help (vs trying to find a solution to a new problem). In general, their behavior mirrored the classroom observations - quiet students kept to themselves even when other children tried to interact with them uninvited, and talkative students were the most vocal (and disruptive) in their sessions. The students connected the learning application to what they learned in the classrooms - on several occasions, we observed children bringing out their notebooks from their bags to double check answers before they selected answers on the tablet. They primarily communicated with their adjacent peers, and those whose screens they could easily see. Across all six groups, we observed only two instances where a student helped a non-adjacent peer. Figure 1 shows the student seating arrangement in the classroom. Students sat in a row, all facing the same direction, allowing them to interact with their peers and see their work. There were no observable differences in how girls interacted with other girls vs boys (and vice versa), although friends seemed more likely to share funny stories and new activities with each other.

In all interactions observed under this condition, students provided knowledge-telling support to their peers either voluntarily or when solicited; they either selected the answers for their adjacent peers or told them what answer to tap without any elaboration. We observed one instance of student collaboration in this condition
(and another in the publicly assigned leader condition) when knowledge-telling was insufficient. In one interaction with L5, an adjacent peer was struggling with a tracing task. After she asked L5 for help, L5 said “andika” [Swahili for “write”]. The girl followed L5’s instruction, but her answer was still not accepted because she wrote beside (rather than over) the trace outline. Then L5 reached in to help but also wrote beside the outline, resulting in a rejected response. They looked for a teacher, but no one was close by, so they tried different strategies until they learn the correct way to trace on the tablet.

<figure><img src="image_198.png" /><figcaption>Figure 1. Typical seating arrangement in experiment sessions.</figcaption></figure>

## Student behavior in privately assigned leader condition

Students in this condition behaved similarly to the baseline condition. L1 remained quiet throughout her sessions, only interacting with adjacent peers, and volunteered help when she noticed a struggling adjacent peer. Since she was not revealed as the group leader, non-adjacent students did not ask her for help. When engrossed in her tablet, she ignored most requests for help even from adjacent students. This observation was consistent in all of L3’s sessions as well. She freely chatted with adjacent peers, but also ignored their requests for help when engrossed in her own work. On the occasions where L1 and L3 helped their adjacent peers, *they showed very little persistence* and routinely abandoned them if their attempt was not a quick fix on the first or second try.

There were clear negative consequences when a student needed help but didn't receive it. In Group 1, a boy struggled with basic application navigation, and could hardly complete a single learning activity in all three sessions despite sitting next to L1 in one session. Also in a session with L1, a student gave up and left the session after trying for 30 minutes to start a learning activity (including asking her for help twice), while another started to wipe tears from her eyes after repeated unsuccessful attempts at getting herself back on task. Figure 2 shows a boy in Group 3 who needed help, and his reaction when he finally learns to navigate the application after receiving help from the girl right beside him (not L3).

L2's behavior as a privately assigned leader was quite different from L1 and L3. In the first session, he interacted mostly with one of his best friends similar to his behavior in the baseline condition. After privately reminding him of his responsibilities prior to the next session, he spent most of his time walking around the room and helping every struggling student (similar to how his teacher behaved in his classroom), without paying much attention to his own tablet. He was only asked for help 4 times from his adjacent friends throughout the session, but volunteered help to 18 additional non-adjacent peers. When his peers noticed that he was providing this help, they called on him to help other struggling peers, and he continued to hover around their tablets to review their progress. If a peer paused briefly, he ran over to solve the problem even if they did not ask for help. Some did not welcome the constant, unsolicited help - near the end of one session where L2 “helped” a girl constantly, she shoved him away when he walked over to her. His monitoring also quickly turned into behavior enforcement. One of the activities in the learning application involved a racecar that made screeching sounds. Students quickly figured out how to exploit this sound and enjoyed producing an almost-constant screech, therefore distracting the whole group. In the first session, L2 and his adjacent peer engaged in this distraction, but by the second session, he monitored and turned off students' tablets if they tried the exploit. His enforcement tactics became progressively stronger in each session, and he started hitting kids on the head and addressing the whole group sternly when he thought they were not paying attention. Although some children had problems with his style of help-giving, overall it had positive effects on the group. Most children stayed on task, students started calling him for help for themselves and for others, and others started to emulate him by standing up and walking around to help as well. Figure 3 shows L2 addressing a student who was distracting the group. On the average, L1 and L3 ignored 79% of help requests overall (and all requests from their friends), while L2 ignored only 33% of his requests for help (13% from his friends).

<figure><img src="image_199.png" /><figcaption>Figure 2 (L-R). Struggling boy; peer notices and helps; excitement when he gets questions correct.</figcaption></figure>
<figure><img src="image_200.png" /><figcaption>Figure 3. L2 addressing a student who was distracting the entire group.</figcaption></figure>

## Student behavior in publicly assigned condition

Publicly assigned leaders had fewer help-giving restrictions; they generally provided help either if they were asked or if they noticed a peer struggling. Unlike L2, none of the leaders in this condition (L4, L5, L6) hovered around the entire group. Despite their working steadily and constantly on their tablets, they responded to most requests for help. The other children in the groups chatted with their adjacent peers about new activities and funny stories, and even attempted to help each other first before calling on the leader. On the average, leaders in this condition responded to requests for help 75% of the time (77% from their friends). They were able to discern when they were called for non-help requests such as discovering new application activities. L5 was even able to tell when she placed someone in an application that was too easy for them. She restarted a boy's application, and placed him in an alphabet song. Immediately, she exited the application, and switched it to a more difficult story for him to read. Although the leaders were called upon frequently in the first session, the help requests decreased in subsequent sessions, suggesting that students became more proficient at navigating the tablets without the leader. By the 3rd session, the publicly assigned leaders got the same number of requests from the whole group as the privately assigned leaders from their adjacent peers. Figure 4 shows a chart of the average number of times the leaders were called upon as the sessions progressed per condition.

The biggest difference between the help offered by publicly vs privately assigned leaders was the degree of leader persistence. We did not observe any cases where the leader abandoned a student even when they could not figure out a solution. One student in Group 4 accidentally exited the application but could not open a new instance. She asked L4 for help, but he could not figure out a solution after several attempts. He started to walk back to his seat but changed his mind, knelt beside her, and kept trying. After much trial and error, she found the application switcher button and selected the learning application. Instead of returning to his seat after this success, he reproduced the problem, so he could practice the solution she discovered. Following that, he returned to his seat. Similarly, L5 spent over 5 minutes trying to help a girl with a technical issue with constant application crashes. After several unsuccessful tries, L5 left the session to seek a help. The application had to be reinstalled to fix the error.

Finally, the leaders helped to maintain engagement with the learning applications. All students enjoyed exploiting the racecar game's screeches. Unfortunately, the noise distracted everyone in the group and increased off-task behavior. It caused students who were trying hard to focus to start tapping the tablet without regard for the current activity, or scribbling furiously in writing activities. Unlike the privately assigned leader condition, the publicly assigned leaders initially joined the fun, but quickly worked to establish order in the groups. L4 warned the students to stop misusing the application; L5 and L6 resorted to turning off students' tablets if it distracted their group, and in one incident, L6 even seized a peer's tablet. Other members of the group began to call for the leader when someone was bothering them individually, and the leaders routinely intervened. This was an unexpected but welcome benefit of having publicly assigned leaders in the group because these interventions minimized the time that students were off task compared to the other leadership conditions.

<figure><img src="image_201.png" /><figcaption>Figure 4. Average Help-Seeking Requests to in Privately vs Publicly Assigned Leader Conditions.</figcaption></figure>

## Discussion and conclusions

Unassisted learning educational settings are becoming more popular due to the increasing demand for education where traditional schooling facilities and instructors are scarce - our research contributes to the body of
knowledge that maximizes their success. The expectation that children, without formal training or assignment, will provide support in unassisted learning situations seems intuitive given their innate curiosity, and previous findings from other cultural contexts. However, our research provides evidence that socio-cultural factors do not only affect the ways that children engage with educational technologies, but also in their willingness and ability to support one another, as well as the quality of help they provide. This is not to say that natural leaders in this cultural context do not exist - a student like L2 demonstrated his enjoyment for teaching and proctoring both in the classroom, and in the experiment sessions. However, in a culture like the one we studied where teachers and adults are likely the singular source of authority, this kind of behavior may be punished similar to seating L2 at the back of the class, and refusing to call on him or answer any of his questions. We also provide evidence that children who show natural helping tendencies in the classroom (like L3 whose classmates depend on her for answers), even when trained, do not automatically emerge as leaders for their peers at the expense of their own interest. Our research shows that in this cultural context (and similar), help-giving did not vary by friendships or even gender, and students provided the best quality of support when they were trained to help **and** when there was a social expectation of them providing such support. Leaders did not just naturally emerge similar to other studies exploring unassisted learning situations e.g. (Kumar et al., 2010) and (Mitra et al., 2005). Rather than expecting children to naturally provide support for their peers, such unassisted learning programs should train and educate children on who, how, and when to ask for support to maximize their success.

Publicly assigning the role of a leader conferred significant benefits to all students (leaders included). Group members, regardless of their proximity to the group leader received much needed help with the learning application, and we observed no incidents of children frustrated, abandoning the sessions, or going multiple sessions without engaging with the learning application (unlike the privately assigned leader condition). Leaders showed that they could regulate their learning and that of their peers, deciphering when a student really needed help vs when they could figure it out for themselves, and leaders became more persistent and provided better support by practicing more nuanced ways to bail their peers out of trouble, improving the collective knowledge of the entire group. These results are in accordance to previous findings that show that students' ability to influence and adjust their own cognitive and help-giving behaviors and that of others is optimal for learning and working together (Zimmerman & Schunk, 2011). Another unexpected but very positive effect of publicly assigning leaders was how much the leader helped to control disruptions to the group. In our experiment sessions, when one student started causing disruptions, the whole group (including those who were trying hard to stay focused) got distracted as well. Having students who automatically took the responsibility upon themselves to manage disruptions was important for maintaining student engagement. Finally, our research provides evidence that in a cultural context where collaboration is often discouraged by teachers e.g. (Uchidiuno et al., 2018), carefully designing unassisted learning environments can foster collaborative behaviors even for simple writing tasks. Further research is required to investigate how these behaviors change as the tasks become more complex, or when the students are explicitly engaged in collaborative tasks.

Our study is limited in several ways - the lack of in-depth explanations provided by students may be a factor of their age and developmental level, as well as the domain area not requiring much explanation (early math, letter identification, and early reading). Future studies should investigate these help-seeking and help-giving behaviors among older students in domain areas that require more explanations such as math and science targeted at later grades. Also, the only training children received was related to technology support. They were not trained on how to properly give domain knowledge feedback to their peers. Although, our study provides evidence that untrained students do not provide knowledge-building naturally, further investigation is required to determine the help-giving behaviors of students who are trained to give proper feedback. Finally, in our study we qualitatively observed a small group of high performing leaders across multiple sessions - these insights may be different in a larger scale study, or with students with lower achievement.

## Endnotes

(1) http://uis.unesco.org/sites/default/files/documents/fs48-one-five-children-adolescents-youth-out-school-2018-en.pdf

(2) https://www.brookings.edu/blog/africa-in-focus/2017/10/06/figures-of-the-week-africa-education-world-development-report-2018/

## References

Ädel, A. (2011). Rapport building in student group work. *Journal of Pragmatics*, *43*(12), 2932–2947.

Breazeal, C., Morris, R., Gottwald, S., Galyean, T., & Wolf, M. (2016). Mobile devices for early literacy intervention and research with global reach. In *Proceedings of the Third (2016) ACM Conference on Learning@ Scale* (pp. 11–20). ACM.

Buchele, S. F., & Owusu-Aning, R. (2007). The one laptop per child (OLPC) project and its applicability to Ghana. In
*Proceedings of the 2007 international conference on adaptive science and technology* (pp. 113–118).

Corbin, J., & Strauss, A. (2008). Basics of qualitative research: Techniques and procedures for developing grounded theory. *Thousand Oaks*.

Duran, D., & Monereo, C. (2005). Styles and sequences of cooperative interaction in fixed and reciprocal peer tutoring. *Learning and Instruction, 15*(3), 179–199.

Fantuzzo, J. W., King, J. A., & Heller, L. R. (1992). Effects of reciprocal peer tutoring on mathematics and school adjustment: A component analysis. *Journal of Educational Psychology, 84*(3), 331.

Graesser, A. C., & Person, N. K. (1994). Question asking during tutoring. *American Educational Research Journal, 31*(1), 104–137.

Halloluwa, T., Vyas, D., Usoof, H., & Hewagamage, K. P. (2018). Gamification for development: a case of collaborative learning in Sri Lankan primary schools. *Personal and Ubiquitous Computing, 22*(2), 391–407.

Hintze, J. M., Volpe, R. J., & Shapiro, E. S. (2002). Best practices in the systematic direct observation of student behavior. *Best Practices in School Psychology, 4*, 993–1006.

Hollow, D., & Masperi, P. (2009). An evaluation of the use of ICT within primary education in Malawi. In *ICTD* (pp. 27–34). Citeseer.

Kumar, A., Tewari, A., Shroff, G., Chittamuru, D., Kam, M., & Canny, J. (2010). An exploratory study of unsupervised mobile learning in rural India. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (pp. 743–752). ACM.

Mann, A.-M., Hinrichs, U., Read, J. C., & Quigley, A. (2016). Facilitator, functionary, friend or foe?: Studying the role of iPads within learning activities across a school year. In *Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems* (pp. 1833–1845). ACM.

Mills-Tettey, G. A., Mostow, J., Dias, M. B., Sweet, T. M., Belousov, S. M., Dias, M. F., & Gong, H. (2009). Improving child literacy in Africa: experiments with an automated reading tutor. In *Information and Communication Technologies and Development (ICTD), 2009 International Conference on* (pp. 129–138). IEEE.

Mitra, S., & Dangwal, R. (2010). Limits to self-organising systems of learning—the Kalikuppam experiment. *British Journal of Educational Technology, 41*(5), 672–688.

Mitra, S., Dangwal, R., Chatterjee, S., Jha, S., Bisht, R. S., & Kapur, P. (2005). Acquisition of computing literacy on shared public computers: Children and the" hole in the wall". *Australasian Journal of Educational Technology, 21*(3).

Papert, S. (2006). Digital development: How the $100 laptop could change education. *USINFO Webchat, via OLPC Talks*.

Parr, J. M., & Townsend, M. A. (2002). Environments, processes, and mechanisms in peer learning. *International Journal of Educational Research, 37*(5), 403–423.

Pauw, D., Clegg, T., Ahn, J., Bonsignore, E., Yip, J. C., & Uchidiuno, J. (2015). Navigating connected inquiry learning with ScienceKit. International Society of the Learning Sciences, Inc.[ISLS].

Roscoe, R. D., & Chi, M. T. (2007). Understanding tutor learning: Knowledge-building and knowledge-telling in peer tutors’ explanations and questions. *Review of Educational Research, 77*(4), 534–574.

Roscoe, R. D., & Chi, M. T. (2008). Tutor learning: The role of explaining and responding to questions. *Instructional Science, 36*(4), 321–350.

Stone, S. J., & Christie, J. F. (1996). Collaborative literacy learning during sociodramatic play in a multiage (K-2) primary classroom. *Journal of Research in Childhood Education, 10*(2), 123–133.

Sun, J., Jackson, J., Burns, M., & Anderson, R. C. (2017). Children’s Emergent Leadership and Relational Thinking in Collaborative Learning. Philadelphia, PA: International Society of the Learning Sciences.

Therias, E., Bird, J., & Marshall, P. (2015). Más Tecnologia, Más Cambio?: Investigating an Educational Technology Project in Rural Peru. In *Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems* (pp. 447–456). ACM.

Uchidiuno, J., Yarzebinski, E., Madaio, M., Maheshwari, N., Koedinger, K., & Ogan, A. (2018). Designing Appropriate Learning Technologies for School vs Home Settings in Tanzanian Rural Villages. In *Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies* (p. 9). ACM.

Warschauer, M., & Ames, M. (2010). Can One Laptop per Child save the world’s poor? *Journal of International Affairs, 33*–51.

Webb, N. M., & Mastergeorge, A. (2003). Promoting effective helping behavior in peer-directed groups. *International Journal of Educational Research, 39*(1–2), 73–97.

Wickham, K. R., & Walther, J. B. (2009). Perceived Behaviors of Emergent and Assigned Leaders in Virtual Groups.

Xie, K., Sun, Z., & Lu, L. (2015). The Role of Time, Engagement, and Self-Perceived Leadership on Peer-Nominated Emergent Leadership in Small Group Online Collaborative Learning. International Society of the Learning Sciences, Inc.[ISLS].

Zimmerman, B. J., & Schunk, D. H. (2011). Self-regulated learning and performance. *Handbook of Self-Regulation of Learning and Performance, 1*–12.
# Taking the Patch Perspective: A Comparative Analysis of a Patch Based Participatory Simulation

Lauren Vogelstein, Vanderbilt University, lauren.e.vogelstein@vanderbilt.edu
Corey Brady, Vanderbilt University, corey.brady@vanderbilt.edu

**Abstract:** In this paper we explore the implementations of an activity designed to engage students in an embodied simulation of an artistically expressive Agent-based modeling environment. We compare how two facilitators implemented the same activity, analyzing how they co-developed differing substrates with their students. We argue that instructors facilitated different embodied agent-based perspectives, which led students to develop considerably different conceptualizations of how these agents interact and how to code in this environment.

## Introduction

Agent-based modeling (ABM) offers a powerful representational infrastructure for simulating and understanding emergent phenomena and complex systems. However, it also provides an expressive medium for creating computational art of various kinds. In the constructionist tradition, a single movable agent (turtle) has been used in environments such as TurtleArt (Bontá, Papert, & Silverman, 2010) and more recently Scratch (Resnick et al, 2009) as a means of exploring artistic expressivity focused on a single agent. In this paper, we extend this line of work into a multi-agent environment, NetLogo (Wilensky, 1999), focusing initially on the expressive visual potential of a large grid of immobile agents (patches), which can be thought of in terms of the panels of a quilt or the pixels of an image. As computational agents, patches have Cartesian coordinates, can change color, and can hold variables. In using the NetLogo patch grid as a medium for visual art, we have found students encounter conceptual challenges and opportunities in learning to use the environment expressively. In some senses, these challenges mirror those found with artistic uses of single and multi-turtle programming, where physical, embodied simulation have provided critical entry points for learners (e.g., "playing turtle" (Papert, 1980) for single turtle concepts, and engaging in "star people" activities (Resnick & Wilensky, 1998) for multiple turtle concepts). We hypothesized, however, that learning to conceptualize patches would require different mappings from social structures to computational phenomena. Engaging in design-based research to explore this area, we were interested in how a group of learners new to ABM could use coordinated social enactment to build familiarity with multi-agent patch programming as an expressive medium. We developed activities that invited participants to physically embody patches as an assembly of social beings that can each interpret and respond to code. We designed activities to foster understandings of how these agents function computationally and what visual effects they can produce. However, we found that teacher facilitators implemented these activities in ways that promoted different conceptions of patch agents, even when these activities were highly specified in professional development and curricular materials. In our study of two cases of implementation these differences appeared to be consequential, opening up two distinct ways of using patches, both of which were powerful and expressive but in different ways. In particular, we found that taking slightly different embodied perspectives as patches could contribute to the development of considerably different conceptualizations of how these agents interact and how one codes in this environment. Thus in this paper we explore the following research question: How do facilitation choices in these socially embodied simulations shape the image that emerges for participants of how one programs in a multi-agent environment?

## Study overview

In this paper we report on data collected from the Computational-Thinking And Mathematics Play Spaces (CAMPS) project, a design based research project created specifically to look at how computational thinking and mathematics could be explored simultaneously with middle school learners in an expressive, artistic ABM environment. As the acronym suggests, this project took the form of a summer camp. The first design iteration consisted of a one-week (five-day) free summer camp for middle school students, held in a middle school in a southeastern U.S. city. The camp was titled "Code Your Art," and was advertised as a camp involving computer programming and art for rising 7th and 8th graders (although a few rising 6th graders asked to participate and attended). Code Your Art camp consisted of two groups each with 16 students. Each group was co-facilitated by two middle school mathematics teachers; Tracy and Isabel were in room 1 and Kiara and Neil were in room 2. Six researchers and two undergraduate computer science students rotated between the rooms to collect data and provide technical and facilitation support as needed.

Over the course of the five-day camp, students created digital works of art in NetLogo to present and
share on the final day in each class' art gallery. We conceptualized using NetLogo to create digital art by thinking of pixels as computational agents (patches). To do this, we had students import images from the web into NetLogo so that they could use this computational environment to manipulate different parts of the image, changing pixels based on color, location, or both. Since we conceptualized images as being composed of computational agents (pixels) it made sense for us to introduce NetLogo from the patch perspective, having students' first introduction to NetLogo be in the context of what a single pixel/patch would interpret in the context of creating a larger composite image. Thus students' first introduction to NetLogo was not on a computer, it was in a series of participatory simulations we referred to as the Stadium Card Activities. We took inspiration from a sports arena phenomenon in which spectators are given one or more colored squares of paper. When held up in a coordinated manner the entire stadium creates a composite image (Figure 1).

<figure><img src="image_202.png" /><figcaption>Figure 1. Stadium cards in increasing scale from left to right.</figcaption></figure>

Since we were neither in a stadium nor in the digital world of NetLogo, but rather in middle school classrooms, we created our own physical computational environment by tapping into the cultural syntonicity (Papert, 1980) between stadium cards in the arena and patches in NetLogo. We taped out a 2 x 8 grid on the floor (one square unit for each participant/patch) and instead of giving participants a single colored card, like in the arena events, we gave each participant five 15" x 15" colored squares (one each of black, red, orange, green, and blue), each labeled with a number on one side (0, 1, 2, 3, 4, respectively), and attached by a binder ring so that students could flip through all five colored squares. Although the numbers and colors did not correspond to the same color system in NetLogo (e.g. in NetLogo red = 15 not 1), they still allowed participants to respond to code that referred to colors either by name or number, and produced an opportunity to think about modular arithmetic (e.g. what happens when you are asked to increase your color by 7).

In the context of the camp, a series of three Stadium Card Activities were used each day to introduce new computational concepts in NetLogo: patch color, patch location, and turtle movement. For this paper we will focus our analysis on the first activity, which we used to introduce NetLogo on day one of the Camp. This activity focused on patches (with color as their primary property), and basic syntax and computational concepts (e.g. brackets, “ask patches,” if/then, colors named and numbered, loops, and randomness). This activity was co-designed with the instructors who taught the camp, using pilot enactments during four days of professional development to change the original researcher-designed activity. Through these conversations with instructors two consequential changes were made. (1) Someone embodied the Observer. We even dedicated a special location in the room for the Observer to stand and give commands. This created a “Simon Says” feel to the activity that the instructors thought would be something students could easily understand, taping into the ego syntonicity (Papert, 1980) between the agents (patches) as beings that can interpret commands and make decisions and the participants themselves who have these human abilities. The instructors suggested that this would also support students in eventually becoming the Observer themselves. (2) A slide deck with all of the agreed upon NetLogo commands for instructors to read as the Observer, along with a translation into English on each slide (e.g. “ask patches [if pcolor = black [set pcolor red]]” and “Ask all patches, ‘if your color is black set it to red’”). This change allowed instructors to make connections for themselves about NetLogo syntax, such as relating brackets to quotation marks when “talking” to patches as the Observer. By the end of the professional development days, a Google slide deck was created for the instructors to use on the first day of camp. Although both teams of instructors had access to the same professional development and the same slide decks, when they ran this activity during the camp they made some changes. Although kids in both rooms playfully engaged in this social ABM environment through their embodiment of agents within the system, the changes instructors made resulted in significant differences in students’ understandings of how to use NetLogo.

## Theoretical framework

As seen in the two groups during Code Your Art camp, when groups work together, they develop their own micro-cultures that shape their understandings as they learn together (Fine, 1979). This development can be traced through an analysis of co-operative action (Goodwin, 2018). Goodwin describes human interaction as a process in which participants draw upon interactional components from a “substrate” of their shared history and reuse them with modification, transforming these resources in ways afforded by the semiotic system to which
they belong so that they serve emergent interactional intentions. Substrates consist of shared resources for
interaction that change as time unfolds and interactions continue. Goodwin defines a substrate as follows:

It is visibly a form of action that organizes simultaneously 1) the *past* as something relevant to the present by not only incorporating, but transforming, the materials it emerged from; and 2) the *future* that will immediately follow from it by providing a constrained but open-ended framework for subsequent action (Goodwin, p. 32, 2018).

This analytic construct makes shared interactional resources visible as influencing future interaction by emphasizing how substrates evolve over time through interaction. Components of a substrate can include discourse, material tools, gestures, the physical environment, and anything else that can ground human interaction. An analysis of the substrates developed in the enactment of the Stadium Card Activities focuses on the social nature of the design through the collective embodiment of an ABM environment. During the Stadium Card Activities, relational metaphors, students position on the grid, and the order in which code was translated were the three primary components of each room's substrates that differed, constraining participants' respective open-ended possibilities for interacting with NetLogo as their computational understandings began to develop. In analyzing the different pedagogical choices instructors made when enacting this activity, an analysis of these substrates allows us to begin to understand how different choices constrained the possibilities for interaction.

## Methods

In both rooms, the first Stadium Card Activity lasted about 40 minutes, starting with (1) a discussion about stadium cards as real life examples, (2) going through a sequence of commands with an instructor as the Observer, and (3) ending with students taking on the role of the Observer. The Stadium Card Activities were video recorded using four cameras situated around the periphery of the classrooms. Researchers observed this activity and took notes while watching. In both rooms the second author walked in during the activity and participated as well. This paper reports initial findings from our review and analysis of recordings of the first Stadium Card Activity. Using Interaction Analysis (Hall & Stevens, 2016; Jordan & Henderson, 1995) we have developed grounded theoretical categories to describe similarities and differences in the substrates (Goodwin, 2018) present in each room, looking specifically at facilitation and pedagogical choices instructors made and how they related to how students made sense of new computational concepts in this socially embodied ABM environment as they began to explore the expressive potential of the system.

## Analysis

The Stadium Card Activity provided students with an accessible entry into an ABM environment that invited expressive play and allowed for participants to productively grapple with computational concepts. Although all four instructors participated in same pilot enactments of the Stadium Card Activities and had access to the same slide deck, both pairs of instructors made changes to the activities when facilitating them during the Camp. Three primary instructional choices created significant differences in how the first Stadium Card Activity was enacted in each room: (1) how the Observer was described and positioned, (2) where students stood on the grid, and (3) the order in which NetLogo code and English interpretation were translated. These differences affected the development of the substrate for the activity and thus how students interacted with each other, the materials in the room, and the instructors, and ultimately how students began to understand what it meant to use NetLogo as evident in how students took up the role of the Observer at the end of the activity through their participation in this socially, embodied ABM environment as they explored visual effects within this computational system.

## Three instructional choices that created substrates for student learning

### Establishing the Observer-patch relationship

Embodying agents in an ABM environment provided participants with easily understandable relational metaphors for the Observer-patch relationship. Facilitators' descriptions of their role as the Observer cued different relational metaphors, emphasizing different aspects of this computational environment. While facilitators in both rooms began by equating patches to pixels in an image, they positioned the Observer-patch relationship differently. In room 1, Tracy introduced the Observer as having authoritarian rule over patches:

> One of us [instructors] is gonna be the Observer. So right now this is gonna be the Observer box. I as the Observer have the privilege of being able to change you as patches. Ok? So I can speak to you, give you a command, and then you HAVE to follow my command.