interested in patterns that emerged around how students used framing in their annotations and how that framing shaped the subsequent discourse in the threaded annotations.

<table><caption>Table 3: Coding scheme for degrees of expansiveness</caption><thead><tr><td>Code</td><td>Description</td><td>Example</td></tr></thead><tbody><tr><td>U<br>Unframed</td><td>The annotation makes no reference to any aspect of expansive framing.</td><td>"I'm not sure I fully understand the difference between the two."</td></tr><tr><td>0<br>Bounded</td><td>The annotation references at least one aspect of expansive framing, but the annotation is confined to the course and content from the current reading.</td><td>"Further analysis and application of what is learned can occur when knowledge of subject matter exists. How the subject matter is organized and connects to itself matters."</td></tr><tr><td>1<br>Slightly<br>Expansive</td><td>Uses <em>at least 1 aspect</em> of expansive framing; Uses <em>vague descriptions or connections</em> to aspects of expansive framing; AND/OR <em>Does not go beyond</em> the framing established by the original document or annotation it responded to.</td><td>"I think the things that they learn in school will have an effect on their lives outside of school, it just depends on the student and how much of the information they retain. I do agree though that the teachers should try to have an understanding of the students' lives outside of school."</td></tr><tr><td>2<br>Moderately<br>Expansive</td><td>Uses <em>more than 1 aspect</em> of expansive framing; Uses <em>specific examples</em> when connecting to aspects of expansive framing; OR <em>Explicitly orients towards others</em> in a future setting.</td><td>"In high school, I had a Spanish teacher that taught English in Spain for about 7 years. She was fluent in Spanish so I thought I would learn a lot during my time in her class. It was the complete opposite, which is interesting to me because of this highlighted portion. Just because she was an expert on the topic, she could not teach it well."</td></tr><tr><td>3<br>Very<br>Expansive</td><td>Uses <em>more than 2 aspects</em> of expansive framing; Uses <em>specific examples</em> when connecting to aspects of expansive framing; AND <em>Explicitly orients towards others</em> in a future setting.</td><td>"This is a great way to engage students in their learning and understanding of art history. When analyzing a painting, students must first understand how to use contextual clues to describe and explain what the painter might have been communicating through his/her work. Taking a step further, to help students improve their flexibility in transferring knowledge, a teacher might ask, 'What if the figure on the right was looking toward the viewer, rather than away. What might we say about the painting then?'"</td></tr></tbody></table>

## Results

The first major finding was that students did indeed reference aspects of expansive framing in their discourse (see Table 4). Across the course, students generated a total of 459 annotations. Students referenced Participants most often (76% of student annotations), followed by Authorship (71%) and Topics (61%). The types of Participants referenced most often were teachers and students. The types of Topics referenced varied but were generally related to three different categories: their own content area, other courses they were enrolled in, or general education topics and activities. Students did not reference all aspects of expansive framing equally, however. Students did not reference Places (28%) or Accountability (39%) as often. This shows that students have expansively framed their discourse, favoring some aspects over others.

The coding results of degree of expansiveness provides further evidence that students used expansive framing in their discourse. Only nine (2%) of the 459 annotations were unframed and 12 (3%) were bounded to the course and course reading. This means that more than 95% of the annotations were at least slightly expansive and 62% of the annotations were coded as moderately or very expansive.

The Pearson correlation between students average score on expansive framing (degrees of expansiveness) and their written final exam score was 0.56 (p<.05). In other words, students who performed better on the final written exam also, on average, were more expansive in their annotations. The final exam assessed students' ability to analyze educational practices from a theoretical perspective as well as design
towards educational practices grounded in theory. These results were promising but warranted a qualitative exploration to examine how students used expansive framing for generative learning.

<table><caption>Table 4: Coding results of students' use of expansive framing in their annotations</caption><thead><tr><th>Code</th><th>#</th><th>%</th></tr></thead><tbody><tr><td colspan="3">Aspects of expansive framing</td></tr><tr><td>Time Past</td><td>179</td><td>39%</td></tr><tr><td>Time Future</td><td>227</td><td>50%</td></tr><tr><td>Place</td><td>129</td><td>28%</td></tr><tr><td>Topic</td><td>279</td><td>61%</td></tr><tr><td>Participants</td><td>349</td><td>76%</td></tr><tr><td>Accountability</td><td>179</td><td>39%</td></tr><tr><td>Authorship</td><td>324</td><td>71%</td></tr><tr><td colspan="3">Degree of expansiveness</td></tr><tr><td>U (Unframed)</td><td>9</td><td>2%</td></tr><tr><td>0 (Bounded)</td><td>12</td><td>3%</td></tr><tr><td>1 (Slightly expansive)</td><td>152</td><td>33%</td></tr><tr><td>2 (Moderately expansive)</td><td>203</td><td>44%</td></tr><tr><td>3 (Very expansive)</td><td>83</td><td>18%</td></tr><tr><td>Total</td><td>459</td><td>100%</td></tr></tbody></table>

Discourse analysis of the annotations revealed several patterns that appeared both productive (Engle & Conant, 2002) and generative. One such pattern was that students were connecting disciplinary practices in their own content areas to the disciplinary knowledge in the course. Students made connections to their own content areas in all but four of the course readings (19 of 23) and in 92 annotations (20% of all annotations). The example below (see Table 5) shows a threaded conversation between students on an article about assessment.

<caption>Table 5: Threaded annotations connecting content-area disciplinary practices to course content (Excerpt 1)</caption>
<p><strong>Anchor text from Shepard (2000):</strong></p><p>Students also reported that they had to be more honest about their own work as well as being fair with other students, and they had to be prepared to defend their opinions in terms of the evidence. (p. 12)</p><p><strong>1. Leslie, direct annotation (coded as moderately expansive)</strong></p><p>This is directly applicable to art critiques where students must analyze and evaluate the work of their classmates, and give them feedback. They must also receive feedback on their own work and be prepared to respond to this feedback (which requires self-assessment).</p><p><strong>2. Bethany (coded as moderately expansive)</strong></p><p>I feel like this can also apply to students studying a foreign language, if they are speaking to another student that is more skilled at the language than they are, then they can learn more and get assistance if they are struggling on vocabulary.</p>

In this example, the original article was discussing implications of self-assessment practices and Leslie, a visual arts education student, points out a “direct” connection to her own content area and provides an example of a disciplinary practice in her content area. Likewise, Bethany, in her reply to Leslie, sees an application to students in her content area of foreign language learning. Both students expansively framed their learning by recognizing the relevance of prior knowledge from their own content area and made productive generalizations from the content to disciplinary practices in their own content area. Additionally, Bethany’s response suggests she read Leslie’s connection to art (“this can also apply”, emphasis added). Connecting to one’s own relevant contexts and being exposed to other relevant contexts likely promotes productive and generative collaborative learning.

Another pattern was students making connections to general disciplinary contexts and practices of teachers. This pattern was found in all but one of the course readings (22 of 23) and in 156 annotations (34% of all annotations). Table 6 shows a threaded conversation about deficit approaches to education.
<table><caption>Table 6: Threaded annotations connecting general disciplinary practices to course content (Excerpt 2)</caption><thead><tr><th>Text</th><th>Annotation</th></tr></thead><tbody><tr><td>Anchor text from Paris (2012):</td><td>Simply put, the goal of deficit approaches was to eradicate the linguistic, literate, and cultural practices many students of color brought from their homes and communities and to replace them with what were viewed as superior practices. (p. 93)</td></tr><tr><td>1. Rose, direct annotation (coded as moderately expansive)</td><td>In one of my other classes, we are learning about Native American education. One thing that I have learned is that by eradicating student's culture, can make them feel unimportant and unvalued. I think this is a really important idea to keep in mind in a future classroom because students will not feel motivated or engage if they feel like their voice is not important.</td></tr><tr><td>2. Jamie (coded as very expansive)</td><td>I agree with you and in my own experiences, I have seen school trying to do special holidays related to China such as the (Chinese New Year), but after talking about it with my friends from China, they all say that the school got some of the major information incorrectly. For example, there is no such thing as fortune cookie and lion dance in mainland China on Chinese New Year. Although it is great to see that American schools are embracing different cultures from other countries, it is also important to keep in mind as future teachers that we get the facts correct before presenting to the students to avoid conflicts between teacher and student.</td></tr><tr><td>3. Instructor (not coded)</td><td>As I mentioned in the mini-lecture, this is very similar to the difference approaches to education where we try to incorporate other cultures but in a way that "fits into" the dominant culture (which is why there are so many inaccuracies as you mentioned). So how do we value our students ways of doing and knowing and being in our classrooms in more culturally sustaining ways?</td></tr><tr><td>4. Brittney (coded as very expansive)</td><td>In one of my other courses (pluralism in education), we discussed this and similar questions. One of the best solutions we thought of was to bring in people, that held a certain view point or background, to talk to students. Bringing in people who have first hand experience, in a culture for example, allows the students to feel valued if they are from that culture and the others to learn about them in an authentic way.</td></tr><tr><td>5. Marrin (coded as moderately expansive)</td><td>I am in class with Brittney and took a lot away from this topic in class. We also talked about how bringing in an outside resource can help bring a different learning experience than a daily teacher can lead. By simply having a new face leading sometimes allows students to be more engaged but also different methods of teaching is important in some instances. Another aspect we talked about is that a specific person teaching a subject can go a lot further than what the teacher may be. For example, if a white male were to be teaching a lesson about american indians students may not be as interested because he obviously would not be a direct source of their culture, though if an american indian were to come into class and teach a lesson students would value the information he shared much more.</td></tr></tbody></table>

In this annotation thread, Rose connects the content of the reading to another course she is taking and views her learning as relevant to the disciplinary practices of “a future classroom.” Jamie responds by sharing an experience that validates Rose’s prior learning. Jamie concludes her response by explicitly orienting towards herself and others in the class (“as future teachers . . . we”) in sharing an example of how this applies to general disciplinary practices (“get the facts correct” about cultural activities). This type of comment was not uncommon throughout the annotations as students often used the phrase, “we need to” or “as future teachers we should” when writing about future applications of the content for students and teachers. The instructor of the course (the first author) responds to Jamie and connects back to the course content, followed by a question that asks students to imagine how they might apply the content of the article to their own future students. Brittney replies to the instructor’s question by providing an example from another course she is enrolled in. Marrin states she is “in class with Brittney” suggesting that her response is not oriented toward Brittney, but rather towards others in the class. Marrin then reiterates Brittney’s suggestion, validating her contribution, and connects to general disciplinary practices by briefly suggesting that “different methods of teaching” may be helpful. Finally,
Marrin shares an additional example of a disciplinary practice that connects to the previous responses. This set of threaded annotations showed students sharing examples of prior learning that was relevant to the content they were engaging in and generalizing the content to their future disciplinary practices.

A final example (see Table 7) demonstrates a pattern that was less common—found in 18 of 23 course readings and 46 annotations (10%)—but was productive for student's engagement and likely to be generative. Most of the annotations that were coded for accountability were simply students responding to a peer's annotation. However, some of the most productive responses occurred when students requested ideas from their peers. These questions often led to students sharing potential disciplinary practices from their own content area or for general education practices.

<caption>Table 7: Threaded annotations stemming from a student question (Excerpt 3)</caption>

Anchor text from Ito et al. (2013):

Further, when individual competence is assessed based on grades, test scores, and other standardized and summative metrics, one student's success highlights another student's failure. Environments like the HPA, Quest to Learn, or Clarissa's online writing group have a different dynamic (p. 48)

1. *Anna, direct annotation (coded as very expansive)*

How can we create this in a classroom? What can we as teachers do to create an environment where one student's success does not highlight another's failure, because standardized tests and evaluative exams will be inevitable. How can we create a group of students that genuinely enjoys collaborating and wants to do well for the good of the class, and not themselves, when we will undeniably have tension between individual students because of personality difference and difference in interest? **Please respond w ideas: I'm genuinely interested in how we can do this!** (emphasis in original)

2. *Abby (coded as very expansive)*

These are great questions that I'm wondering about myself also! For your first Q "What can we as teachers do to create an environment where one student's success does not highlight another's failure?" One example that popped into my head is how during many classes I took during high school (and even in college) the teacher/prof would state "the highest grade was a 98 and the lowest was a 32" or something like that. I feel like this isn't appropriate to do, especially in a high school classroom where it is VERY easy to find out which student it was that the highest or lowest score belonged to. I feel like announcing the scores out loud kind of pits the students against one another, and highlights failures. Instead of the teacher announcing the scores, they should talk to the struggling students privately and see what needs to be done to bring their grades up. There's no point in discouraging the students by stating how much better another student did compared to them. That's just one example for the first question, but I'd also like to hear more from others! (that goes for the rest of the questions, too)

3. *Brittney (coded as moderately expansive)*

Personally I have found goal setting to be a great way to motivate students without creating a competitive environment. I'm currently in a 1 credit swimming class, and at the beginning of the semester the instructors asked us to write down several goals we wanted to accomplish. I thought this activity helped make the class more personal to all of us and helped us appreciate when we or others accomplished their goals. This task would obviously look very different in a school classroom, but I think there are always ways to apply goal setting to our individual content areas.

In this example, Anna asks a series of questions on how to implement the ideas from the original article to their own potential future educational practices. At the end of her questions Anna emphasizes her request by using bold font to catch the attention of her peers and elicit their responses. Anna is promoting the authorship of her peers and holding them accountable for their ideas by requesting them to share. In Abby's response, she recalls a bad example from a prior situation (highlighting students' failures) and suggests a general alternative practice (talking with struggling students privately). Abby ends her annotation with a request for more responses ("I'd also like to hear more from others!"). Brittney's response shares a prior experience that is relevant (setting goals) to the questions Anna asked and suggests that this practice could be applicable in anyone's content area. Holding students accountable for their own ideas by asking questions, although not used as often as other framing aspects, encouraged generative learning.
## Discussion and implications

This study provided initial evidence that online course readings can be expansively framed, that students can collaboratively take up expansive framing in online annotations, and that expansive framing appears to result in more generative learning. While acknowledging that a course with pre-service teachers may have been particularly conducive, expansive framing via Hypothesis enabled these students to collaborate in productive threaded conversations by connecting content-specific and more general disciplinary practices to the disciplinary knowledge presented in course readings. As Engle and colleagues (2006; 2012) have argued, these connections encourage generative learning which will likely transfer to future educational, personal, and professional contexts. This study showed that expansive framing is possible in online settings. Additionally, the prompt (see Table 1) could easily be modified to function in other similar online courses, especially pre-professional courses where students are working towards specific future contexts.

## References

Day, S. B., & Goldstone, R. L. (2012). The import of knowledge export: Connecting findings and theories of transfer of learning. *Educational Psychologist, 47*(3), 153–176.

Engle, R. A. (2006). Framing interactions to foster generative learning: A situative explanation of transfer in a community of learners classroom. *The Journal of the Learning Sciences, 15*(4), 451–498.

Engle, R. A., & Conant, F. R. (2002). Guiding principles for fostering productive disciplinary engagement: Explaining an emergent argument in a Community of Learners classroom. *Cognition and Instruction, 20*(4), 399–483. http://doi.org/10.1207/S1532690XCI2004_1

Engle, R. A., Lam, D. P., Meyer, X. S., & Nix, S. E. (2012). How does expansive framing promote transfer? Several proposed explanations and a research agenda for investigating them. *Educational Psychologist, 47*(3), 215-231. http://doi.org/10.1080/00461520.2012.695678

Engle, R. A., Nguyen, P. D., & Mendelson, A. (2011). The influence of framing on transfer: Initial evidence from a tutoring experiment. *Instructional Science, 39*(5), 603–628.

Eryilmaz, E., van der Pol, J., Ryan, T., Clark, P. M., & Mary, J. (2013). Enhancing student knowledge acquisition from online learning conversations. *International Journal of Computer-Supported Collaborative Learning, 8*(1), 113-144.

Fasso, W., & Knight, B. A. (2015). Knowledge transfer in community-embedded learning: A case study. *Journal of Education Research, 9*(3), 267–281.

Gao, F. (2013). A case study of using a social annotation tool to support collaboratively learning. *Internet and Higher Education, 17*(1), 76–83. http://doi.org/10.1016/j.iheduc.2012.11.002

Hickey, D. T., & Rehak, A. M. (2013). Wikifolios and participatory assessment for engagement, understanding, and achievement in online courses. *Journal of Educational Multimedia and Hypermedia, 22*, 407–441.

Ito, M., Gutiérrez, K., Livingstone, S., Penuel, B., Rhodes, J., Salor, K., ... Watkins, S. C. (2013). *Connected learning: An agenda for research and design*. Irvine, CA: Digital Media and Learning Research Hub.

Jaber, L. Z., Dini, V., Hammer, D., & Danahy, E. (2018). Targeting disciplinary practices in an online learning environment. *Science Education, 102*(4), 668–692. http://doi.org/10.1002/sce.21340

Lam, D. P., Mendelson, A., Meyer, X. S., & Goldwasser, L. (2014). Learner alignment with expansive framing as a driver of transfer. In J. L. Polman, E. A. Kyza, D. K. O’Neill, I. Tabak, W. R. Penuel, A. S. Jurow, ... L. D’Amico (Eds.), *The International Conference of the Learning Sciences* (pp. 689–696). Boulder, CO: International Society of the Learning Sciences.

Mendelson, A. (2010). Using online forums to scaffold oral participation in foreign language instruction. *L2 Journal, 2*(1), 23–44. Retrieved from https://escholarship.org/uc/item/8xs1r2tq

Novak, E., Razzouk, R., & Johnson, T. E. (2012). The educational use of social annotation tools in higher education: A literature review. *Internet and Higher Education, 15*(1), 39–49.

Paris, D. (2012). Culturally sustaining pedagogy: A needed change in stance, terminology, and practice. *Educational Researcher, 41*(3), 93–97. http://doi.org/10.3102/0013189X12441244

Shepard, L. A. (2000). The role of assessment in a learning culture. *Educational Researcher, 29*(7), 4–14. Retrieved from http://onlinelibrary.wiley.com/doi/10.1002/9780470690048.ch10/summary

Wooffitt, R. (2005). *Conversation analysis and discourse analysis: A comparative and critical introduction*. London, UK: SAGE.

van der Pol, J., Admiraal, W., & Simons, P. R. J. (2006). The affordance of anchored discussion for the collaborative processing of academic texts. *International Journal of Computer-Supported Collaborative Learning, 1*(3), 339–357. http://doi.org/10.1007/s11412-006-9657-6
# Unpacking Collaborative Learning Processes During Hands-on Activities Using Mobile Eye-Trackers

Bertrand Schneider, Harvard University, bertrand_schneider@gse.harvard.edu

**Abstract:** This paper describes a multimodal dataset captured during a collaborative learning activity typical of makerspaces. Participants were introduced to computational thinking concepts using a block-based environment: they had to program a robot to solve a variety of mazes. Mobile eye-trackers, physiological wristbands and motion sensors captured their behavior and social interactions. In this paper, I leverage prior work on joint visual attention (Tomasello, 1995) and analyze the eye-tracking data collected during the study. This paper provides three contributions: 1) I use an emerging methodology to capture joint visual attention in a co-located setting using mobile eye-trackers (Schneider & al., 2018); 2) I then replicate findings showing that levels of joint visual attention are positively correlated with collaboration quality; 3) finally, I present a new measure that captures cycles of collaborative / individual work, which is positively associated with learning gains (but not with collaboration quality). I discuss these results and conclude with implications for capturing students' interactions in co-located spaces using Multimodal Learning Analytics.

## Introduction

In the last decade there has been a growing interest in cultivating skills that are not traditionally taught in traditional school settings. Those skills are often referred to as “21st century skills” (Dede, 2010; e.g., Collaboration, Communication, Creativity, Critical Thinking) because they are deemed essential for jobs that do not yet exist. New learning environments, such as digital fabrication labs and makerspaces, are ideal spaces for their development. They allow students to learn complex concepts in STEM (Science, Technology, Engineering, Mathematics) through hands-on learning and applied projects. Measuring the development of those skills and providing formative assessment, however, remains a challenge (Berland, Baker, Blikstein, 2014), because each student is unique, and the development of those 21st century skills takes different forms depending on interacting factors (e.g. learners’ personalities, prior knowledge, SES background). The CSCL community has long been studying those skills before they gained a renewed attention from researchers and the general public.

For the scope of this paper, I focus on students’ collaboration and communication by leveraging a new field of research called Multimodal Learning Analytics (Blikstein & Worsley, 2016; MMLA) to capture the quality of learners’ interactions. MMLA uses multiple high-frequency sensors to capture users’ behavior and applies data mining techniques to find trends and predictors in large datasets. Joint visual attention has been extensively studied by social and developmental psychologists and has been shown to be critical to many social interactions. Based on prior literature (e.g., Richard & Dale, 2015; Schneider & Pea, 2013), the main hypothesis of this paper is that productive groups exhibit higher levels of joint attention compared to less productive groups. This construct was captured using multiple mobile eye-trackers in co-located spaces (Schneider & al., 2018). More specifically, I designed a hands-on task typical of makerspaces (i.e., learning to program a robot to solve a variety of mazes) and computed measures of joint visual attention. correlated them with three outcomes measures: the quality of their collaboration (coded with a validated rating scheme in the learning sciences), their task performance (i.e., how successful they were) and their learning gains (computed from a pre and post-test). Finally, because collaboration can be a powerful way to support learning, I analyzed the eye-tracking data to find behaviors that were not just related to collaboration quality, but also learning outcomes.

This paper is structured as follows: the first part reviews the literature on dual eye-tracking and the various measures that researchers have developed over the years to capture joint visual attention. The second part describes the study, participants and data collection protocol. The third part discusses the steps to pre-process the data and compute metrics of joint visual attention to correlate them with outcomes of interests. Finally, I discuss our results and conclude with future steps for capturing students’ 21st skills in makerspaces using MMLA.

## Literature review

This section provides a succinct review of foundational work in developmental and social psychology, as well as in Computer-Supported Collaborative Learning (CSCL) and Computer-Supported Collaborative Work (CSCW) where multiple eye-trackers are used to look at participants’ visual alignment.

There are currently three strands of research studying collaborative processes through dual eye-tracking. The first strand uses remote eye-trackers to capture joint visual attention, where users are each looking at a different computer displays (for example through video conferencing). In an early study, Richardson & Dale (2005) explored the coupling between speakers’ and listeners eye movements and its relationship with discourse
comprehension. They found a positive correlation between discourse comprehension and dynamic coupling
between conversants' eye movement. In a subsequent study, they replicated those results for a live conversation
(Richardson, Dale & Kirkham, 2007). In CSCL, researchers have used this methodology to study pair
programming tasks (Jermann, Mullins, Nüssli & Dillenbourg, 2011) and found that collaboration quality was
characterized by higher levels of "gaze cross-recurrence" (i.e., joint visual attention). A second strand of research
has started to study more ecological settings using mobile eye-trackers. Yu and Smith (2013), for example, used
mobile eye-trackers to explore infant cross-situational word learning through eye-hand coordination. In education,
Schneider & al. (2018) studied apprentices in logistics interacting with a tangible user interface. They found that
levels of joint visual attention (as captured by mobile eye-trackers) were correlated with their quality of
collaboration. Additionally, they developed a methodology to capture leadership behaviors from dual eye-tracking
data by identifying who initiated and who responded to an offer of joint visual attention. Imbalances of these
behaviors were negatively correlated with learning gains. Finally, a last strand of research has started to explore
the benefits of novel visualization techniques to improve learning in a collaborative setting, for example by
displaying participants' gaze to each other. This intervention is sometimes called a "gaze awareness tool", "shared
gaze visualization" or "Bidirectional Gaze" (for a review, see D'Angelo & Schneider, 2018). D'Angelo & Begel
(2017) have enhanced remote pairs' speed and success in communicating when resolving a coding problem by
using eye tracking devices to show each participant where their partner is looking on the screen. In education,
Schneider & Pea (2013) found that making the gaze of each partner visible promoted interactions of higher quality
and consequently increased students' learning gains.

In conclusion, there is ample work showing that joint visual attention is a central mechanism by which
group members coordinate their actions and establish a common ground (Clark & Brennan, 1991). Furthermore,
recent research has been leveraging new sensing technology to quantify joint visual attention in dyads of users
(e.g., Jermann, Mullins, Nüssli & Dillenbourg, 2011). While most studies have looked at remote collaborations,
there is some nascent work in co-located settings using mobile eye-trackers (e.g., Yu & Smith, 2013). Ultimately,
however, the goal from a CSCL perspective is to understand how collaborative processes *contribute* to learning.
Joint visual attention (JVA), for example, is a necessary but not sufficient condition for productive social
interactions. This paper is about going beyond capturing JVA and finding more precise indicators of collaborative
learning. This paper builds upon prior findings (e.g., Schneider & al., 2018), replicates results, and provides new
contributions by isolating collaborative learning processes from the eye-tracking data.

## Methods

### Summary of the study

In this study, participants with no prior programming knowledge were given 30 minutes to program a robot to autonomously solve a series of increasingly complex mazes (see Fig. 1 for the setup of the experiment). Two different interventions were developed and used to support collaboration: a visualization of relative verbal contributions of the participants shown in real time and a brief informational explanation delivered verbally summarizing literature findings on the value of collaboration for learning. While dyads completed the activity, a variety of sensors described in 2.2 collected eye gaze, movement, verbal, and electrodermal activity data on participants. Dependent measures were an assessment of the quality of the collaboration, how well the participants coded the robot to perform the assigned task, and learning gains related to computational thinking. The study is described in more detail in Starr, Reilly & Schneider (2018).

### Participants

Participants were drawn from an existing study pool at a university in the northeastern United States. 42 pairs of participants (N=84) were used in the analysis. 62% of participants identified as students, with ages ranging from 19 to 51 years old (mean age = 26.7 years). 60% of participants identified as female. Participants were paid $20 for the 90-minute session and did not know each other prior to the study.

### Experimental design

The study utilized a two-by-two between-subjects design where dyads were assigned to one of four conditions that would receive different interventions. 25% of dyads received neither intervention (Condition #1), 25% received solely the visualization intervention (#2), 25% received solely the informational intervention (#3) while the remaining quarter received both interventions (#4). The speech equity visualization utilized speech collected by the sensors in the experiment to display how much each participant spoke as a proportion of total talk during the activity. Dyads with this intervention saw a tablet display representing this data over the past 30 seconds by presenting colored rectangles that grew to take up more of the screen as relative contribution increased. The informational intervention involved a researcher reading a short passage that reminded dyads that they were
expected to collaborate and invited dyads to think about how they were collaborating during the activity. They were also told that research has found that equity of each partner's speech time is predictive of the quality of collaboration and learning gains. For an analysis of the differences between each experimental condition, please see Starr, Reilly & Schneider (2018).

## Procedure

After taking the pre-survey and calibrating all sensors, participants were shown a short tutorial video that introduced the basics of writing a program in Tinker, a block-based programming language designed for use with the microcontroller of the robot. Participants were then given five minutes to write code that would move the robot forward across a red line roughly two feet directly in front of it. The robot consisted of a microcontroller, two DC motors, and three proximity sensors. Following this tutorial activity, a second tutorial video was shown that highlighted more advanced features of Tinker such as using prewritten functions to turn the robot and using sensor values to trigger conditional statements. Dyads were also given a reference sheet summarizing the content covered in the tutorial video. Dyads then had 30 minutes to write code to navigate a robot through a series of mazes. Once the robot successfully completed a maze twice, a more challenging maze was provided. Dyads did not know the layout of the mazes ahead of time and were encouraged to write code that would allow the robot to solve any simple maze. During this portion of the study, the researcher provided standard hints at 5-minute intervals to all dyads regarding common pitfalls researchers identified in pilot testing of the activity.

## Independent, dependent measures and process data

The quality of the dyad's collaboration and task performance was assessed during the task by the researcher running the session. The quality of collaboration was measured by aggregating the nine scales adapted from Meier, Spada, & Rummel (2007): sustaining mutual understanding, dialogue management, information pooling, reaching consensus, task division, time management, technical coordination, reciprocal interaction, and individual task orientation (refer to Meier, Spada, & Rummel, 2007 for a definition of those terms). Researchers double-coded 20% of the sessions and had a Cronbach's alpha of .65 (75% agreement). Task behaviors evaluated included task performance (how many mazes were completed by the robot in 30 minutes), task understanding (how much major coding concepts were included in the design such as using sensors with appropriate thresholds in conditional statements), and improvement over time (how much a team's understanding of requisite technical skills and conceptual understanding of the task changed over time). The final written code of the dyad's was also assessed to determine theoretically how well it could have performed the assigned task barring technical issues. To assess learning of computational thinking principles, participants individually completed pre- and post-surveys consisting of four questions related to conditional statements, looping, and interpreting the output of given code (adapted from Brennan & Resnick, 2012; Weintrop & Wilenski, 2015). After the activity, participants also self-assessed their collaboration and wrote a brief reflection regarding how their thinking changed over the course of the activity. Table 1 presents a summary of the measures described in this section.

<table><caption>Table 1: Independent, process and dependent measures looked at in this paper (described above)</caption><thead><tr><td>Independent Measures (2x2)</td><td>Process Measures</td><td>Dependent Measures</td></tr></thead><tbody><tr><td>Speech visualization (on/off)</td><td>Eye-tracking data:</td><td>Collaboration quality (9 sub-scores; 1 overall score)</td></tr><tr><td>Verbal intervention (yes/no)</td><td>Individual gaze points on AOIs, Joint Visual Attention</td><td>Task performance: code quality, improvement, # of mazes solved)<br/>Learning gains (computational thinking)</td></tr></tbody></table>

## Dual eye-tracking measures and hypotheses

Prior work has explored multiple ways of capturing joint visual attention and collaborative processes from dual eye-tracking data. In this paper, I first follow a methodology described by Schneider & al. (2018) to compute joint visual attention from mobile eye-trackers and attempt to replicate previous results showing that JVA is associated with collaboration quality. Second, I was inspired by previous results showing that collaborative problem-solving is a cycle between moments of understanding and non-understanding (Miyake, 1986), and that ideal cycles of communication are related to group performance (Tschan, 2002). In this paper, I hypothesize that collaborative learning interactions are characterized by more frequent cycles of individual work and group interactions – which are captured from the eye-tracking dataset. In short, the hypotheses of this paper are as follows:

1. JVA is associated with higher quality of collaboration; more specifically, JVA is associated with participants' ability to sustain mutual understanding (Schneider & Pea, 2013).

2. The number of cycles of individual work (no-JVA) and collaborative interactions (JVA) is positively associated with the three outcome measures (collaboration, task performance, learning gains).
In the next section, I describe the data, preprocessing steps and measures.

## Data collection (multimodal sensors)

Several sensors were used to collect data from both participants in each session. Tobii Pro Glasses 2 eye-tracking glasses were worn by each participant to follow eye gaze relative to a set of fiducial markers placed around the study environment. An Empatica E4 wrist sensor tracked participant electrodermal activity, blood volume pulse, and acceleration. Finally, a Kinect motion sensor was used to track the movement and position of the participants in space. This sensor collects approximately 100 variables related to a person's body joints and skeleton (24 different points with columns for x, y, z coordinates), their facial expressions, and their amount of speech. Typically collected at 30 Hz, this results in roughly 5.4 million observations per individual during a 30-minute session.

This paper focuses on the Tobii eye-tracker and the data it generates. The glasses include multiple cameras (two infrared cameras recording eye movements and one scene camera recording the participant's field of view), an accelerometer, gyroscope, microphone, a wearable recording unit running and associated controller software running on Windows. The Tobii eye-Tracker outputs data of multiple kinds including an audio recording of the session, a video recording from the point of view of the user, the x and y coordinate of the user's eye-gaze relative to its point of view. These glasses sampled at 50 Hz, generating roughly 90,000 observations per person during the main 30-minute activity. No participant reported being bothered by the glasses. Anecdotally, a few participants forgot that they were wearing them and attempted to leave the room without removing the glasses at the end of the study.

<figure><img src="image_33.png" /><figcaption>**Figure 1.** Example of a video frame generated for sanity-checking purposes where a homography was used to remap participants' gaze (shown in blue and green on the right side of the image) onto a ground truth (left side). The white lines represent the points detected from the fiducial markers to do the homography. On the left, the gaze points turn red if there are within a certain radius (e.g., 100px), which signifies some joint visual attention.</figcaption></figure>

## Data preprocessing – Temporal and spatial alignment

### Temporal alignment

In order to clearly mark when transitions between different portions of the study took place across all sensors and recording devices, several fiducial markers with accompanying audio cues were placed in a PowerPoint presentation used by researchers and participants to guide the flow of the study. Whenever specific points in the study were reached, participants would simultaneously see the fiducial marker, hear the sound, and press a button on the EDA bracelet. In this way, all of the sensors on both participants as well as the video recorder would have
some tagged record of the event and therefore a way to synchronize all of the data. The eye-tracking data analyzed in this paper is solely from the main 30-minute portion of the study and was synchronized via these tags.

## Spatial alignment

One challenge of using mobile eye-trackers is that users can freely move - they can walk around, stand, sit, and change the orientation of their head. Their eye gaze is calibrated on the frames provided by the scene camera, which changes in its content depending on where users are looking. This kind of data is significantly more challenging to analyze compared to traditional (i.e. remote) eye-trackers, where the main area of interest is the screen of a computer. Thus, when using a mobile eye-tracker, we need to identify which part of the environment users are looking at. The solution used in this paper is to add fiducial markers to the environment (they look like QR codes on Figure 1). Detecting those markers is relatively easy for computer vision algorithms, and since they each have a unique ID they provide common coordinates across different perspectives. More specifically, a panoramic picture of the workspace (Fig. 1, left side) and the markers detected from the scene camera of the mobile eye-tracker (Fig. 1, right side) were associated to the markers of the workspace (referred to as “ground truth” below). Knowing this common set of points allowed to infer the location of users’ gaze points on a common plane using a homography. The left side of Figure 1 shows the last 5 gaze points for each user (shown as a gaze plot; additionally, the dots turn red if they are within 100 pixels of each other). Finally, for each group a video recording was generated for sanity checking purposes and to make sure that the homography was accurate.

## Results

### Areas of Interest (AOIs)

<figure><img src="image_34.png" /><figcaption>**Figure 2.** AOIs on the ground truth and Gaze points on each AOI. On the left: the distribution of gaze points for one group (42). On the right: the percentage of eye-tracking data for each AOI (y-axis) and for each group (x-axis). “Outside Ground Truth” refers to the gaze points outside the image shown on the left side of Fig. 1.</figcaption></figure>

<figure><img src="image_35.png" /></figure>

Areas of Interests (AOIs) divide the participants view into 7 different regions, at three different height levels. At the lower level, they differentiate between looking at the computer screen where participants wrote code and
looking around it. At the level of the maze, they differentiate gaze points within the maze and outside the maze. Finally, at the level of the wall, they separate the area corresponding to the speech visualization (only relevant to groups having access to it, i.e. condition #2 and #4) from the one around it.

Running Pearson's correlations between the number of gaze points on each AOI at the individual level generated the following results: looking at the maze and code quality ($r(37) = 0.331, p = 0.040$) / learning gains ($r(37) = 0.360, p = 0.025$). Additionally, there were negative correlations between looking at the computer screen and code quality ($r(37) = -0.320, p = 0.047$); looking at the first cheat sheet and sustaining mutual understanding ($r(36) = -0.364, p = 0.025$) / quality of collaboration ($r(36) = -0.323, p = 0.048$); looking at the second cheat sheet and task performance ($r(36) = -0.608, p < 0.001$) / code quality ($r(37) = -0.350, p = 0.029$). In summary, looking at the number of times that individual participants looked at different AOIs seemed to be mostly associated with negative outcomes.

## Cross-recurrence graphs

Before computing measures of joint visual attention, it is recommended to generate cross-recurrence graphs to sanity check the data. A Cross-Recurrence Graph (Jermann, Mullins, Nüssli & Dillenbourg, 2011) is a plot representing the eye-tracking data of the dyad. One axis is the time for one person and the other axis is the time for the other participant. If the two people are looking at the same location at the same time, we plot a black dot along the diagonal. If there is a delay, we plot this point above and below the diagonal. The distance from the diagonal is proportional to the delay. Therefore, by looking at the points on the diagonal we can estimate visual coupling within the pair. Gray dots represent no joint visual attention and white dots represent missing data.

Cross-recurrence graphs provided a visual representation of the groups' attentional alignment. I generated one for each group and used them as a sanity check for the JVA measures (Fig. 3): for example, it confirmed that group 42 had high levels of JVA, which is represented by more black pixels. Group 38 had low levels of JVA which is represented by more white pixels. Color-coded cross-recurrence graphs (i.e., using the colors from Fig. 3) also helped us observe patterns of interaction: groups spent most of their time looking at the computer screen (gold), and the maze (green).

<figure><img src="image_36.png" /><figcaption>**Group 42 (high JVA and learning gains)**</figcaption></figure>

<figure><img src="image_37.png" /><figcaption>**Group 38 (low JVA and learning gains)**</figcaption></figure>

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

**Figure 3.** Cross-recurrence graphs for two dyads. The left side shows moments of joint visual attention (black), no joint visual attention (gray) and missing data (white). The right side shows joint attention on particular AOIs (gold = computer screen, green = maze). The two graphs on the left show a productive group with high learning gains (42), and the two graphs on the right shows a group with low learning gains (38).

## Joint Visual Attention

Joint Visual Attention (JVA) was computed according to prior research (Richard & Dale, 2015; Schneider & al., 2018; Schneider & Pea, 2013; Gergle & Clark, 2011). As a first pass, I used a radius of 100 pixels for two gaze points to be considered as JVA (this radius is shown on Figure 1). For each gaze point, I also checked whether the other group member looked at the same area within +/- 2 seconds (which has been shown to be the amount of time necessary for someone to disengage from what they are doing and pay attention to a partner's actions; Richardson). The results are summarized below.

I found significant correlations between JVA and: “sustaining mutual understanding”: $r(36) = 0.397, p = 0.014$; “task division”: $r(36) = 0.351, p = 0.031$; and their overall quality of collaboration: $r(36) = 0.341, p = 0.036$ (see Meier, Spada, & Rummel. 2007 for a definition of those constructs). There was no significant correlation with task performance or learning gains. It should be noted that I also looked at other radius sizes in addition to 100 pixels (50, 150, and 200 pixels) which defined the distance between two gaze points to be considered as a moment of joint visual attention. In these analyses, the size of the radius did not influence the correlations reported above.

## Cycles of collaboration (JVA) and individual work (no-JVA)
For those analyses, I look at the number of times participants shifted between collaborative work (i.e., with increased levels of joint visual attention) and individual work (i.e., with lower levels of joint visual attention). I tried several approaches and found that the following steps provided the most conclusive measure: 1) I summed the number of moments of JVA for different time windows (a 60 second time window is shown on Fig. 4); 2) I compared each observation with the previous point and looked at whether JVA was going up or down; 3) I counted the number of times the group shifted from increasing to decreasing (and vice versa) their levels of JVA. In other words, this measure offers an estimate for cycles of individual and collaborative work.

<figure><img src="image_40.png" /><figcaption>**Figure 4.** Levels of joint visual attention over time for two groups. Group 15 had the lowest number of cycles according to the measure above (12) and group 28 had the larger number of cycles (24).</figcaption></figure>

This measure was correlated with learning gains using 30 seconds increment (i.e., 30sec., 60sec., 90sec., 120sec.). I found significant correlations with a 30 second window: r(35) = 0.349, p = 0.035, 90 sec. window: r(35) = 0.355, p = 0.031, 120 sec. window: r(35) = 0.515, p = 0.001 but not with a 60 sec. window: r(35) = 0.001, p = 0.99. Finally, by looking at smaller time windows between 10sec and 60sec., there was a time window (40 sec.) that was significantly correlated with all three dependent measures: overall quality of collaboration r(34) = 0.347, p = 0.038, Task Performance r(34) = 0.355, p = 0.034 and Learning gains r(35) = 0.398, p = 0.015. There was no significantly correlation between the aggregated JVA measure reported in the section above (i.e., the total amount of JVA) and the measure described in this section - which suggests that they are capturing two different constructs. I discuss these results below.

## Discussion

This paper replicates prior results showing that JVA is positively correlated with high quality collaborative interactions (Jermann, Mullins, Nüssli & Dillenbourg, 2011; Schneider & Pea, 2013; Schneider, Sharma, Cuendet, Zufferey, Dillenbourg & Pea, 2018). It also provides further evidence that JVA can be computed in a co-located setting using fiducial markers disseminated in the environment. Since most of the prior work was done in remote settings, it is timely that new approaches allow researchers to study collaborative learning in more ecological ways. The final and main contribution of this paper is a new measure that captures cycles of collaboration and individual work in dyads. This measure provides a complementary lens into collaborative processes: I found JVA to be positively associated with collaboration quality, and this new measure with learning gains (as well as task performance and collaboration quality, depending on the threshold used). This suggests that an important feature of successful collaborative learning groups is to balance individual cognition with group work. While this is beyond the scope of this paper, future work will study this effect in more detail by qualitatively analyzing dyads that are driving this effect (i.e., groups with low learning gains and low scores on this measure, and dyad with high learning gains and high scores on this measure). Additionally, I am planning to replicate the findings above on a different dataset, which would provide further evidence that cycles of collaboration and individual work positively contribute to learning.

## Conclusions

This paper presents a study where dyads of participants worked on programming a robot to solve a variety of mazes. I found that Joint Visual Attention can be captured using dual eye-trackers in co-located settings, and that this measure is positively correlated with collaboration quality. Additionally, I designed a new measure intended to capture cycles of individual work and group collaboration. This measure shed a new light on what constitutes productive interaction in dyads of students.

It should be acknowledged that this paper has several limitations (for a discussion of the limitations related to the task and the dependent measures used, please refer to Starr, Reilly & Schneider, 2018). First, this
paper mostly relied on correlations. While this provides intuitive results, future work should use more comprehensive statistical tests to model participants' interactions and control for collinearity. Second, the new measure presented in this paper relies on several parameters (minimum distance between two gazes to qualify as joint visual attention, different time windows) that were arbitrary defined. A better understanding of how those parameters need to be fine-tuned is important for generalizing this measure to other settings. Finally, dual eye-tracking only offers a limited view of collaborative processes. Future work will integrate sensor data from multiple modalities (e.g., electrodermal, motion and speech data) to get a more complete picture of what constitutes productive interactions in co-located settings.

In conclusion, this study shows that it is possible to develop new ways of capturing 21st century skills in hands-on tasks typical of makerspaces. Even with the limitations mentioned above, this work makes a first step in this direction and opens the way to more rigorously studying collaborative processes in open-ended learning environments using dual mobile eye-trackers.

## References

Berland, M., Baker, R. S., & Blikstein, P. (2014). Educational data mining and learning analytics: Applications to constructionist research. *Technology, Knowledge and Learning, 19*(1–2), 205–220.

Blikstein, P., & Worsley, M. (2016). Multimodal Learning Analytics and Education Data Mining: using computational technologies to measure complex learning tasks. *Journal of Learning Analytics, 3*(2), 220–238.

Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), *Perspectives on socially shared cognition* (pp. 127–149). Washington, DC, US: American Psychological Association.

D’Angelo, S., & Begel, A. (2017). Improving Communication Between Pair Programmers Using Shared Gaze Awareness. In *Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems* (pp. 6245–6290). New York, NY, USA: ACM.

Dede, C. (2010). Comparing frameworks for 21st century skills. *21st Century Skills: Rethinking How Students Learn, 20*, 51–76.

Gergle, D., & Clark, A. T. (2011). See What I’M Saying?: Using Dyadic Mobile Eye Tracking to Study Collaborative Reference. In *Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work* (pp. 435–444). New York, NY, USA: ACM.

Jermann, P., Mullins, D., Nüssli, M.-A., & Dillenbourg, P. (2011). Collaborative Gaze Footprints: Correlates of Interaction Quality. *Connecting Computer-Supported Collaborative Learning to Policy and Practice: CSCL2011 Conference Proceedings., Volume I-Long Papers*, 184–191.

Meier, A., Spada, H., & Rummel, N. (2007). A rating scheme for assessing the quality of computer-supported collaboration processes. *International Journal of Computer-Supported Collaborative Learning, 2*(1), 63–86.

Miyake, N. (1986). Constructive interaction and the iterative process of understanding. *Cognitive Science, 10* (2), 151-177.

Richardson, D. C., & Dale, R. (2005a). Looking To Understand: The Coupling Between Speakers’ and Listeners’ Eye Movements and Its Relationship to Discourse Comprehension. *Cognitive Science, 29*(6), 1045–1060.

Richardson, D. C., Dale, R., & Kirkham, N. Z. (2007). The Art of Conversation Is Coordination Common Ground and the Coupling of Eye Movements During Dialogue. *Psychological Science, 18*(5), 407–413.

Schneider, B., & Pea, R. (2013). Real-time mutual gaze perception enhances collaborative learning and collaboration quality. *International Journal of Computer-Supported Collaborative Learning, 8*(4), 375–397.

Schneider, B., Sharma, K., Cuendet, S., Zufferey, G., Dillenbourg, P., & Pea, R. (2018). Leveraging Mobile Eye-Trackers to Capture Joint Visual Attention in Co-Located Collaborative Learning Groups. *International Journal of Computer-Supported Collaborative Learning*.

Starr, E., Reilly, J., & Schneider, B. (2018). Toward Using Multi-Modal Learning Analytics to Support and Measure Collaboration in Co-Located Dyads. *ICLS2018 Conference Proceedings., Volume I-Long Papers*, 448–455.

Tomasello, M. (1995). Joint attention as social cognition. In C. Moore & P. J. Dunham (Eds.), *Joint attention: Its origins and role in development* (pp. 103–130). Hillsdale, NJ, England: Lawrence Erlbaum Associates, Inc.

Tschan, F. (2002). Ideal Cycles of Communication (or Cognitions) in Triads, Dyads, and Individuals. *Small Group Research, 33*(6), 615–643.

Yu, C., & Smith, L. B. (2013). Joint attention without gaze following: Human infants and their parents coordinate visual attention to objects through eye-hand coordination. *PloS One, 8*(11), e79659.
# The Lifespan and Impact of Students' Ideas Shared During Classroom Science Inquiry

Camillia Matuk, New York University, cmatuk@nyu.edu

Wanjing (Anya) Ma, University of Pennsylvania, wma821@nyu.edu

Garima Sharma, New York University, garimasharma@nyu.edu

Marcia C. Linn, University of California, Berkeley, mclinn@berkeley.edu

**Abstract:** Sharing ideas can strengthen students' science explanations. Yet, how to guide uses of peers' ideas, and what the impacts of those ideas are on students' learning, are open questions. We implemented a web-based cell biology unit with 116 grade 7 students, and explored how peers' ideas are used during explanation building, and how prompts to draw on peers to either diversify or reinforce existing ideas impacted the quality of students' written explanations. Among other findings, exchanging ideas with peers led to all students improving their explanation quality upon revision; and students prompted to diversify their ideas showed greater learning gains by the end of the unit, while students prompted to reinforce ideas, who used more peer-generated ideas in preparation to write their explanations, produced higher quality explanations. This study builds our understanding of the influence of peer ideas on learning, and offers insight into supporting students in engaging effectively with peers' ideas.

## The role of peer ideas in knowledge integration

Because scientific knowledge is socially constructed (Latour & Woolgar, 1979; Lemke, 1990), the ability for science learners to engage productively with one another's ideas is critical to develop, as is their ability to refine explanations and arguments in light of new ideas (NGSS Lead States, 2013; Kuhn, 2012). Nurturing a culture around sharing and improving collective understanding can position students as creators, rather than just readers of knowledge (Scardamalia & Bereiter, 2014). However, research is mixed on the best ways to support students' encounters with others' ideas (De Jonge, 2018). For example, one study documented how the ideas that middle school students collected during their inquiry investigations related to the quality of their later scientific explanations. Specifically, students' tendencies to find ideas that reinforce, as opposed to diversify their existing repertoire of ideas before writing an explanation, and their tendencies to self-generate those ideas rather than to use peer-generated ideas, also resulted in them writing stronger scientific explanations (Matuk & Linn, 2018). Another study suggests that diversifying rather than reinforcing ideas is a more successful strategy for students of higher prior knowledge than it is for students of lower prior knowledge (Matuk & Linn, under review). These findings begin to suggest how different ways of engaging with peers' ideas can support students' own thinking; and for whom these ways may be more or less productive.

Our research is guided by the Knowledge Integration (KI) framework (Linn & Eylon, 2011), a perspective based in research in the learning sciences, and that views learners as constructing understanding by distinguishing among various new and existing ideas. KI instruction supports learning by eliciting students' existing ideas, assisting them in distinguishing these from newly encountered ideas, and organizing these into a coherent understanding. However, the role of peers in knowledge integration is less clear. Little work has focused, for example, on the lifespan of shared ideas in a classroom. That is, when students share ideas, which types of ideas are taken up and used throughout the course of students' inquiry? How do these shared ideas support students in constructing sound explanations? Moreover, what strategies for using these ideas might be more or less effective?

This study extends prior research by exploring the kinds of ideas that students exchange during inquiry, and ultimately incorporate into their explanations. It also explores the value of different strategies for using peers' ideas on the quality of their explanations, and on their overall learning outcomes.

## Research questions

We implemented a web-based science inquiry unit in a middle school classroom, which features a tool that supports students in sharing ideas with classmates as they work toward constructing scientific explanations. Our specific research questions are:

1. What overall impact does the unit have on students' conceptual learning? How does prompting students to seek similar vs. different ideas from peers impact their overall learning?

2. How does prompting students to seek similar vs. different ideas from peers impact the quality of
students' explanations?

3. What is the trajectory of ideas across students' explanation building process? That is, how did students take up and incorporate their peers' shared ideas? Which kinds of ideas were more popular? How did peers' ideas impact the quality of students' explanations?

## Methods

### WISE and the Idea Manager

The Web-based Inquiry Science Environment, or WISE (wise.berkeley.edu) is a free, open-source platform created to design and deliver classroom-based science inquiry curriculum (Slotta & Linn, 2009). The Idea Manager is a tool integrated into WISE to support students in collecting, distinguishing, and organizing their ideas into coherent science explanations during their inquiry investigations (Matuk et al., 2016). Following the KI framework, the Idea Manager offers a persistent space called the *Private Idea Basket*, within which students can document their existing and new ideas in brief entries; sort and distinguish among these ideas using a visual organizer called the *Explanation Builder*; and refer to their organized ideas to integrate them into a written explanation (Figure 1). Students can choose to add any of their idea entries to a *Public Basket*, which anonymously lists all ideas shared by students in the same class. Students can select publicly shared ideas to “copy” into their own Private Baskets. Thus, peer-generated ideas become available for use alongside students’ self-generated ideas.

<figure><img src="image_41.png" /><figcaption>Figure 1. Foreground: An idea entry dialog box. Background: the Explanation Builder, in which students organize ideas, then refer to them to write an explanation (by clicking the “Ready to Explain” button).</figcaption></figure>

### The Mitosis unit

The unit, *What makes a good cancer medicine?: Observing mitosis and cell processes (Mitosis)*, introduced students to the process of cell division and its relationship to cancer and cancer treatment. Animations, diagrams, and narrative explanations introduced students to the phases of normal cell division, and to the notion that cancer is a disease in which cells have lost the mechanism that controls their division. An effective cancer treatment must thus stop cancerous cells from dividing, but this does not occur without side effects.

Students then compared the effectiveness of three potential, plant-derived cancer medicines. They viewed animations of dividing cells that were treated by each medicine, and noted their observations in their Private Idea Baskets. Following this, they organized their ideas in the Explanation Builder to sort pros and cons for each medicine, and then referred to these ideas to write an explanation for the medicine they recommended.

Students were then asked to make public the ideas they used in their recommendations, and to select public ideas from their peers to add to their own Idea Baskets. They then reorganized their previously and newly
added ideas within the Explanation Builder and referred to these to revise their recommendations (Figure 2).

<figure><img src="image_42.png" /><figcaption>Figure 2. The *Mitosis* activity sequence and study design.</figcaption></figure>

## Participants and study design

Participants were 144 grade 7 students across 5 class periods of one teacher, from a diverse public middle school on the West coast of the United States. Students worked on the unit in pairs at their own pace during class time for 10 consecutive school days. The teacher, who had 5+ years of experience teaching with WISE, circulated to assist students, led occasional whole class discussions to address common conceptual challenges, and offered guidance on upcoming unit activities. Collaboration occurred at different levels: First, by working on shared computers, student partners had to come to consensus through discussion over their responses. Second, the teacher regularly reminded students to document and share their ideas with others in class, such as by pointing out good ideas during her conversations with partners, and encouraging them to add these to the Public Basket.

Students individually completed a pre and posttest on the first and last days of the study. One of the three items on this pre/posttest addressed students' understanding of the order and importance of the phases of cell division. A second item tested their ability to apply this understanding to reason about the action of an effective cancer medicine. A third item asked students to select ideas from fictional peers that would help them to write an explanation for the role of spindle fibers during cell division.

To investigate the value of different strategies for using peers' ideas, students were divided into two conditions. In the Reinforce condition (N=66 students, 36 workgroups), students were prompted to collect peer ideas that were similar to their own ideas. In the Diversify condition (N=50 students, 27 workgroups), they were prompted to collect peer ideas that differed from their own. We excluded from our analysis students who had not completed the pre/posttest, nor submitted both initial and revised recommendations. The total number of students in our dataset was therefore 116.

## Data and analysis

Data consisted of each workgroup's individual ideas, including information on which ideas they kept private and which they made public, which public ideas were copied by which workgroups, and which ideas were used in students' Explanation Builders. We also collected students' initial and revised written recommendations and their responses to the pre/posttest.

To determine differences between conditions in the overall impact of the unit on students' learning outcomes (RQ1), we scored students' pre and posttests based on previously developed KI rubrics, which give credit to responses that integrate key ideas (Matuk & Linn, 2018). Two independent raters achieved high inter-rater reliability across the three items ($κ=0.91, 0.83$ and $0.71$). We summed scores across items to obtain a total score for each student, and used a t-test to detect differences in mean total scores between conditions.

To compare the effects on students' recommendations of prompts to seek peer ideas that differed and that resembled their own (RQ2), we scored students' initial and revised recommendations. One rubric identified the presence of *Key Concepts* (Table 1) and another rated the *Argument Structure* (Table 2), while the sum of the scores on these two criteria produced an overall score out of 10 of the explanation's quality. Two independent raters scored 20-30 student responses at a time, and resolved disagreements through discussion
until inter-rater reliability had been achieved (Key Concepts κ=0.83; Argument Structure κ=0.96). We used t-tests to detect differences in students' gains on the individual Key Concepts and Argument Structure dimensions, and on their overall explanation quality, between workgroups' initial and revised recommendations, and between the Reinforce and Diversify conditions.

<table><caption>Table 1: Rubric for identifying Key Concepts in students' initial and revised recommendations</caption><thead><tr><th>Score</th><th>Criteria</th><th>Examples</th></tr></thead><tbody><tr><td>0/1</td><td>No Answer/Offtask or uninterpretable</td><td>(Blank)/I don't know.</td></tr><tr><td>2</td><td>Irrelevant, incorrect or ambiguous</td><td>We think plant A will work the best because it kills off the cell without leaving with empty cells that clog the body which is what cancer does as well.</td></tr><tr><td>3</td><td>Any ONE of the three key ideas is correctly explained.</td><td>I think that the best cancer medicine is plant A, because it will <u>stop the cell from undergoing mitosis [Need]</u>. It will stop the [sic] cell from ever doing mitosis, but the cell might still be able to survive.</td></tr><tr><td>4</td><td>Any TWO of the three key ideas are correctly explained.</td><td>We will recommend plant A because it stopped the <u>spindle fibers [Org]</u> from working before a new cell was created therefore it <u>stopped mitosis [Need]</u>.</td></tr><tr><td>5</td><td>All THREE key ideas are correctly explained.</td><td>Plant A because it is the most effective at <u>stopping mitosis [Need]</u>. The pros to this plant is that the <u>spindle fibers [Org]</u> don't go all the way so mitosis isn't complete. So if the spindle fibers <u>don't go all the way [Fxn/Process]</u> through then the <u>chromosomes [Org]</u> won't <u>line up [Process]</u>. The cons are that their will be side effects from the medicine.</td></tr></tbody></table>

<table><caption>Table 2: Rubric for evaluating the Argument Structure of students' initial and revised recommendations</caption><thead><tr><th>Score</th><th>Criteria</th><th>Examples</th></tr></thead><tbody><tr><td>0/1</td><td>No Answer/Offtask or uninterpretable</td><td>(Blank)/I don't know.</td></tr><tr><td>2</td><td>Claims stated with no supporting evidence.</td><td>I think Plant A is most effective.</td></tr><tr><td>3</td><td>Interpretation given that includes a claim supported by evidence, which may or may not properly align. No counter-arguments are given.</td><td>we think that the most effective cancer medicine is plant A due to the fact that it stopped the spindle fibers from growing and the spindle fibers will not grow to redirect the chromosomes [supported interpretation]</td></tr><tr><td>4</td><td>Interpretation given that includes a claim aligned with supporting evidence. Offers unelaborated counter-arguments (e.g. pros/cons, side effects, comparison to alternatives).</td><td>Plant A because it is the most effective at stopping mitosis [supported interpretation]. The pros to this plant is that the spindle fibers don't go all the way so mitosis isn't complete. So if the spindle fibers don't go all the way through then the chromosomes won't line up. The cons are that their will be side effects from the medicine [unelaborated counterargument].</td></tr><tr><td>5</td><td>Interpretation given that includes a claim aligned with supporting evidence. Offers at least one elaborated counter-argument (e.g. pros/cons, side effects, or comparison).</td><td>We think that medicine A is the best because it stops the cell from dividing, which would work the best [valid interpretation]. Medicine B only destroys the second pair of chromosomes, which means when the medicine wears off, the cell can split again. Medicine C only destroys the cells' membranes which means that the cells might still be able to split [elaborated counterargument].</td></tr></tbody></table>

To track the trajectory of ideas (RQ3), we gave each idea a unique “private idea ID” that indicated the workgroup that had generated it; and each copied idea a unique “copied idea ID” that indicated the workgroup that copied it. These IDs allowed us to identify the trajectory of each idea from its origin, to its sharing in the Public Basket, and to its use in an Explanation Builder step. Finally, we used these IDs to note which ideas were
present in students' written recommendations. We used t-tests to detect significant differences in students' engagement with ideas across the unit. We also tested how the proportion of peer- vs. self-generated ideas impacted the quality of students' revised recommendations.

Because students were collecting ideas to construct arguments for the medicines they recommended, we characterized the ideas in terms of three main components of an argument: *Claim*, *Observation*, and *Interpretation* (Table 3). These categories are based on theories of argument structure, which describe the function of argument to be in creating links between claims and evidence (Burleson, 1979; Kneupper, 1979; Toulmin, 2003). Two independent raters scored 20 ideas at a time until they had achieved Cohen's kappa values of 0.83, 0.90, and 0.83 on *Claim*, *Observation*, and *Interpretation*, respectively. One coder then categorized the rest of the ideas. We excluded from our analyses nine of the ideas that were irrelevant to the task of constructing a recommendation (e.g., “Three plant idea”).

We determined the relative *popularity* of ideas by the number of times these were copied by other workgroups. We then used an ANOVA to uncover associations between the kinds of ideas and their popularity.

<table><caption>Table 3: Scoring rubrics for distinguishing ideas as Claim, Observation, and Interpretation</caption><thead><tr><td>Type</td><td>Description</td><td>Example</td></tr></thead><tbody><tr><td>Claim</td><td>Declarative, factual statements not supported by evidence from the unit.</td><td>Cancer is the overgrowth of cells. It can travel down the body in the blood stream.</td></tr><tr><td>Observation</td><td>Descriptions, without accompanying inferences, of information from the unit (e.g., of animations).</td><td>The treated cell's spindle fibers stopped. They didn't grab the chromosomes. They're also green.</td></tr><tr><td>Interpretation</td><td>Causal/explanatory statements or inferences that integrate ideas that are not otherwise explicit in the unit.</td><td>The mitosis is not prevented but slowed down because the daughter cells wont be able to divide.</td></tr></tbody></table>

# Findings

## RQ1: How did the unit impact students' learning overall?

The unit as a whole appeared to positively impact students' conceptual learning. All students showed significant gains between the pre and posttest (N=116; M=3.16; SD=2.27; t(115)=15.00, p<.001, d=1.63). Prompts for students to use peers' ideas to diversify their own ideas seemed to be particularly beneficial, as students in the Diversify condition (N=50; M=3.74; SD=2.26) showed significantly greater gains than students in the Reinforce condition (N=66, M=2.73, SD=2.20), t(114)=-2.43, p=.017, d=.45).

## RQ2: How did exchanging ideas impact students' written explanations?

Students' overall recommendations improved significantly upon revision following their exchange of ideas with their peers (M=1.14; SD=1.80; t(62)=5.05, p<.001, d=.75). In both conditions, students' final recommendations reflected more key concepts (M=0.57, SD=1.04, t(62)=4.35, p<.001, d=.58) and better argument structure (M=0.57; SD=0.14; t(62)=4.11, p<.001, d=.65) than their initial recommendations. There was no significant difference in the overall explanation quality between the Reinforce condition (M=8.94, SD=1.59, N=36) and the Diversify condition (M=8.78, SD=1.37, N=27), t(61)=0.438, p=.663, d=.108. There were also no significant differences in gains between the Key Concepts and Argument Structure criteria within nor between conditions. These findings suggest that exchanging ideas generally had a positive impact on students' revisions, regardless of the manner by which students were prompted use their peers' ideas.

## RQ3: What was the trajectory of ideas across the unit?

### How did students take up and incorporate their peers' ideas into their explanations?

To understand the overall trajectory of self- and peer-generated ideas, we graphed the average number of ideas at six time points (Figure 3). These included the ideas before and leading up to the initial recommendation, specifically, the ideas (1) generated in students' Private Baskets (Reinforce, M=7.33; Diversify, M=8.66); (2) organized in the Explanation Builder (Reinforce, M=5.02; Diversify, M=5.22); and (3) incorporated into the initial written recommendations (Reinforce, M=2.22; Diversify, M=2.55). Following students' exchange of ideas with their peers, we tracked (4) the copied ideas that students added to their Private Baskets (Reinforce, M=3.86; Diversify, M=3.33); as well as the additional private ideas generated at this point (Reinforce, M=7.58; Diversify, M=8.99); (5) the private ideas re-organized in their Explanation Builders (Reinforce, M=4.61; Diversify, M=4.88); and the copied ideas organized alongside them (Reinforce, M=2.08; Diversify, M=2.33).
Finally, we noted (6) which self-generated ideas (Reinforce, M=3.16; Diversify, M=3), and which peer-generated ideas students incorporated into their revised recommendations (Reinforce, M=1.69; Diversify, M=1.18).

<figure><img src="image_43.png" /><figcaption>Figure 3. Trajectory of ideas generated by students (N=63 workgroups) across both conditions. (EB=Explanation Builder)</figcaption></figure>

Figure 3 shows that across conditions, students generated more ideas (M=7.9) than they organized in their Explanation Builders (M=5.11), and used in their initial recommendations (M=2.38). This same pattern is seen in students’ uses of their peers’ ideas. Specifically, students copied more of their peers’ ideas (M=3.77) than they organized in their Explanation Builders (M=2.19), and incorporated into their revised recommendations (M=1.47). Notably, students also generated more (M=8.18), and incorporated more of their own ideas, as opposed to their peers’ ideas, into their revised recommendations (M=3.09) following their exchange with their peers.

Compared to students’ uses of ideas leading up to their initial recommendations, students added more ideas to their Private Baskets (N=63; M=4.06; SD=3.33; t(62)=6.70, p<.001, d=.88), organized more ideas in their Explanation Builders (N=63; M=1.81; SD=1.87; t(62)=7.70, p<.001, d=.72), and incorporated more ideas into their final recommendations (N=63; M=2.21; SD=0.25; t(62)=8.68; p<.001, d=1.01) following their exchange of ideas with peers. Students also used significantly more self-generated than peer-generated ideas in their revised Explanation Builders (N=63; M=2.54; SD=2.75; t(62)=7.43, p<.001, d=1.29) as well as in their revised recommendations (N=63; M=1.62; SD=2.22; t(62)=5.80, p<.001, d=0.99) compared to the steps leading up to their initial recommendations.

These patterns suggest that students were discerning in the ideas they chose to incorporate into their explanations, a finding that supports the knowledge integration process of distinguishing and sorting among ideas to find the most relevant ones. They also suggest that students used their peers’ ideas to complement, rather than to supplant, their own ideas, which attests to the success of the Idea Manager in enabling students to draw upon their peers as supports, rather than being reliant upon them in building explanations.

## Which ideas were most popular?

Across both conditions, *Observations* was the most frequently generated kind of idea (73% Reinforce; 63% Diversify), followed by *Interpretations* (15% Reinforce; 28% Diversify) then *Claims* (12% Reinforce; 18% Diversify). *Observations* was also the most highly copied kind of idea (Reinforce, M=2.08; Diversify, M=1.93), and significantly more so than either *Interpretations* (Reinforce, M=2.00; Diversify, M=1.18) or *Claims* (Reinforce, M=0.50; Diversify, M=0.59), (F(2,186)=3.99; p=.020) (Figure 4).

It may be that students found observations to be the most persuasive, and *Claims* to be the least robust kinds of peer ideas. For example, the most popular *Claim* across our dataset, copied just 4 times, was “there would be only half as many cells.” This idea, which refers to the effect of one of the three medicines on cell division, offers neither supporting evidence, nor an explanation of the consequences of having “half as many cells.” Students may thus have regarded it as being of little use for strengthening their own explanations. In contrast, students may have viewed *Interpretations*, particularly from their peers, to not be as trustworthy as a teacher’s explanation, for example. For instance, the most popular Interpretation, copied 8 times, was “Plant A is good because it stops mitosis before it reaches anaphase.” This idea offers an explanation, albeit a limited one, for why Plant A is the favored medicine. However, it also offers little detailed evidence, such that students
might be required to trust it at face value. Meanwhile, students may have found observations to be most easily grasped because they could verify these themselves by examining evidence in the unit. For example, the most popular Observation, copied 21 times, was “the plant chemicals don’t stop the division of the cell. the chemicals get rid of the copy of the chromosome so the daughter cell has no chromosomes.” This idea articulates observable events from the animations, which may either offer students new language to rearticulate their existing observations, or highlight details that they might have initially overlooked.

<figure><img src="image_44.png" /><figcaption>**Figure 4.** Totals of each kind of idea shared within the Public Idea Manager, and the proportion of ideas that were copied vs. not copied by students across both conditions.</figcaption></figure>

## How did peers' ideas impact the quality of students' explanations?

Students in the Reinforce condition who organized a greater proportion of peer- vs. self-generated ideas in the Explanation Builder tended to produce higher quality recommendations, that is, recommendations with scores of 9 or more out of 10, $t(35)=-2.18, p=.036, d=.78$. For students in the Diversify condition, however, the relative proportion of peer- vs. self-generated ideas did not have a significant impact on the quality of their explanations ($t(26)=1.00, p=.327, d=.38$). This finding suggests a benefit of organizing peer ideas that reinforced students' existing ideas.

However, there was no relationship between the proportion of self-generated and peer-generated ideas used in students' recommendations and the quality of those recommendations. This suggests that students' improved recommendations were not simply because of access to their peers' ideas, but more likely because of the effort they put into integrating their own and their peers' ideas into their explanations.

Figure 5 shows two examples of how students in each condition used their peers' ideas to improve their recommendations. In both cases, peers' ideas helped to elaborate recommendations with evidence and to strengthen arguments with counter-arguments.

<figure><img src="image_45.png" /><figcaption>**Figure 5.** Examples of students' self-generated ideas, ideas chosen from peers, and initial and revised recommendations. (Left: Workgroup 119924; Right: Workgroup 119945)</figcaption></figure>

## Discussion and significance

Whereas the unit benefited all students' learning overall, students prompted to diversify their ideas gained significantly more. Meanwhile, students prompted to reinforce ideas, and who relied more so on peer-generated ideas than on self-generated ideas, produced higher quality in-unit explanations. These findings resonate with prior mixed research on the relative value of diversifying vs. reinforcing ideas. For example, encountering diverse ideas can improve conceptual learning (e.g., Asterhan & Schwarz, 2009; Matuk & Linn, 2018). At the same time, encountering one's own ideas rearticulated by others can prompt students to revise and improve their own ideas (e.g., Matuk & Linn, 2008; Edge, 2006; Hayes, 2004). Other work suggests that the benefits of reinforcing and diversifying ideas may differ for students with high vs. low prior knowledge (e.g., Matuk &
Linn, under review). Findings from this study suggest that in addition to different students benefiting from different prompts, students may also benefit at different times while building explanations.

Continued analyses might investigate whether there were pre-existing differences between conditions (e.g., prior knowledge) that would offer alternative explanations for our findings, and also whether students actually followed prompts to choose either diversifying or reinforcing ideas. Further research might explore how different students (e.g., high vs. low prior knowledge) benefit from different prompts to use their peers' ideas; and how various strategies (e.g., collecting mostly interpretations vs. observations) impact their success in explanation building.

Visualizing the trajectory of peers' ideas throughout the unit offered insights into the kinds of ideas that students were more or less likely to copy. Importantly, prompts to diversify or to reinforce ideas had no apparent effect on the ways that students used ideas throughout their inquiry. Continued analyses will investigate how the kinds of ideas generated, organized, and used, impact the quality of students' explanations. Future research might also explore other ways to characterize ideas besides in terms of the components of the argument. For instance, we might explore which science concepts are more or less challenging for students to identify at different stages of explanation (cf. McElhaney et al., 2012). Finally, we might examine the role of the teacher in conjunction with technology, in supporting students' productive interactions with peers.

In all, exchanging ideas with peers appeared to positively impact both students' written explanations and their overall learning outcomes. This study adds nuance to the literature on learning from peers. It moreover has implications for technology-rich knowledge sharing supports in classroom-based science inquiry.

## References

Asterhan, C.S.C. & Schwarz, B.B. (2009). Argumentation and explanation in conceptual change: Indications from protocol analyses of peer-to-peer dialog. *Cognitive Science*, **33**, 374-400.

Burleson, B. R. (1979). On the foundations of rationality: Toulmin, Habermas, and the a priori of reason. *The Journal of the American Forensic Association*, **16**(2), 112-127.

De Jonge, K. M., Rietzschel, E. F., & Van Yperen, N. W. (2018). Stimulated by novelty? The role of psychological needs and perceived creativity. *Personality and Social Psychology Bulletin*.

Edge, J. (2006). Computer-mediated cooperative development: Non-judgemental discourse in online environments. *Language Teaching Research*, **10**(2), 205-227.

Hayes J.R. (2004) What Triggers Revision?. In: Allal L., Chanquoy L., Largy P. (eds), *Revision Cognitive and Instructional Processes. Studies in Writing*, Vol 13. Springer, Dordrecht

Kneupper, C. W. (1979). Paradigms and problems: Alternative constructivist/interactionist implications for argumentation theory. *The Journal of the American Forensic Association*, **15**(4), 220-227.

Latour, B., & Woolgar, S. (1979). *Laboratory life: The social construction of scientific facts*. Beverly Hills: Sage.

Lemke, J. L. (1990). Talking science: Language, learning, and values. Norwood, NJ: Ablex Publishing Corporation.

Linn, M. C., & Eylon, B. S. (2011). *Science learning and instruction: Taking advantage of technology to promote knowledge integration*. New York: Routledge.

Matuk, C. & Linn, M. C. (2018). Why and how do middle school students exchange ideas during science inquiry? *International Journal of Computer-Supported Collaborative Learning*, **13**(3), 263-299.

Matuk, C., McElhaney, K., King Chen, J., Lim-Breitbart, Kirkpatrick, D. & Linn, M. C. (2016). Iteratively refining a science explanation tool through classroom implementation and stakeholder partnerships. *International Journal of Designs for Learning*, **7**(2), 93-110. DOI: 10.14434/ijdl.v7i2.20203

McElhaney, K., Miller, D., Matuk, C., & Linn, M. C. (2012). Using the Idea Manager to promote coherent understanding of inquiry investigations. In *ICLS'12: Proceedings of the 10th International Conference for the Learning Sciences*, (Vol. 1, pp. 323-330). Sydney: Intl Society of the Learning Sciences.

Scardamalia, M., & Bereiter, C. (2014). Knowledge building and knowledge creation: Theory, pedagogy, and technology. *Cambridge Handbook of the Learning Sciences*, pp. 397-417. Cambridge: Cambridge University Press.

Slotta, J. D., & Linn, M. C. (2009). *WISE science: Web-based inquiry in the classroom*. New York: Teachers College Press.

Toulmin, S. E. (2003). *The uses of argument*. Cambridge University Press.

## Acknowledgments

We thank the students and teachers who participated in our study. We also acknowledge funding from the National Science Foundation (Award No. 1119670) and from New York University.
# Collective Knowledge Advancement through Shared Epistemic Agency: Socio-Semantic Network Analyses

Jun Oshima, RECLS Shizuoka Univ., joshima@inf.shizuoka.ac.jp

Ritsuko Oshima, RECLS Shizuoka Univ., roshima@inf.shizuoka.ac.jp

Ayano Ohsaki, AIIT, ohsaki-ayano@aiit.ac.jp

Jin Michael Splichal, Univ. of Helsinki, jin.splichal@helsinki.fi

**Abstract:** This study examined how high-school students engage in collective knowledge advancement through shared epistemic agency during jigsaw instruction, and how their collective knowledge advancement (through shared epistemic agency) relates to learning outcomes. To achieve this, we applied a double-layered socio-semantic network analysis (SSNA). In the first layer of the analysis, we conducted an SSNA to numerically represent collective knowledge advancement and to compare group performances between different levels of learning outcomes. In the second layer, we carried out an Epistemic Network Analysis to examine the relationship between students' shared epistemic agency and learning outcomes. The results revealed that high learning-outcome groups were engaged in epistemic actions to generate new ideas; this led them to quick and sustainable knowledge advancement.

## Research purpose

This study has analyzed the way in which high-school students engage in collective knowledge advancement through shared epistemic agency (Damşa et al., 2010) during jigsaw instruction (Brown & Campione, 1996). It has also investigated the extent to which this collective knowledge advancement relates to learning outcomes. Although studies (e.g., Miyake & Kirschner, 2013) have demonstrated that jigsaw instruction is effective in facilitating conceptual understanding, few studies have shown how learners engage in collective knowledge advancement during collaboration (Oshima, Oshima & Matsuzawa, 2012). The present study explored the students' collective knowledge advancement by applying a double-layered socio-semantic network analysis (SSNA). In the first layer of the analysis, we conducted an SSNA to numerically and visually represent collective knowledge advancement and to compare group performance based on learning outcomes, as evaluated by the SBF framework (in pre- and post-testing). In the second layer, we carried out an Epistemic Network Analysis (ENA) (Shaffer, 2017) to examine the relationship between epistemic frames of shared epistemic agency and learning outcomes.

## Theoretical background

### Inquiry into collective knowledge advancement during jigsaw instruction

Many studies have attempted to design learning environments to facilitate students' collective knowledge advancement. One such instructional method is jigsaw instruction. In jigsaw instruction, learners engage in two different phases of an activity. In the phase known as the "expert group activity," learners work collaboratively on the same materials. Collaboration in the expert group facilitates constructive interaction among the learners (Miyake, 1986), enabling them to develop their own understanding by mutually monitoring each other's ideas from different points of view. After learning their materials during the expert group activity, learners join the jigsaw group activity, where those who studied different materials gather to integrate different sources of knowledge collaboratively. In the jigsaw group activity, learners are predicted to engage in productive interaction within multiple zones of proximal development, depending on the different knowledge sources (e.g., Brown & Campione, 1996). With respect to any one component of the learning activity, one student is an expert on that domain and teaches the other group members. This teaching-learning process is repeated multiple times.

Although studies have demonstrated that jigsaw instruction is effective in facilitating conceptual understanding and collective knowledge advancement (e.g., Miyake & Kirschner, 2014), few studies have examined in detail the process through which learners take responsibility for contributing to collective knowledge advancement through collaboration, i.e., shared epistemic agency. In relation to the knowledge-creation metaphor of learning (Paavola & Hakkarainen, 2005), students are expected to practice knowledge creation through collaboratively constructing knowledge objects (Bereiter, 2002). Scardamalia (2002) has discussed intentional engagement in collective knowledge advancement as epistemic agency—proposing this agency as a new goal for instruction (Scardamalia et al., 2012). Damşa et al. (2010) have also argued in favor of shared epistemic agency, focusing more on group-level agency. Through an in-depth discourse analysis, Damşa et al. (2010) found that
students in collaborative groups engage in the wholly joint epistemic actions of (1) being aware of their lack of knowledge, (2) alleviating the lack of knowledge, (3) creating a shared understanding, and (4) generative collaboration. To regulate their joint epistemic actions, students have also been found to engage in (1) projection by setting goals and creating joint plans, (2) regulation by monitoring and reflecting on their advancement, and (3) developing relations by transcending conflicts, redirecting critical feedback, and creating space for others' contributions. Studies by Scardamalia (2002) and Damşa et al. (2010) suggest that we need to analyze collaboration as knowledge creation, based on at least two layers of collective knowledge advancement: (1) how learners' ideas are improved through their collaborative discourse (i.e., idea improvement) and (2) how learners engage in improving their ideas collectively (i.e., shared epistemic agency).

## Double-layered socio-semantic network analysis

In this study, we propose using two layers of socio-semantic network analyses (SSNA). Recent studies (Oshima et al., 2012; Shaffer et al., 2009) have argued that existing social network models are unable to examine the way in which collective knowledge advances through learner collaboration. Two lines of research have been used to solve this problem. The first approach is to apply a procedure that is similar to an ordinary SNA to examine a different type of social network: a socio-semantic network based on the words that learners use in their discourse. Connections between words are assumed to represent clusters of ideas; the change in the socio-semantic network structure over time is examined both visually and computationally to investigate how a group of students engages in collective knowledge advancement (e.g., Ma et al., 2016; Oshima, Oshima & Fujita, 2018; Oshima et al., 2017). The second approach is another SSNA, based on epistemic frame theory, called Epistemic Network Analysis (ENA) (Shaffer, 2017). ENA is an algorithm that identifies and calculates connections among elements in coded data and visualizes them in dynamic network models that illustrate their structure and strength over time. With ENA, researchers can qualitatively and quantitatively examine cultural practices that participants engage in, such as engineering projects (e.g., Svarovsky, 2011), through their discourse. The present study uses both approaches to investigate different layers of student activity. We have carried out a vocabulary network analysis of their collective knowledge advancement and an ENA to assess their shared epistemic agency.

This study combines the approaches and findings of our previous studies, which examined students' collective knowledge advancement (Oshima et al., 2017), with a further ENA by coding the same discourse data from the perspective of shared epistemic agency.

## Conceptual understanding of complex scientific concepts

Through collective knowledge advancement, learners are expected to engage in complex tasks and comprehension of phenomena. Complex systems are multiple levels of organizations locally interacting with one another, such as financial economies and weather systems (Wilensky & Jacobson, 2013). Studies have revealed that students have difficulty mastering such complex subjects, despite their importance. One reason for the difficulty is that the complex concepts conflict with learners' prior experience. They usually have a "centralized" mindset and tend to favor explanations that assume central control and simple causality. In his interview study, Jacobson (2001) found that undergraduate students were more likely than experts to generate simple causality, central control, and predictability; by contrast, the experts exhibited decentralized thinking about multiple causes, such as stochastic and equilibration processes.

Hmelo-Silver and Pfeffer (2004) have proposed a structure-behavior-function (SBF) framework for assessing different levels of student understanding of complex systems. To assess students' understanding of an aquarium as a complex system, for instance, they used the SBF framework in the following way. Structures are elements of a system; in an aquarium, there are fish, plants, and a filter. Behaviors represent the way in which system structures achieve their purpose; for example, filters remove waste by trapping large particles, absorbing chemicals, and converting ammonia into harmless chemicals. Finally, functions express why an element exists within a given system—in other words, they express the purpose of that system element. For example, the filter removes aquarium byproducts. When researchers studied the verbal responses and pictorial representations of middle-school students, preservice teachers, and experts, they found that novices focused on perceptually available, static-system components. Experts, on the other hand, focused more on interrelations among structures, functions, and behaviors. These results suggest that the SBF framework could be a useful formalism for understanding complex systems.

The present study has used this framework to assess high-school students' conceptual understanding of the human immune system as a complex system. Both before and after the lesson, we asked the students to explain how vaccination protects us from infection by using their knowledge of the human immune system.

## Methods
## Student sample

Thirty-nine tenth-grade students at a high school in Japan participated in this study, as part of their regular curriculum. A science teacher with a Ph.D. in biology and more than ten years of teaching experience taught the students.

## Lesson design

### Structure of the collaborative learning activity: Jigsaw instruction

Twelve groups were formed with three or four students in each and were given a challenge, such as “Can you explain how vaccinations protect us from infections?” They were then provided with three study documents, each of which was needed to solve the challenge. First, individual students from each group gathered to form twelve expert groups (four groups to study each expert material) and worked on their allocated materials over 1.5 lesson periods (each lasting 50 min.). Second, students returned to their original group (the *jigsaw* group) and shared and integrated their knowledge in order to solve the challenge problem. This jigsaw activity took another 1.5 lesson periods. The teacher was responsible for group composition in both group activities.

### Study documents

We created the SBF framework for the human immune system based on a textbook description (Figure 1). We then discussed with the collaborating teacher the way to separate content, during expert group activities, into pieces of knowledge based on three essential functions: *humoral immunity*, *primary and secondary response*, and *cell-mediated immunity*. In the SBF framework, the three functions interact with one another as subsystems.

## Study design

### Pre- and post-tests

Students were questioned individually about their understanding of how the human immune system responds to vaccination (i.e., *Can you explain how vaccination protects us from infection?*). They were given a worksheet with a printed question, on which they could write or draw their ideas. The pre-test was conducted right before the lesson started. The post-test was conducted right after the lesson finished. Each test took one lesson period. Thirty-five students completed both tests; their responses were then analyzed.

### Process data collection

Student conversations during group activities were video-recorded and transcribed for vocabulary network analysis and ENA. We used transcripts of the jigsaw group activity to examine how students exerted their shared epistemic agency in advancing their collective knowledge.

### Analytical procedure

With reference to the SBF framework for the human immune system, student explanatory discourse during the pre- and post-test was categorized into three types: (1) no understanding, (2) single-document understanding, or (3) integrated understanding. When students did not manifest any conceptual understanding of the three documents used in the expert group activity, they were categorized as having no understanding. Students who appropriately demonstrated understanding of one of the three documents were categorized as having single-document understanding. Those who manifested a more complete understanding of the human immune system, based on more than two documents in the expert group activity, were categorized as having integrated understanding. The first and third authors independently evaluated ten randomly selected examples of student discourse in each of the pre- and post-tests. Cohen's Kappa coefficient for the agreement between the two raters was 0.92. Disagreements were resolved through discussion. The first author evaluated the remaining data. Because all students were categorized as having no understanding in the pre-test, we focused on levels of understanding in the post-test to assess learning outcomes.

To visualize and computationally investigate collective knowledge advancement, Oshima et al. (2017) selected 23 nouns to represent the structures and functions of the human immune system in the SBF framework. On average, students engaged in discourse exchange 358.5 times in the jigsaw groups (SD = 211.8). The socio-semantic network of vocabulary refers to meaningful links (i.e., co-occurrence) between words in exchanges. We then used an application called KBDeX (http://www.kbdex.net) to calculate the transition of the total value of degree centralities of nodes in the network across discourse exchanges, following the method used in previous research (Oshima et al., 2012). We then calculated the *term frequencies* (TF) of the selected vocabulary words, using the formula, $tf(t, d) = 1 + \log(ft,d)$. We compared TF means across the groups to test whether the amount of
discourse related to the human immune system differed significantly (particularly between high and low learning-outcome groups).

We conducted an ENA of the twelve groups’ discourse, using the seven epistemic actions of shared epistemic agency (Damşa et al., 2010) to code each discourse exchange. Cohen’s Kappa coefficients for the agreement between the two raters ranged between 0.47 and 1.00. Disagreements were resolved through discussion. Since we were interested in detecting the epistemic frames of shared epistemic agency exerted by all of the groups in relation to their learning outcomes, we conducted separate ENAs for high and low learning-outcome groups.

<figure><img src="image_46.png" /><figcaption>Figure 1. Structure–behavior–function framework of the human immune system.</figcaption></figure>

## Results

### Students’ learning outcomes at the individual and group level (Oshima et al., 2017)

We found that twenty-one students integrated SBF understanding. Eleven students demonstrated an understanding of a single part of a learned document; three did not sufficiently learn any piece of the SBF framework. A chi-square analysis of student frequencies across three types of learning outcome showed significance ($\chi^2 = 13.944, df = 2, p < .05$); in addition, the proportion of students who integrated SBF understanding was higher than the proportion with no understanding. These results suggest that the jigsaw activity did facilitate student integration of knowledge through collaboration, although group differences remained in the learning outcomes. Based on the SBF evaluation, we identified three high learning-outcome groups in which all members acquired integrated conceptual understanding. Nine low learning-outcome groups demonstrated mixed levels of understanding.

### Collective knowledge advancement: the difference between high and low learning-outcome groups

Figure 2 shows the transitions in total values of degree centralities, an index presented in our previous study (Oshima et al., 2017) to detect collective knowledge advancement. We found that the values quickly increased and ultimately exceeded 10.0 in the high learning-outcome groups, while values stayed low and increased slowly across discourse exchanges in the low learning-outcome groups. These results suggest that students in the high-learning-outcome groups engage in collective knowledge advancement more quickly and sustainably.

An additional one-way ANOVA of TFs found no significance among the twelve groups, $F(11, 198) = 2.35, p > .05$. This result reveals that students in all groups engaged in an equal amount of discourse related to the human immune system.
High Learning-Outcome Groups (N = 3)

<figure><img src="image_47.png" /></figure>

Low Learning-Outcome Groups (N = 9)

<figure><img src="image_48.png" /><figcaption>**Figure 2.** Transitions of total values of degree centralities over discourse exchanges.</figcaption></figure>

## Epistemic frames of shared epistemic agency: the differences between high and low learning-outcome groups

To examine epistemic frames of shared epistemic agency, we conducted an ENA of the discourse of students in high and low learning-outcome groups, using epistemic action codes (Figure 3). Seven codes of shared epistemic agency were plotted in a two-dimensional epistemic space, based on adjacency matrices representing the co-occurrence of the codes. A comparison of code connections in high and low learning-outcome groups revealed the following critical differences. First, we found differences in variances, explained by the two-dimensional epistemic spaces. For high learning-outcome groups, the epistemic space explained 100% of variances in the two-dimensional space (88.9% in the first/horizontal dimension and 11.1% in the second/vertical dimension). By contrast, for low learning-outcome groups, the epistemic space explained only 80% (46.3% in the first dimension and 33.7% in the second). In addition, the relationship between the epistemic and regulative aspects of actions differed in the two types of groups. In high learning-outcome groups, three epistemic actions—alleviating a lack of knowledge (ALoK), creating shared understanding (CSU), and generating collaborative actions (GCA)—were strongly linked to both regulative and projective actions. In low learning-outcome groups, although the three epistemic actions (ALoK, CSU, and GCA) were linked to regulative action, they were only weakly connected to projective action.

In addition to analyzing the differences associated with learning outcomes, we carried out a temporal analysis within each group, dividing the whole process of each group discourse into three phases (Figures 4 and 5). Based on the temporal ENA within each group, we found the following results. First, in high learning-outcome groups, the most active actions changed across the three phases, from ALoK through CSU toward GCA. In each phase, the active action was linked to regulative and projective actions. Low-learning outcome groups, by contrast, began with ALoK and continued to CSU but did not move further toward GCA. ALoK and CSU were linked to regulative action in the first and second phases; they were tied only weakly to projective action.

<figure><img src="image_49.png" /></figure>

<figure><img src="image_50.png" /><figcaption>**Figure 3.** Epistemic frames based on seven epistemic actions in shared epistemic agency— a comparison of high learning-outcome groups (left) and low learning-outcome groups (right).</figcaption></figure>
<figure><img src="image_51.png" /><figcaption>**Figure 4.** Temporal change in epistemic frames of shared epistemic agency by high learning-outcome groups: first phase (top left), second phase (top right) and third phase (bottom left).</figcaption></figure>

<figure><img src="image_52.png" /></figure>
<figure><img src="image_53.png" /><figcaption>**Figure 5.** Temporal change in epistemic frames of shared epistemic agency by low learning-outcome groups: first phase (top left), second phase (top right), and third phase (bottom left).</figcaption></figure>

Discussion

In our previous study, presented at CSCL2017, we looked only at how a jigsaw group activity improved students’ ideas involving subject matter knowledge. Although significantly more students succeeded in acquiring integrated SBF understanding, we also found big differences in learning outcomes among student groups. Only three of the twelve groups (25%) succeeded in their collaboration, a problem that has now been examined more systematically using a double-layered socio-semantic network analysis. This approach has allowed us not only to analyze what happened in the jigsaw group activity (collective knowledge advancement) but also how it happened (shared epistemic agency). The new findings, mainly obtained through an ENA, led to the following interesting interpretations of the differences between high and low learning-outcome groups.

First, our additional analysis of vocabulary term frequencies related to the SBF framework for the human immune system revealed no significant difference in the amount of subject-related discourse across the groups. The fact that the amount of discussion was the same suggests that learning outcomes were not related to how much the students talked about the study topic. Instead, the way in which they scrutinized their own ideas appears to have been the key to more productive group-learning outcomes.

Second, our ENA of students’ epistemic actions showed that the two types of groups (high and low) engaged in different types of epistemic agency. The high learning-outcome groups established more robust epistemic space (100% of explained variances) than the low learning-outcome groups (80% of explained variances). The most epistemic aspect of actions was linked to the regulative aspect of actions in the high learning-outcome groups. In addition to constructing a shared understanding, they also collaboratively generated actions based on their shared understanding to explain how vaccination protects us from infection. They took regulative and projective actions in relation to epistemic actions. In other words, they monitored their understanding and attempted to improve their ideas continuously, in order to create their own theories of the human immune system. No strong link between epistemic and regulative actions was found in the low learning-outcome groups.

Finally, our temporal ENA of the students’ epistemic agency provided a more accurate understanding of the way in which they engaged in shared epistemic agency to advance collective knowledge. The high learning-outcome groups were more engaged in alleviating their lack of knowledge during the first phase of activity—before they constructed a shared understanding. By contrast, the low learning-outcome groups engaged in two types of epistemic actions simultaneously, starting in the first phase. During the third phase, they continued to construct a shared understanding and did not take the next step toward generating collaborative actions (for example, by creating new collaborative ideas). These results suggest that there may be an optimal sequence of epistemic actions, such as alleviating a lack of knowledge before constructing a shared understanding. The high learning-outcome groups appeared to monitor themselves and proceed in accordance with a sequential process.

By analyzing learning outcomes, collective knowledge advancement, and shared epistemic agency, we were able to see how students engaged in collective knowledge advancement through their shared epistemic agency when successfully acquiring a deep conceptual understanding. As previous research (e.g., Damşa, 2014; Scardamalia, 2002) suggests, epistemic agency plays a key role in successful collaboration. The present study introduces a new finding: epistemic actions in collective knowledge advancement are likely to follow a sequential
process. This hypothesis should be further tested in future studies, which could incorporate design elements into
jigsaw instruction to help students become aware of the sequence.

References

Bereiter, C. (2002). *Education and mind in the knowledge age*. Hillsdale, NJ: Erlbaum.

Brown, A. L. & Campione, J. C. (1996). Psychological theory and the design of innovative learning environments: On procedures, principles, and systems. In L. Schauble & R. Glaser (Eds.), *Innovations in learning: New environments for education*. (pp. 289–325). Mahwah, NJ: Lawrence Erlbaum Associates.

Damşa, C. I. (2014). The multi-layered nature of small-group learning: Productive interactions in object-oriented collaboration. *International Journal of Computer-Supported Collaborative Learning*, 9, 247–281.

Damşa, C. I., Kirschner, P. A., Andriessen, J. E. B., Erkens, G., & Sins, P. H. M. (2010). Shared epistemic agency – An empirical study of an emergent construct. *Journal of the Learning Sciences*, 19(2), 143–186.

Hmelo-Silver, C. & Pfeffer, M. G. (2004). Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions. *Cognitive Science*, 28, 127–138.

Jacobson, M. J. (2001). Problem solving, cognition, and complex systems: Differences between experts and novices. *Complexity*, 6(2), 1–9.

Ma, L., Matsuzawa, Y., Chen, B., & Scardamalia, M. (2016). Community Knowledge, Collective Responsibility: The Emergence of Rotating Leadership in Three Knowledge Building Communities. In C. K. Looi, J. L. Polman, U. Cress, & P. Reimann (Eds.), *Transforming Learning, Empowering Learners: The International Conference of the Learning Sciences (ICLS) 2016, Volume 1*. Singapore: International Society of the Learning Sciences.

Miyake, N. (1986). Constructive interaction and the iterative process of understand- ing. *Cognitive Science*, 10, 151–177.

Miyake, N. & Kirschner, P. A. (2013). The social and interactive dimensions of collaborative learning. In K. Sawyer (Ed.), *The Cambridge Handbook of the Learning Sciences, the Second Edition*. (pp. 418–438). NY: Cambridge Univ. Press.

Oshima, J., Ohsaki, A., Yamada, Y., & Oshima, R. (2017). Collective knowledge advancement and conceptual understanding of complex scientific concepts in the jigsaw instruction. In Smith, B. K., Borge, M., Mercier, E., and Lim, K. Y. (Eds.), *Making a Difference: Prioritizing Equity and Access in CSCL, 12th International Conference on Computer Supported Collaborative Learning (CSCL) 2017, Volume 1* (pp. 57–64). Philadelphia, PA: International Society of the Learning Sciences.

Oshima, J., Oshima, R., & Fujita, W. (2018). A Mixed-Methods Approach to Analyze Shared Epistemic Agency in Jigsaw Instruction at Multiple Scales of Temporality. *Journal of Learning Analytics, 5*(1), 10–24.

Oshima, J., Oshima, R., & Matsuzawa, Y. (2012). Knowledge Building Discourse Explorer: A social network analysis application for knowledge building discourse. *Educational Technology Research & Development*, 60, 903–921.

Paavola, S., & Hakkarainen, K. (2005). The knowledge creation metaphor—An emergent epistemological approach to learning. *Science & Education, 14*, 535–557.

Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith (Ed.), *Liberal education in a knowledge society* (pp. 67–98). Chicago, IL: Open Court.

Scardamalia, M., Bransford, J., Kozma, B., & Quellmalz, E. (2012). New assessments and environments for knowledge building. In P. Griffin, B. McGraw, & E. Care (Eds.), *Assessment and teaching of 21st century skills* (pp. 231–300). New York, NY: Springer Science+Business Media.

Shaffer, D. W. (2017). *Quantitative ethnography*. Madison, WI: Cathcart Press.

Shaffer, D. W. et al. (2009). Epistemic network analysis: A Prototype for 21st Century assessment of Learning. *International Journal of Learning and Media, 1*(2), 1–21.

Svarovsky, G. N. (2011). Exploring complex engineering learning over time with epistemic network analysis. *Journal of Pre-College Engineering Education Research, 1*(2), 19–30.

Wilensky, U. & Jacobson, M. J. (2013). Complex systems and the learning sciences. In K. Sawyer (Ed.), *The Cambridge Handbook of the Learning Sciences, the Second Edition*. (pp. 319–338). NY: Cambridge Univ. Press.

Acknowledgments

We thank the participating teacher and students. This work was supported by JSPS KAKENHI Grant Number 16H0187.
# Families and Media Multi-tasking: Reorganizing Collaborative Learning at Home

Deborah Silvis, University of Washington, dsilvis@uw.edu

**Abstract:** As the number of tools available for learning within computer-based tasks proliferates, so too do the tasks. This paper describes how families use technology and new media for multi-tasking in order to collaboratively accomplish many routine tasks around their homes. Families employ a number of approaches to media multi-tasking, and their sociotechnical practices reflect how children learn *to* multi-task, learn *through* multi-tasking, and learn *despite* multi-tasking. I draw on these practices to explain how families manage shared- and divergent objects of activity at home, and how technology is implicated in this maintenance work. I offer a multi-dimensional framework for multi-tasking that suggests how determining what is consequential for learning with technology is often contingent on a number of tools, tasks, purposes, and people.

## Introduction

On a bright April day, after nine-year-old twins Oscar and Eddie had returned home from school, I interviewed their mom Steph about her sons' technology routines. After detailing their “daily media round”, or all the routine places people go and things they do with media and technology (Taylor, Takeuchi & Stevens, 2017), Steph described some of the activities they did as a family on a more infrequent basis. For example, she participated in several community organizations, and the twins usually accompanied her when she attended weekly or biweekly meetings. Steph gave the following account of her sons’ involvement during such family outings.

> If they want to take off the headphones and engage in the dialog at the meeting, they can participate and soak things in. Sometimes in the car on the way home they start to talk about what the adults were discussing. Even though they are [playing] on their tablets or with their headphones, they are still soaking in what's going on around them. *They are obviously multi-tasking*, because they will reengage in the conversation [emphasis added].

Later in this paper I will describe how the twins (and their mom) accomplished such multi-tasking, a phenomenon I observed across all the families I studied. This paper examines how families’ approaches to media engagement allow learners to “soak in what’s going on around them” despite multiple demands on their attention.

In any given day, families must accomplish meal prep and eating, clean-up and chores, commuting, attending school, paid work, homework, community meetings and parties, shopping, sports or music practice, paying bills, worship, play time, pet care, bedtime, and more. The work is never done, and in today's socioeconomic context, families are busier than ever (Ochs & Kremer-Sadlik, 2013). But, as Graesch (2009) suggested, “busier families are not necessarily working harder or longer hours, but instead are having to reconfigure their lives around a growing number of bids for their attention” (p. 86). How is it that families manage to do more in the same amount of time? How is technology involved in reconfiguring family life? What material and human resources must they collaboratively assemble to manage it all? This paper examines multi-tasking as one response to these questions and explores how it is part and parcel of learning in new media ecologies (Barron et al., 2009).

The current paper represents a condensed version of a longer dissertation chapter analyzing media multi-tasking in the context of family life. In what follows, I address the question: *what forms of collaborative activity does multi-tasking make possible, and how do families collaboratively contribute to what ends being consequential for learning?* Drawing on three instances of families’ media multi-tasking, I interrogate school-like notions of being “on task” and instead take the perspective that multi-tasking is intrinsic to most everyday activities. I elaborate dimensions of multi-tasking that bear on learning: how tasks involve arranging for multiple sub-tasks, multiple tools, multiple purposes, and multiple people. Analyzing instances of families’ multi-tasking across these dimensions showed how families collaboratively organize activities to learn *through*, learn *to*, and learn *despite* multi-tasking.

At stake across all these elements of multi-tasking is a question of whose perspective on a task matters
for learning. Children and adults have different views of what they want to accomplish during precious hours spent at home (Goodwin & Cekaite, 2018). Likewise, their objectives within a shared task can vary; adults may see media engagement as useful for one purpose, while for children, it serves an entirely different aim. Children frequently choose the course of their media engagement; however, doing so in ways that conform to families' shared rules and norms can necessitate children's multi-tasking. Multi-tasking then becomes a lens through which what is consequential for learners becomes visible, opening up endogenous perspectives with the potential to expand what counts as meaningful media engagement and learning and for whom (Hall & Jurow, 2015; Stevens, 2000). Viewed in this way, routine tasks are sites of on-going negotiation in families, where multi-tasking can resolve- or renew- tensions over technology use. As the current analysis will show, home is a powered place, where decisions about sociotechnical practices are charged and pivot around multiple tools, people, objectives, and tasks.

## Conceptual framework

According to some researchers, young people are both the biggest multi-taskers and the ones least likely to perform well while multi-tasking (Rosen, Carrier & Cheever, 2013). Because multi-tasking performance has consistently been found to improve with age, teaching and learning has been a major focus of multi-tasking research. And perhaps unsurprisingly, in recent years, given the proliferation of mobile media and screen-based learning designs, there has been renewed interest in multi-tasking research. Lin and colleagues (2011) have produced a number of studies of media multi-tasking (MMT), which they define in terms of dual-tasking, as “engaging in multiple media and tasks simultaneously” (P. 183), for example reading a news story while watching a video (Lin, Mills, Ifenthaler &, 2016). The bulk of literature has pointed to negative effects of media multi-tasking on studying at home (Rosen et al., 2013), classroom learning (Wood et al., 2012), and social success of young people (Pea et al., 2012).

Studies of families' multi-tasking leave the distinct impression that children either are primary distractions (for parents) or they are primarily distracted by media (Craig, 2016). While there may be some truth to this gloss on family multi-tasking, it neglects how children's learning might be involved in new forms of media multi-tasking or how learning may be supported through collaborations with other family members (Shapiro et al., 2017; Silvis, Taylor & Stevens, 2018b; Lin et al., 2016). Recently, Mejía-Arauz and colleagues (2018) have examined collaborative learning at home, distinguishing between task objectives that are met through “negotiation,” completed individually and then combined together, and those achieved “collaboratively,” through joint synchronous activity. This work points to how families' collaborative activities frequently exhibit multi-tasking during culturally valued routines like cleaning, cooking, care-giving, or conversing, though this was more normative in families from non-dominant backgrounds, such as Mexican heritage families.

In other research with families, joint media engagement (JME) may also provide a supportive collaborative context for multi-party media multi-tasking (Takeuchi & Stevens, 2011. Studies of JME have by and large looked at how multiple people arrange for learning around *single* devices or media, which are generally digital. However, in today's dynamic media ecologies, digital technologies are not the sole means of multi-tasking (e.g. Andrade, 2010). A significant amount of paper-based activities persist in homes and are interleaved with screen-based activities. Therefore, multi-tasking not only involves doing multiple tasks at once, it also represents a *multi-tooled* approach to tasks. As collaborators, whose labor is distributed, draw these materials together in task, the task itself may break down into multiple sub-tasks, with separate objectives that must be articulated and coordinated (Stevens, 2000).

The nature of task objectives is a central issue in research on multi-tasking. A single task may achieve multiple aims (Engeström, 2001). Take for example a routine activity like checking email. Because many different topics and tasks sit within different email messages, the objects of an activity such as “checking email” can range from coordinating a meeting with colleagues to getting the latest deals on produce from the local grocery store. Task objectives proliferate in complex media environments, even within a single task. Engeström suggests how “the object of activity is a moving target, not reducible to conscious short-term goals” (p. 136). There is no “task” only “task(s)” as people and objects co-construct them. Or, as Erickson (2016) has commented, “in any human interaction, there is always more than one thing going on at the same time.” Relatedly, activities that serve one person’s objectives may engage someone else in the family for a variety of different reasons, only some of which are shared among collaborators. Managing interactional “floors” (i.e. interactional spaces for establishing topics or tasks), then requires balancing dynamic tasks and topics across time and space (Erickson, 2004). Multi-tasking,
therefore, is not only a means of accomplishing multiple discreet tasks or sub-tasks with multiple tools, it is also
*multi-party* and responds to the *multiple purposes* of different participants.

Study design and methods

In this analysis, I take a situated view of learning (Brown, Collins & Duguid, 1989; Lave & Wenger, 1991) and draw on data collected in an ethnographic study of family life at home as it is currently being transformed by new (especially mobile) forms of media and technology. Participants included eighteen focal children in twelve families from diverse racial, ethnic, geographic, and socioeconomic backgrounds. Participants were recruited from local youth-serving organizations, camps, and other places where young people between the ages of nine and thirteen years old spend time; this period of development is significant for media engagement because it is about the time when children get their own devices (Rideout & Katz, 2016).

Data collection in this ethnographic study of families “daily media rounds” (Taylor, Takeuchi & Stevens, 2017) took place over two years in two separate US cities, and methods included the following: semi-structured interviews with parents and children (48 total); video recorded observations conducted during home visits, some of which were recorded by children using point-of-view cameras (i.e. GoPros®) (approx. 100 hrs); experience sampling through nightly phone calls (90 total calls), and a novel research activity for digital mapping of participants’ technology use (Silvis, Taylor & Stevens, 2018a), which was also video recorded (16 digital artifacts). To answer the research questions- how families’ media multi-tasking organizes learning arrangements- I drew on ethnographic and interaction analysis (Jordan & Henderson, 1995). I produced multi-modal transcription of the talk, gesture, gaze, coordination of body movements (or lack thereof), uptake of tools, use of space, and other aspects that played into moment-to-moment interactions (Tulbert & Goodwin, 2011).

Analytic findings

A wide variety of everyday tasks took place across families participating in the study. I analyzed a number of key instances where multiple people, tools, tasks, and aims were being configured and re-configured, as opposed to those where a child was independently engaged with a single task for a period of time (i.e. reading a book or e-book for thirty minutes without interruption, doing math homework at the kitchen table before moving on to play time). Looking across tasks that were multi-tooled, multi-party, multi-purpose and involved multiple sub-tasks, a number of patterns emerged. I found that families approaches to multi-tasking supported children learning *through* multi-tasking, learning *to* multi-task, and learning *despite* multi-tasking. These patterns reflect the dynamics of multiple dimensions of multi-tasking in-task (i.e. multiple tools, people, tasks, and aims). In what follows, I focus on three instances of children multi-tasking, emphasizing the role of collaboration in each case. Each of these examples serves as a representative case of three ways in which multi-tasking relates to learning.

Learning *through* multi-tasking using a single device

Because wired digital technologies afford endless possibilities for media engagement, a single device can organize
a space for a great deal of media multi-tasking. Katherine, a ten-year-old girl whose family had recently moved
to a large city in the Northwest US from Shanghai, was rarely without her prized possession, a MacBook Air.
According to her mom, when she was not participating in a summer program, taking ballet lessons, or making
plans to attend her new school in the fall, Katherine was inseparable from this mobile device. She relied on the
tool for a myriad of activities including: *tasks assigned by her Mom* (e.g. researching animal shelters where they
could adopt a family dog or places where they could go to explore their new city); *homework assigned by teachers*
(e.g. mythology reading, poetry writing); *preparing for school in the fall* (e.g. checking summer reading list,
studying for a gifted program placement test); *learning about topics of interest* (e.g. receiving a video from a
friend in her Kitchen Chemistry class, receiving email from a list serve about “science-y” cooking); and
*independently surfing websites for news or videos* (e.g. watching an old episode of Pokémon on YouTube, reading
articles in her news feed).

During one visit to her home, I observed how Katherine used her laptop over the course of an hour for a number of separate and simultaneous activities, a practice I call “single-device multi-tasking.” Zeroing in on this observation alone illustrates the density of media multi-tasking Katherine engaged in during more than an hour spent using her laptop. As she worked, a single device provided the means of easily multi-tasking by tabbing back and forth between active windows, continuously checking the status of uploads, or opening new windows and programs in order to accomplish a task at hand. When I joined her in her bedroom work site on this day, Katherine was already busy working on a video she had animated in QuickTime the previous evening. She was uploading it to a site called Magisto.com, that hosts videos for sharing or publishing on social media. While this underlying activity progressed in the background, she busied herself with a number of other tasks (Figure 1).
<figure><img src="image_54.png" /><figcaption>**Figure 1.** Single device multi-tasking. Timeline (in minutes) of all the separate, overlapping, tasks Katherine engaged in during a single observation. *Except for Mythology reading, all of these tasks utilized only her laptop.</figcaption></figure>

After about thirty minutes (and at least seven discrete tasks), Katherine began composing a poem in a Google Doc that she shared with her summer program writing teacher, who provided her with virtual feedback. A few minutes into her poetry writing, Katherine opened Google Images in a separate tab, something she said she sometimes did “for inspiration.” She referred to several images of poems, tabbing back and forth between these and the one she was authoring, and eventually positioning multiple windows in order to simultaneously view multiple poems on screen. When she hit an impasse in her writing, she opened up another tab to search on Thesaurus.com, which she identified as her “favorite website.” In the course of writing the poem, she managed the display by juxtaposing- and tabbing back and forth between- these multiple tabs and windows. Configuring multiple resources on-screen enabled her to navigate sub-tasks in a way that supported learning through multi-tasking.

After working on the poem for fifteen minutes, she resumed watching an episode of Pokémon on YouTube that she had interrupted to write the poem, and later moved on to do some research on *Animal Farm*, which was on her summer reading list for school next fall. However, she kept open the multiple tabs she had used to write the poem, so it was possible to return to the task later on. Katherine's multi-tasking led her across multiple sites and platforms to engage in multiple tasks using a single device. She was able to learn about poems, literature, vocabulary, and much more through her multi-tasking and also with the help of virtual collaborators. In the course of my observations, she interacted with her friends, her mom, and her dance teacher via email. At any given time, a wired laptop gives young people multiple avenues for interaction, and who they choose to connect to shifts as the focal task changes. Consequently, learning through single-device multi-tasking is not adequately characterized as a solo activity, nor does it simply serve as a means of individual efficiency or personal productivity. Media multi-tasking is mutually monitored by multiple collaborators (Goodwin & Cekaite, 2018)- albeit virtually and asynchronously. Even when a single person uses a single device, multi-tasking demands- and is a response to demands for- collaborative learning across tools, people, and tasks.

## Learning to multi-task by mixing tools and negotiating objectives

For Katherine, a single tool became a site for organizing learning through multi-tasking across heterogeneous objects of activity. Conversely, it was often the case for the families I studied that a single *task* became a site for learning to multi-task across multiple tools and task objectives. In today's media ecologies, families have many options for equipping and jointly accomplishing routine tasks. Often, family members' purposes for performing tasks vary and need to be negotiated amongst people and devices. Placing a multi-tasking lens on a single task can surface the often hidden sociotechnical work required to stabilize a task so that people can jointly accomplish their aims with multiple technologies (Star, 1990; Stevens, 2000). Learning to multi-task becomes necessary when task-objectives diverge, when tools are distributed, or when one person delegates their task to another.

Such was the case when Natalie, an eleven-year-old who lived with her grandmother and mother Gina near a
large Midwestern US city, helped her mom complete an Excel spreadsheet containing information from a survey Gina had distributed at work at a local service organization. Gina introduced the task to Natalie as work they could do together, enlisting her daughter in her own work as a valued coparticipant. Barron and colleagues (2009) referred to this arrangement as “parents as employers,” a situation that arises when parents “entrust their technologically skilled children to perform technical services for them” (p. 69). Rifling through the questionnaires, Gina told Natalie that they needed to make a spreadsheet and that Natalie could do the data entry. When Natalie was unsure of what this meant and feigned resistance, claiming it was a weekend and she shouldn’t have to work, her Mom reassured her by playfully smacking her arm and announcing that Natalie was going to learn something new. What for adults may seem mundane (e.g. building a spreadsheet, doing data entry) presents productive challenges and novel tasks for young learners, who may have very different task objectives and motives than their adult collaborators.

It is also important to note here that Natalie was recruited *by her Mom* to do this work, which is a different starting point for the delegation of young people's computer-based work that typically originates outside the home in school. As opposed to homework assigned by a teacher, where a single task is undertaken for the putative purpose of completing an assignment (or earning a grade), building an Excel spreadsheet was more multi-purpose; this contributed to the pair approaching their task with divergent objectives. Gina reported that she often recruited Natalie into tasks for her job if technology was involved, in order to teach her daughter new technical skills. For Natalie, spending time with her Mom during their free time on a Saturday was a powerful motive for completing data entry, a task that adults may view as thankless grunt work and some children might regard as a pointless chore. Learning to build a spreadsheet entailed learning to negotiate multiple people's objectives in-task and to coordinate their separate sub-tasks. Therefore, in addition to learning to use Excel, Natalie was also learning to multi-task.

<figure><img src="image_55.png" /><figcaption>**Figure 2.** Multiple tools within a task. (Left to right) Natalie's mom reports information from paper-based surveys; Natalie enters data into Excel spreadsheet; they collaborate over the screen; they collaborate over paper.</figcaption></figure>

Learning *to* multi-task meant drawing on multiple material means of mediation as well. As they began the spreadsheet together on the couch in the living room, Gina held the paper surveys, while Natalie was responsible for computer data entry (Figure 2). Despite the proliferation of digital means of doing domestic tasks, there is continued need to mix multiple types of tools together to accomplish a task (Silvis et al., 2018b), and paper is still a widely used resource for collaborative knowledge work (Brown & Duguid, 2017). The mixing of technologies within a single task was pervasive in children's activities across all the families we observed; this spreadsheet multi-tasking literally spread the task out across sheets of paper, software, and screens and fractured the single task into sub-components made visible through an emergent division of labor. Learning to build a spreadsheet meant learning to multi-task by drawing on- and drawing together- multiple material means and multiple learning objectives.

## Learning *despite* multi-tasking as domestic complexity multiplies

The complexities of maintaining everyday routines and organizing family life are compounded when multiple people employ multiple technologies across multiple tasks, timeframes, and spaces. A family of three, with jobs, hobbies, school, and close family who lived nearby, multi-tasking was typical in the Hernandez household. This was exemplified one day after school when nine-year-old twins Eddie and Oscar completed a homework assignment with their mother's assistance. In this example, multi-tasking was multiplied through the complex spatiotemporal and material arrangements, whereby people accomplish a number of different tasks together at home. When I arrived to observe them one day after school, Oscar and Eddie were already immersed in their typical after-school routine: watching TV, completing homework, and playing video games. Their homework
assignment on this particular day was an on-going project in their language arts class that asked them to create a book cover for their favorite book. Both Eddie and Oscar had chosen to create book covers based on *Pokémon Deluxe Essential Handbook*, an encyclopedia of over seven hundred Pokémon, thoroughly dog-eared and annotated by the twins, who each had their own copy.

Rounds of working on homework, which the boys completed primarily at the kitchen table, were punctuated by “technology breaks,” a strategy their family implemented in the afternoons each day after school. The twins’ breaks occurred at different times, creating a dynamic task environment in which tasks were not entirely distinct. For example, the TV stayed on in the background while they worked on their homework, compelling the twins to engage in dual-tasking and splitting their attention during tasks (Lin et al., 2011). For his part, Eddie designed and colored his book cover illustration while watching cartoons, and later continued watching TV while taking a technology break and playing *Angry Birds* on his tablet. However, with multiple people engaged in multiple tasks simultaneously, multi-tasking quickly multiplied. The living room was usually reserved for technology breaks, but Steph made an exception so that Oscar could research a Pokémon illustrator for the back flap of his book cover at a moment when Eddie’s work crowded the kitchen table. Oscar and Steph sat comfortably together in a side-by-side configuration, co-viewing her laptop screen while he worked (and stole glances at the TV), a prototypical configuration of bodies during JME (Takeuchi & Stevens, 2011).

As in any complex learning environment, there are always multiple things going on at once (Erickson, 2016). While the twins did their homework, their mom was busy coordinating the delivery of new furniture for their bedroom. Steph had told the boys that she wanted them to each tip the movers one dollar for assembling their furniture, and she placed two one-dollar bills in the threshold between the kitchen and living room, directly in the pathway the twins would likely travel (Figure 3). She likened this to tipping the pizza guy, and she emphasized to them that “I want you to tip them because they’re providing a service.” The boys meanwhile remained unresponsive and continued gazing at the TV from the kitchen table where they worked. A short time later, Oscar and Eddie raced down the hall after snatching up the bills, and they diligently tipped the movers who subsequently departed. Mom immediately noticed that the movers had left their drill in the entrance hallway and chased after them down the driveway. Eddie wryly commented “maybe that is a tip for us,” referring to the drill. This re-voicing of his mother’s task directive indicates how he had heard and understood her words in a way that allowed him to recycle them in an entirely different context (Goodwin, 2018). Whereas, several weeks prior, Steph had reported how the twins were able to “soak things in” despite their multi-tasking, in this instance they demonstrated how this looks in practice. Despite their media multi-tasking, Eddie and Oscar learned valuable lessons about what it means to contribute to household routines and collaborations, where multiple tasks vie for people’s attentional resources.

<figure><img src="image_56.png" /><figcaption>**Figure 3.** Multiplying media multi-tasking. Mom and Eddie both discuss “tipping” the delivery guys, despite the demands of multiple tasks, people, and tools in their home media ecology.</figcaption></figure>

## Conclusions and implications

Across these three cases, I have shown how media multi-tasking supports learning at home and some forms that multi-tasking can take, including *single-device multi-tasking*, *mixing materials and multiple purposes within a single task*, and *multiplying multi-tasking* by adding more tasks and people to the task setting. I have used these examples to elaborate how children learn *through*, learn *to*, and learn *despite* multi-tasking. Families’ designs for learning at home support these forms of multi-tasking, and this takes sustained interactional work and
intergenerational collaboration. Media multi-tasking is common in families, and we can see vividly how multi-tasking multiplies quickly as people move themselves and their tasks around their homes. Even in the case of Katherine and Natalie, relatively stable (seated) sites of single tasks can quickly expand into sub-tasks using networked devices.

Prevailing perspectives on multi-tasking caution against it, for how it distracts learners, divides attention (and families), and threatens successful task completion. As a response, this analysis contributed to a different perspective on multi-tasking: media multi-tasking is a significant and pervasive sociotechnical practice, and children engage in many forms of multi-tasking in diverse learning environments. I have suggested a multi-dimensional model for multi-tasking that supports analysis of the multiple tasks, tools, objectives, and people entailed by many everyday learning activities. Understanding how these are multiply configured from multiple perspectives contributes to an expansive view of multi-tasking, widening the lens on what counts as valuable learning arrangements, where and when this takes place, and who decides.

Whereas conventional multi-tasking research finds that adults are better than children at multi-tasking and then uses this to justify cautionary narratives about children’s media multi-tasking, I am arguing that adults are important supporters of learning through, to, and despite children’s multi-tasking. Because most of us do frequently multi-task, it is worth considering how children come to be multi-taskers and to design for this. Simple exposure to multiply mediated learning environments is insufficient for learning to multi-task; Natalie was quite proficient using her laptop but needed her Mom’s assistance to learn to multi-task on it. As with children’s development of other technological competencies, learning to multi-task benefits from parental support, and is a sociotechnical achievement (Barron et al., 2009).

In the current analysis, I treated home environments as especially conducive to learning while multi-tasking, and I focused on families’ designs for multi-tasking. In response to anxious or nostalgic narratives that eschew multi-tasking as a valued learning practice, I believe much can be learned from understanding families’ approaches to learning through, to, and despite multi-tasking. At the same time, I do not see these three approaches to multi-tasking as entirely distinct. It was often the case that learning to manage multiple tabs on a screen, for example, also involved learning about whatever content was contained in those windows. Furthermore, families’ multi-tasking organized learning such that background processes- like watching TV, monitoring other family member’s activities, or uploading a video- intermittently became the central focus of activity. This suggests that task objectives at home are better characterized as moving targets than as categorically “focal” or “background” tasks.

Distinguishing background from foreground is a paradigmatic problem of perspective. What one person (i.e. a parent or teacher) thinks is important to do at any given time may not be valued by children and vice versa. Keying into how tasks serve multiple purposes points to how assigning and distributing tasks over time, space, people, and materials is frequently a powered practice. While school-like versions of attention often emphasize individuals “staying on task,” home media ecologies are rich sites for multi-party multi-tasking where the objectives of tasks- and their means of completion- originate from more equitably distributed starting points. My analysis supports the suggestion by Lin and colleagues (2016) that collaborative learning is enhanced by multi-tasking because of the excitement of engaging in more than one task at once, a learning configuration often forbidden in formal learning settings. Instead of barring learners from using multiple media and devices to accomplish tasks in school, designs for learning might incorporate more people, technologies, and tasks in order to simulate how children immerse themselves in dynamic home media ecologies and to stimulate interest-driven learning.

## References

* Andrade, J. (2010). What does doodling do? *Applied Cognitive Psychology, 24*:100-106.

* Barron, B., Martin, C. K., Takeuchi, L., & Fithian, R. (2009). Parents as learning partners in the development of technological fluency. *International Journal of Learning and Media, 1*(2), 55-77.

* Brown, J.S., Collins, A., Duguid, P. (1989). Situated cognition and the culture of learning. *Educational Researcher, 18*(1), 32-42.

* Brown, J.S. & Duguid, P. (2017). *The social life of information*. Boston, MA: Harvard Business Review Press.

* Craig, L. & Jenkins, B. (2016). The composition of parents’ and grandparents’ child-care time: gender and generational patterns in activity, multi-tasking, and co-presence. *Aging & Society, 36*, 785-810.

* Engestrom, Y. (2001). Expansive learning at work: Toward an activity theoretical reconceptualization. *Journal of*
education and work, 14:1, 133-156.

Erickson, F. (2004). *Talk and social theory: Ecologies of speaking and listening in everyday life*. Polity Press.

Erickson, F. (2016). Learning to Look and Listen: Group Viewing Session. *Spencer Foundation Conference*, Arizona State University.

Goodwin, C. (2018). *Co-operative action*. NY: Cambridge University Press.

Goodwin, M.H. & Cekaite, A. (2018). *Embodied family choreography: Practices of control, care, and mundane creativity*. London: Routledge.

Graesch, P.A. (2009). Material indicators of family busyness. *Social Indicators Research, 93*, 85-94.

Hall, R. & Jurow, S. (2015). Changing concepts in activity: Descriptive and design studies of consequential learning in conceptual practices. *Educational Psychologist, 50:3*, 173-189.

Jordan, B. and Henderson, A. (1995). Interaction analysis: Foundations and practice. *The Journal of the Learning Sciences 4* (1), 39-103.

Lave, J. & Wenger, E. (1991). *Situated learning: Legitimate peripheral participation*. Cambridge, UK: Cambridge University Press.

Lin, L., Lee, J. & Robertson, T. (2011). Reading while watching video: The effect of video content on reading comprehension and media multitasking ability. *Journal of Educational Computing Research, 45*(2).

Lin, L., Mills, L.A. & Ifenthaler, D. (2016). Collaboration, multi-tasking and problem solving performance in shared virtual spaces. *Journal of Computers in Higher Education, 28*, 344-357.

Mejía-Arauz, R., Rogoff, B., Dayton, A., and Henne-Ochoa, R. (2018). Collaboration or negotiation: Two ways of interacting suggest how shared thinking develops. *Current Opinion in Psychology, 23*;117-123.

Ochs, E. & Kremer-Sadlik. (2013). Introduction. In E. Ochs & T. Kremer-Sadlik (Eds.), *Fast-forward Family: Home, Work, and Relationships in Middle-Class America*. Berkeley, CA: University of California Press.

Pea, R., Nass, C., Meheula, L, Rance, M, Kumar, A., Bamford, H... & Zhou, M. (2012). Media use, face-to-face communication, media multitasking, and social well-being among 8-12-year-old girls. *Developmental Psychology, 48:2*, 327-336.

Rideout, V. & Katz, V. (2016). *Opportunity for all? Technology and learning in lower-income families*. A report of the Families and Media Project. New York: The Joan Ganz Cooney Center at Sesame Workshop.

Rosen, L.D., Carrier, L.M. & Cheever, N.A. (2013). Facebook and texting made me do it: Media-induced task-switching while studying. *Computers in Human Behavior, 29*, 948-958.

Shapiro, B., Hall, R., & Owens, D. A. (2017). Developing & using interaction geography in a museum. *International Journal of Computer-Supported Collaborative Learning, 12*, 377-399.

Silvis, D., Taylor, K.H., Stevens, R. (2018a). Community technology mapping: Inscribing places when ‘everything is on the move.’ *International Journal of Computer-supported Collaborative Learning, 13*(2), 137-166.

Silvis, D., Taylor, K.H. & Stevens, R. (2018b). Reassembling home-work: Mixing “newer” and “older” technologies in home learning environments. In J. Kay & R. Luckin (Eds.), *Proceedings of International Conference of the Learning Sciences (CLS) 2018*. London, England: International Society of the Learning Sciences.

Star, S.L. (1990). Power, technology, and the phenomenology of conventions: On being allergic to onions. *The Sociological Review, 38*(1), 26-56.

Stevens, R. (2000). Divisions of labor in school and in the workplace: Comparing computer and paper-supported activities across settings. *The Journal of the Learning Sciences, 9*(4), 373-401.

Stevens, R. (2010). Learning as a members phenomenon: Toward an ethnographically adequate science of learning. *National Society for the Study of Education, 109:1*, p. 82-97.

Takeuchi, L., & Stevens, R. (2011). *The new co-viewing: Designing for learning through joint media engagement*. A report of The Joan Ganz Cooney Center at Sesame Workshop and LIFE Center.

Taylor, K.H., Silvis, D. & Stevens, R. (2017). Collecting and connecting: Intergenerational learning with digital media. In E. Gee, L.M. Takeuchi & E. Wartella (Eds.), *Children and Families in the Digital Age: Learning Together in a Media Saturated Culture*. NY: Routledge.

Taylor, K.H., Takeuchi, L. & Stevens, R. (2017). Mapping the daily media round: Novel methods for understanding families' mobile technology use. *Learning, Media and Technology, 1*-15.

Tulbert, E. and Goodwin, M.H. (2011). Choreographies of attention: Multimodality in a routine family activity. In J. Streeck, C. Goodwin, & C. LeBaron (Eds), *Embodied Interaction: Language and Body in the Material World*. Cambridge, UK: Cambridge University Press.

Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D. & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. *Computers & Education, 58*, 365-374.
# Designing Representations in Deeply Disciplinary Educational Games

Ravit Golan Duncan, Rutgers University, ravit.duncan@gse.rutgers.edu

Frieda Reichsman, The Concord Consortium, freichsman@concord.org

Kiley McElroy-Brown, The Concord Consortium, kmcelroy-brown@concord.org

Veronica Cavera, Rutgers University, veronica.cavera@gse.rutgers.edu

**Abstract:** Educational games are common in classrooms and have been extensively studied in multiple domains. The *Geniventure* game was developed to support student learning of core concepts in genetics via challenges that engage students with genetic phenomena at the molecular level. A core feature of the game is that it simulates behavior of molecular entities (genes, proteins, organelles) with high disciplinary fidelity to how these mechanisms really operate in the cell. In this sense, it is deeply disciplinary. The commitment to disciplinary fidelity presents design challenges regarding the ways in which entities, activities, and mechanisms are represented and manipulated in the game across the biological organization levels. We discuss five distinct types of design challenges that we identified based on data from focus groups with students who played the game and provide design heuristics for addressing these challenges.

## Introduction

The development and research on educational games has increased substantially in the past decade (for recent reviews see Clark, Tanner-Smith, & Killingsworth, 2016; Ravyse, Blignaut, Leendertz, & Woolner, 2017). This work has identified key features that make games more engaging and productive for learning such as having them be pleasantly frustrating, involving well-ordered problems, and providing the player with agency, to name a few (Gee, 2005). This research has also provided some guidelines for the design of successful educational games (e.g. Kafai, 2006; Winn, 2009).

In addition to these critical aspects of game design, we wish to draw attention to considerations of designing *deeply disciplinary* games and in particular designing representations of key disciplinary entities and processes in these games. We use the term *deeply disciplinary* to refer to games that aim to engage students with key disciplinary phenomena and in which the game play involves manipulating entities and mechanisms within these phenomena that may be unfamiliar to students. This construct draws on the notion of *conceptually integrated* games, which integrate domain concepts and relationships into the core mechanics and representations of the game (e.g. Habgood & Ainsworth, 2011) and expands it to include a stronger emphasis on the representation of domain-specific entities and mechanisms in ways that maintain high fidelity to the structure, function, and interactions of these entities. For example, the game *Geniventure* engages students in the genetic study of drakes, the model organism for dragons; students directly manipulate chromosomes, genes, and proteins in order to change the drakes' traits. The representations of these genetic entities in the game are therefore a central component of making the phenomena accessible on the one hand, while maintaining fidelity to the disciplinary entities represented on the other. The tension between accessibility of the representations and their disciplinary fidelity generates interesting challenges in terms of design decisions regarding the representations and the linking of representations across the multiple user interfaces of the game.

Research on students' understanding of disciplinary inscriptions (such as graphs, models, equations) provides some insights that can help inform these design decisions. For example, zoom-in features, alternative perspectives, and control of speed of animations/simulations can help facilitate perception and comprehension of important disciplinary entities and processes (Tversky, Bauer Morrisony, & Betrancour, 2002). Similarly, tighter coordination and coherence between multiple representations (especially when visualizing phenomena at the macro and molecular levels) is also important to reducing cognitive load and helping students make appropriate connections (Kozma, 2003). In addition, it is often difficult to provide, within the representations, all the needed conceptual and social resources that can help convey the utility and purpose of disciplinary representations/inscriptions (Roth & Bowen, 1999). However, much of this research was conducted in the context of students using and creating inscriptions within, mostly traditional, instructional pedagogies such as the use of inscriptions, animations, or simulations to explain a science concept. Moreover, the research on using computerized models for understanding molecular-level representations and ideas predominantly comes from chemistry (Barnea & Dori, 1996; Williamson & Abraham, 1995; Wu et al., 2000), whereas only a few studies deal specifically with molecular biology (e.g. Pallant & Tinker, 2004; Tsui & Treagust, 2004). Therefore, while
relevant, this body of work does not address the specific challenges associated with using disciplinary representations in game contexts, which are not as conducive to traditional ways of scaffolding and providing information. In such environments, engagement with disciplinary representations and phenomena is in the context of a game challenge and students figure out the key ideas through game play. Exiting the game play to “instruct” or explain the representations or processes can be distracting and problematic in terms of motivation.

Here we present findings regarding middle school students’ understandings of representations and processes in a genetics game and their implications for the design of representations in deeply disciplinary games. The online game, *Geniventure*, is a fictional and narrative-based game in which students are enlisted to help breed dragons with traits that can help them in the struggle against a nearby kingdom’s attacks. As the game opens, students find themselves inside the Mission Control room of a secret underground headquarters for dragon development and breeding. A diverse cast of characters presents students with a series of challenges that address a need for specific traits and an understanding of how those traits are achieved via genetic instructions. Students “travel” to different rooms in the underground compound to breed drakes, manipulate genes, or zoom into cell simulations to interact directly with proteins and DNA. Student actions in a challenge are tracked and upon completion of a challenge students are awarded “crystals” based on how well they performed the task.

The game is designed to help students develop core understandings in genetics including patterns of inheritance (alleles on paired chromosomes control each trait), how random assortment of chromosomes into sex cells results in the observed probabilities of the expression of traits, and how genes on those chromosomes code for proteins that bring about those traits. In this article, we focus predominantly on the game challenges involving proteins (described in more detail below). We conducted focus group interviews with three groups of students who played the protein challenges with the aim of finding out how they understood the nature and role of proteins in mediating the genetic traits involved in the challenges. Our research question is: how are students interpreting and responding to the different representations of molecular entities, mechanisms, and levels of the phenomena (genes, protein, cell, trait) embodied in the game? We next describe the online game, *Geniventure*, and some of the key design decisions that were made in developing the representations of the phenomena.

## The Geniventure game: Protein challenges

The protein challenges are designed to engage students with the mechanisms that connect genes and traits. Genes are DNA sequences that specify instructions to the cell for building protein molecules. Proteins are the workhorses of the cell; they have many different functions, and their functions are closely tied to their structure. Seemingly small differences in the DNA sequences of genes can have profound effects on an encoded protein's structure and in turn on its function. Different versions of a gene, called alleles, produce different traits by virtue of these differences in the structure and function of the proteins they encode. Thus, proteins are an essential link for productive reasoning about how genes confer traits on individuals.

Proteins are often characterized by experts as molecular machines (e.g., Goodsell, 2009). This is a useful analogy since students understand that machines have functions, and that the form or shape of a machine is essential to its function. In designing the look and feel of proteins in the game, we chose to emphasize their machine-like qualities by using simple machine structures like gears, and we designed their shapes to match

<figure><img src="image_57.png" /><figcaption>Figure 1. The size challenge.</figcaption></figure>
very obviously with their function. We used color gradients and softened edges to convey their biological
nature; influenced by Goodsell’s (2009) use of color in illustrating actual protein structures.

The protein mechanisms we targeted in the game are situated in drake scale cells (analogous to human
and other animal's skin cells) and focus on the production and transport of melanin. We chose drake color as a
trait for our protein focus because it is a readily discernable trait and it is constantly and actively maintained by
cells through protein-based mechanisms. In the game, students are exposed to the proteins "doing their jobs"
and then are given the opportunity to either interfere with what the proteins are doing or to help them along the
way, all with the goal of changing the color that the cell produces. The melanosome organelle stores melanin
and changes in color based on how much melanin is present. In the game, there are two types of challenges that
involve proteins. In the first, which we call the "size" challenge, students try to create smaller or bigger
melanosomes. We have reduced the complexity of the mechanism to consist of two proteins, both of which are
active in real organisms. One is an enzyme that synthesizes melanin, represented in the game by a set of gears
that can assemble melanin, and its "helper protein" that provides stability to the gear enzyme, represented by a
shaft that holds the gears together (see Figure 1). The proteins build melanin, which is represented as stars, by
assembling the triangular building blocks (white triangles) into melanin stars. Melanosomes change color from
orange to gray depending on how many melanin stars are in them; bigger melanosomes are darker gray and
smaller melanosomes appear orange. Students work through several instantiations of the size challenge in which
they have to either reduce the size of the melanosome (to make an orange drake) by breaking the melanin stars
and slowing down the gears, or increase the size of the melanosomes (to make a gray drake) by helping to
assemble the gear-shaft complex so it works faster to make melanin stars and thereby grow the melanosome.

In the second challenge, which we call the “gates” challenge, students try to create shiny versus
dull/matte drakes (that can be either orange or gray). Shininess (or sheen) is a function of not having any color
(no melanosomes) in the outer layer of scale cells, causing those cells to better reflect light and making the
scales look shiny. Melanosomes can travel to the outer scale layer through gated channels. These channels are
plugged by a protein that selectively lets melanosomes through. The protein is represented by a corkscrew-like
structure with a pink tip that can sense the incoming melanosome (see Figure 2). In shiny drakes, the protein
plug is not functioning properly and does not ever open the channel; therefore, no melanosomes can get to the
outer layer and the scales are shiny. Making the outer layer of scales actually look shiny in the game was a
drawbacks of this design, and there did not seem to be a simple way of showing sheen. Therefore, we opted for
a different solution—having a *sheen indicator bar* positioned at the top of the field of play, within the outer
scale layer. When the indicator is “full” it means there are lots of melanosomes in the outer layer and the scale is
non-reflective (matte); when the indicator is empty, the scale is shiny. In this challenge, students are tasked with
changing drakes from shiny to matte or vice versa by plugging or unplugging the channels.

<figure><img src="image_58.png" /></figure>

D = Sheen indicator bar E = Protein plug

<caption>Figure 2. The gates challenge.</caption>

Both challenges involve students trying to change the color or sheen of the drakes from a provided
initial state to a required target state. To help students keep track of their progress (the extent to which the drake
is changing) we developed a Heads Up Display (HUD) that includes three elements: a) the initial (start) state of
the drake shown as both a thumbnail of the drake and a thumbnail of a cross-sectional view of the skin tissue; b) the current state of the drake with a thumbnail of the drake only, that actually updates based on progress in the game; and c) the target state shown as dual thumbnails of the drake and cross section of skin. The HUD appears along the right side of the field of play (see Figure 1 & 2) and is present in both challenges.

# Methods

## Study context

The game was played by n=51 middle school students in the context of an eight-week summer program hosted by a community center in a metropolitan city in the North East. The program was open to all middle school students in the city yet the majority of the participating students attended the local public middle school where the program was held. This 6th–8th grade middle school is ethnically diverse: 38% Black, 25% White; 16% Hispanic, 10%, Asian; with 43.4% economically disadvantaged students. The genetics program in which the game was used ran for 90-minute sessions twice per week and was led and instructed by the Middle School Program Leader and STEM Specialist of the community center. One or two researchers were also present during the genetics programming to assist the instructor. Each session was structured to include some hands-on instructional activities in which students explored various aspects of genetics—for example, students built catapults using normal and ‘mutated’ instructions to see how changes to the instructions can result in changes to the catapults shape and function. These activities were often followed by whole class discussions about the biological meaning of the activity and how it relates to genetics. Students typically spent about 30 minutes, on average, playing the online game during these sessions. The protein challenges, which are the focus of this article, occurred in weeks 3–5 of the summer program. We wish to note that due to various logistical challenges, frequent absences of students, and varying level of engagement of students, the instructor and research team were not able to implement the planned curriculum with high fidelity. Thus, some of the activities and resources that were expected to support student understanding during game play were not fully implemented. While this was a general problem for the ongoing research, we feel that the information from the focus groups is still highly valuable in highlighting issues with the game; some of which may be ameliorated through curriculum activities while others likely require revisions to the actual game design.

## Data collection and analysis

The focus group interviews occurred in week 4 of the program. Student groups were formed based on the game challenges that the students had completed. Focus groups of students who did not reach the protein challenges were not included in this analysis. A total of 9 students participated in the three focus groups included in our analysis. The focus group interview lasted between 19–26 min and involved showing students screenshots of interfaces in the protein game and asking them questions about specific entities or processes occurring in those interfaces such as: “how close is the person playing the game to winning?” “what would you do to make the dragon darker?” “what would you click on next?”. Students took turns answering each question and the interviewer made sure none of the students had anything else to add to the discussion before moving on to the next question. Focus group interviews were transcribed verbatim and analyzed by the research team (authors).

Analysis involved viewing the videotapes together and identifying episodes of interest (e.g. students misunderstanding a representation, students explicitly expressing confusion, etc.). While we cannot claim to have identified all potential problems with the game, and in some cases only one student in the group struggled with a particular issue, we can assume that if these issues happened once, they may happen again. Therefore, we describe the issues that came up without making claims about their prevalence among student players. We grouped the issues we identified under five themes that we see as being potentially relevant for game design in other contexts and discuss them in the results section. We then offer some heuristics for designing representations for deeply disciplinary games that could help address the general problems we identified.

# Results

Before we discuss specific themes of problems with the game we wish to note that overall the students did seem to interpret representations, game phenomena, and game play as intended by the designers. Many of the features that we had hoped they would notice about the representations of genes, proteins, and traits in the game were indeed attended to by the students. It was also clear that students who had more prior knowledge in genetics, which came through in the interviews without us explicitly asking questions about genetic concepts, understood the game phenomena, mechanisms, and entities better than students who did not. This is not surprising and has been documented extensively in prior research (Cook, Wiebe, & Carter, 2007; Kindfield, 1994).
The five thematic problems we identified are: (1) incongruence between phenomenological knowledge and representations, (2) unintended distractors, (3) non-salient ontological distinctions, (4) misinterpretation of linked representations, and (5) confusing game progress and disciplinary process indicators. We discuss them in turn, by first describing the problem as it arose in the focus groups, then providing our interpretation of what is the specific problem is a case of, and lastly discussing some possible solutions for this problem in the game.

## Theme I: Incongruence between phenomenological knowledge and representations

As we noted earlier, the building blocks of melanin (white triangles) can be assembled into melanin “stars”, which are initially light orange. As they accumulate, however, they become darker—as does the overall melanosome (Figure 1). In one of the initial challenges, students are tasked with making a darker gray drake by interacting with the process of melanin star production in the melanosomes of a lighter orange drake. One of the students in the focus groups noted a confusion he initially had when playing this challenge. He assumed that *breaking* the bright orange stars would lead to darker melanosomes. We interpret this situation as the student drawing on phenomenological primitive-like knowledge (diSessa, 1998) of the sort- “less bright means dark” and acting on it to break apart (remove) the bright-looking stars. While the student eventually recognized that the strategy of breaking stars had the opposite effect (smaller melanosome), the student’s initial interpretation of the representation (bright stars) and its relation to the phenomenon (skin color of drake) is important. This is because the student’s initial reaction was intuitive and yet counterproductive given the specific representations used. We argue that in general terms this is a problem of having representations and/or processes that behave in ways that counter, or are incongruent with, students’ phenomenological knowledge. By this we do not mean that the representations are complex or unfamiliar, or that the processes are counterintuitive. Rather we mean that the actual choice of color (or other feature) of the representation, from a semiotic perspective, is incongruent with students’ knowledge of how entities behave in the world at a basic phenomenological level. The decision to make the stars light orange on a dark blue background was driven by the aesthetics of having the stars stand out as brighter objects on the screen. In hindsight, this was problematic in that it cued students to take an intuitive but unproductive action. Fortunately, the problem also has a fairly simple solution of changing the background color of the field of play to a lighter color such that the stars are not so bright that they appear to generate light.

## Theme II: Unintended distractors

Research has shown that students, especially those with low prior knowledge, tend to select the most noticeable features of the representations for further processing and may ascribe meaning to un-important features of representations when mapping between macro and molecular representations (Cook, Weibe, & Carter, 2007; Seufert, 2003). While we found that students did notice most of the relevant details of entities and processes in the game, they also noticed and attempted to interpret details that were not relevant. For instance, in the melanosome size challenges, when a melanosome is small the triangular building blocks of melanin tend to cluster in a higher concentration around the melanosome; when the melanosome is larger the same number of triangles are still there but seem less concentrated due to the larger circumference of the melanosome. This is a randomly occurring situation that is an artifact and not a feature of the representation. However, one student assumed that the discernable clustering of triangles around the melanosomes as somehow contributing to the change in melanosome color. In general terms, this is a problem of unintended (unforeseen) and sometimes unavoidable distractors in a complex disciplinary display of mechanisms. The clustering of triangles, while not a designed behavior per se, is nonetheless reflecting the real-world complexity of the mechanisms and the random nature of particle movement in the cell. A solution therefore cannot be to avoid this occurrence. The point we wish to make is that student will attend to all features of the representation, those that were deliberately engineered as well as those that were not (and are meaningless). Our solution here will be to try and alter the underlying dynamics of the representation such that the clustering behavior is less frequent.

## Theme III: Non-salient ontological distinctions

In the game, there are several distinct kinds of entities such as proteins, their substrates (triangles) and their products (stars). As noted above, we used representational elements that resemble little machines to convey the functional aspect of proteins (as molecular machines that carry out functions in the cell). In contrast, we used simpler geometrical shapes to represent protein substrates and products. Our attempts to convey this ontological distinction through semiotic cues were not always interpreted as intended by students. While students did notice the more complex and gear-like shapes of proteins, they did not view these as inherently different kinds of things from the stars and triangles, and did not realize the biological significance of the differences. One might argue that this is not surprising given that students lacked prior knowledge in the domain and therefore were unlikely to understand the biological meaning of the differences in representations. We agree that students’
domain knowledge plays a role here, however, if the game is intended to help novices lean about the biological significance of key entities and mechanisms in genetics then finding ways to convey important ontological distinctions in the discipline is an important user interface design goal. It is not clear to us how we can necessarily improve the representations in order to make the meaning of the distinctions more salient. The solution to this problem will likely involve adding information to the game through hints or other cues that help students make sense of the differences they are already noticing in the representations of entities. Or it may be that the sense making needs to occur in the context of group/class discussion in which the teacher facilitates student reflection on the differences and what they mean.

## Theme IV: Misinterpretation of linked representations (HUD)

A central representational feature in the protein challenges is the HUD, which provides real-time information about the current color of the drake relative to the initial state of the challenge and the target state. The HUD provides representations both at the cell level, using a cross sectional skin tissue view, and at the whole organism level by displaying a thumbnail of the drake (see Figure 1). The “current state” cell and drake representations in the HUD are linked to the field of play. We identified two representational problems with the HUD design. First, some students did not understand that the centered part of the HUD was representing current state. They instead thought it was representing an intermediate state between the initial and target states and therefore when shown a screenshot including the HUD, they struggled to determine how far along the player was on that particular challenge. This type of confusion of the HUD was surprising to us given the central role the HUD plays in helping the players determine their progress in the challenge.

The second problem with the HUD related to the specific representations chosen to portray the cell/tissue levels, and was more severe for the “gates” challenges than for the “size” challenges. In part, this relates to the complexity of the phenomenon in each challenge. The gates challenges entail understanding that melanosomes move (or are blocked from moving) from one layer of cells in the skin into another, more surface layer. The ideas that skin is multilayered and that organelles like melanosomes can travel between cells were likely new to students, and while they could successfully play the challenge it was clear that many did not understand the underlying biological phenomenon. The HUD in this case was particularly confusing because it provided a cross section of the two layers of cells. Students were able, for the most part, to interpret their state in the game using the HUD but could not explain what the representations in the HUD were actually showing (i.e. where is this in the drake?). We believe that students may be unfamiliar with cross sectional images and how to relate them to a “view from above” perspective. While they could pattern-match entities in the field of play to their smaller representations in the HUD, they did not understand how the zoomed-out view of the HUD is spatially oriented in relation to the representations on the field of play. Stated in general terms this is a problem of both misinterpreting the linked representations and being unfamiliar with particular disciplinary inscriptions. A solution to this problem could be to change the representation used in the HUD from a cross section view to some other more familiar view. However, the cross-sectional view is a fairly prevalent disciplinary way of showing locations of cells within a larger organismal context and there are reasons to keep it given the work it can do (when someone understands it). If the representation remains, the solution to this problem will likely entail providing more cues about the relationships between the spatial orientation of the HUD and the field of play. Perhaps adding a short video that shows how you get from one perspective to another (at the start of the gate challenges) will allow students to “see” the connections between these linked representations.

## Theme V: Confusing game progress and disciplinary process indicators

In the gates challenges, students try to change the sheen of the scales of the dragon from matte to shiny or vice versa. As noted earlier figuring out how to represent an increase or decrease in scale sheen in the game was a design challenge that we opted to solve by incorporating an indicator bar for sheen that shows how shiny the outer cell layer is at any point in time (an empty bar means the scales are shiny and a full bar means the scales are matte). The sheen indicator bar sits within the field of play (see Figure 2) and includes a small indicator arrow that shows the target state (empty or full bar). The location of the sheen indicator bar compounded with the complexity of the mechanism of the phenomena (students struggled to understand why it is that having no melanosomes in the outer cell layer makes a dragon shiny) resulted in some confusion about what that bar was actually representing. Some students understood it as showing sheen (as intended), while others thought it showed progress in the challenge (like the HUD). In the latter case, this was a problem when the target dragon was shiny and the goal was to have an empty bar. The discrepancy between the bar being empty and the challenge being successfully completed was confusing to some students who expected a full bar to indicate the challenge was complete. In general terms, the problem here is one of inconsistent use of general game navigation cues (like a progress bar) to represent mechanistic processes of a disciplinary phenomenon. Overall
using a progress-bar type representation to show changes in a mechanism is not an issue in and of itself; in fact, the HUD provides such support in the protein challenges. The case here is that this was a new navigational/indicator feature that was used sporadically in *only* one set of challenges- the protein gates challenges. One possible solution, therefore, is to either include a similar indicator bar in the other challenges or to try again to figure out how to represent sheen within the phenomenon itself (in the outer layer of cells).

# Discussion

The focus group interviews, while limited in scope, nonetheless provided us with valuable information about the ways in which students attended to, interpreted, and acted on the representations of phenomena in the game. The problems we identified reflected design and usability challenges associated with representing entities and mechanisms that are inherent and endemic to the discipline of genetics. In this section, we wish to 'pop up a level' and discuss some of the implications from this work more generally and provide a set of heuristics that can guide the design of representations in deeply disciplinary games.

The first heuristic, *balancing aesthetics, disciplinary fidelity, and phenomenological congruence*, stems from themes I and II. Deeply disciplinary games need to maintain high fidelity to the discipline in terms of how entities, activities, processes, and so on are represented. They also need to engage and attract students, and whenever possible they should be consistent with phenomena and behaviors that students may be familiar with (e.g. correspond p-prims that students likely hold). Balancing these three demands is not trivial and decisions that privilege aesthetics over fidelity or congruence (intentionally or not) could be costly in that it may take students longer to figure out the game dynamics. In addition, while disciplinary fidelity may seem like the most important consideration, adhering to it may have inadvertent consequences if it leads to meaningless aspects of a mechanism or entity being overly salient (e.g. clustering of triangles around the melanosome, Figure 1). It may be prudent at times to forgo some of the fidelity in favor of avoiding such situations.

The second heuristic, *pointing out core disciplinary distinctions*, stems from theme III. We have found that students were able to notice differences between ontologically distinct entities, but they did not recognize the biological significance of these differences. Altering the representations to make them even more different from each other, and different in ways that better reflect their ontological origins, may minimize the problem to some extent. Yet such changes are unlikely to be enough and we believe that additional scaffolds are needed to help students grasp the disciplinary nature and significance of the differences. The scaffolding design framework developed by Quintana et al., (2004) included a guideline about *making disciplinary strategies explicit in learners' interactions with the tool* (p. 345), which addresses the similar problem of students not having disciplinary knowledge to guide them in reasoning about problems in the discipline. The heuristic we provide builds on and extends this guideline by suggesting that one also needs to make explicit *ontological distinctions and features* of core entities and mechanisms. For example, in our game, it will be beneficial to explicitly point out that proteins resemble little machines because they act as such in the cell, whereas the melanin stars are inactive molecules comprised of smaller building blocks. Such "pointing out" needs to be done in the context of the game in ways that do not interrupt the game play. In a sense this conceptually important ontological distinction was not integrated well enough in the game. This presents a true design challenge in terms of making these key distinctions better conceptually integrated into the game. Another potential solution may be to draw on the Constructed Authentic Representations (CAR) principle (Holbert & Wilensky, 2014) and to allow students to build (through manipulation of DNA sequences- the instructions) proteins with particular functional domains. This may make more salient the distinction between what can be constructed (protein machines) by changing the genetic instructions and what is synthesized by using the protein machines.

The third, and final, heuristic, *supporting the linking of phenomena states to progress indicators*, stems from themes IV and V. We found that students had difficulties in relating both the HUD and the sheen indicator bar to the state of affairs in the field of play. There are actually multiple issues that contributed to this problem including lack of disciplinary knowledge of common representations (cross sectional views of tissues), the difficulty in representing sheen, and misinterpreting the indicator bar to be a general progress indicator. Therefore, in designing HUDs one needs to consider what types of links to the phenomenon are represented. Links could be temporal, showing ongoing progress of a mechanism as in the start-current-target states in the HUD; and/or the link can be spatial, showing where the mechanism is occurring in relation to the rest of the phenomenon as in the cross-sectional images of the cell and the images of the drakes in the HUD. In both cases students need support in making the connections and understanding what the HUD is showing. It is likely that the HUD in the protein challenges by itself is insufficient to support this linking and that additional scaffolds may be needed to help orient students to relevant relationships (such as an animation that connects the HUD and the phenomenon).
In designing *Geniventure* to accurately reflect complex genetic phenomena at the molecular level, we made many design decisions regarding how to represent key disciplinary entities and processes in the game in ways that can convey their biological characteristics. In many cases, the decisions proved effective and students were able to comprehend what they are doing in the context of the game and make progress across challenges. In other cases, the decisions hindered student progress. Moreover, even students who were able to progress substantially in the game did not always understand some of the underlying biology, such as the differences between proteins and other molecules in the cell (e.g. melanin). This underscores the importance of providing supporting curriculum and instruction to help students reflect on and generalize the mechanisms they are seeing in the game and their biological significance. A key question that arises is how can we decide on the division of scaffolding labor between the game, curriculum, and instruction (teacher and peers). This issue is even thornier in educational game contexts where there is a risk of making the game feel more school-like in ways that disengage students.

## References

Barnea, N., and Dori, Y. J. (1996). Computerized molecular modeling as a tool to improve chemistry teaching. *Journal of Chemical Information and Computer Sciences, 36* (3), 629–636.

Clark, D. B., Tanner-Smith, E., & Killingsworth, S. (2016). Digital games, design, and learning: A systematic review and meta-analysis. *Review of Educational Research, 86*(1), 79–122.

Cook, M., Wiebe, E. N., & Carter, G. (2007). The influence of prior knowledge on viewing and interpreting graphics with macroscopic and molecular representations. *Science Education, 92*, 848–867.

diSessa, A. (1988). Knowledge in pieces. In G. Forman & P. Putall (Eds.), *Constructivism in the computer age* (pp. 49-70). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Gee, J. P. (2005). Learning by Design: Good video games as learning machines. *E-Learning, 2*(1), 5–16.

Goodsell, D. S. (2009). The Machinery of Life, 2nd ed., Springer, New York.

Habgood, M. P. J., & Ainsworth, S. E. (2011). Motivating children to learn effectively: Exploring the value of intrinsic integration in educational games. *Journal of the Learning Sciences, 20*(2), 169–206.

Holbert, N., & Wilensky, U. (2014). Constructible authentic representations: Designing video games that enable players to utilize knowledge developed in-game to reason about science. *Technology, Knowledge and Learning, 19*(1–2), 53–79.

Kafai, Y. B. (2006). Playing and making games for learning: Instructionist and constructionist perspectives for game studies. *Games and Culture, 1*(1), 36–40.

Kindfield, A. C. (1994). Biology Diagrams: Tools to Think With. *Journal of the Learning Sciences, 3*(1), 1–36.

Kozma, R. (2003). The material features of multiple representations and their cognitive and social affordances for science understanding. *Learning and Instruction, 13*(2), 205–226.

Pallant, A., and Tinker, R. (2004). Reasoning with atomic-scale molecular dynamic models. *Journal of Science Education and Technology, 13*(1), 51–66.

Quintana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G., et al. (2004). A scaffolding design framework for software to support science inquiry. *Journal of the Learning Sciences, 13*(3), 337-386.

Ravyse, W. S., Blignaut, A. S., Leendertz, V., & Woolner, A. (2017). Success factors for serious games to enhance learning: A systematic review. *Virtual Reality, 21*, 31-58.

Roth, W., & Bowen, G. (1999). Complexities of graphical representations during ecology lectures: an analysis rooted in semiotics and hermeneutic phenomenology. *Learning and Instruction, 9*(3), 235-255.

Seufert, T. (2003). Supporting coherence formation in learning from multiple representations. *Learning and Instruction, 13*(2), 227–237.

Tsui, C.Y., & Treagust, D. (2004). Motivational aspects of learning genetics with interactive multimedia. *The American Biology Teacher, 66*, 277–285.

Tversky, B., Bauer Morrisony, J., & Betrancourt, M. (2002). Animation: Can it facilitate? *International Journal Human-Computer Studies, 57*, 247–262.

Williamson, V.M., & Abraham, M.R. (1995). The effects of computer animation on the particulate mental models of college chemistry students. *Journal of Research in Science Teaching, 32*, 522–534.

Winn, B. M. (2009). The Design, Play, and Experience Framework. In *Handbook of Research on Effective Electronic Gaming in Education* (Vol. 3, pp. 1010-1024). Hershey, PA: IGI Global.

Wu, H.K., Krajcik, J.S., & Soloway, E. (2000). Promoting understanding of chemical representations: Students' use of a visualization tool in the classroom. *Journal of Research in Science Teaching, 38*, 821–842.

## Acknowledgments

The research reported was funded by the National Science Foundation under Grant No. DRL-1513086.