## References

Cazden, C. (2001). *Classroom discourse: The language of teaching and learning*. Second Edition. Portsmouth, NH: Heinemann.

Cohen, E. G., & Lotan, R. A. (2014). *Designing Groupwork: Strategies for the Heterogeneous Classroom*. Third Edition. Teachers College Press.

Feldman, M. S., & Pentland, B. T. (2003). Reconceptualizing organizational routines as a source of flexibility and change. *Administrative Science Quarterly*, 48, 94–118.

Greeno, J. G. (1998). The situativity of knowing, learning, and research. *American psychologist*, 53(1), 5.

Horn, I. S. (2012). *Strength in Numbers: Collaborative Learning in Secondary Mathematics*. National Council of Teachers of Mathematics.

Horn, I. S. (in press). Supporting the Development of Pedagogical Judgment: Connecting Instruction to Context. *International Handbook of Mathematics Teacher Education*, 2nd edition, Volume 3. Sense Publishers.

Horn, I. S., & Kane, B. D. (2015). Opportunities for professional learning in mathematics teacher workgroup conversations: Relationships to instructional expertise. *Journal of the Learning Sciences*, 24(3), 373-418.

Horn, I. S., & Little, J. W. (2010). Attending to problems of practice: Routines and resources for professional learning in teachers' workplace interactions. *American educational research journal*, 47(1), 181-217.

Jordan, B., & Henderson, A. (1995). Interaction analysis: Foundations and practice. *The journal of the learning sciences*, 4(1), 39-103.

Lampert, M., Boerst, T. A., & Graziani, F. (2011). Organizational Resources in the Service of School-Wide Ambitious Teaching Practice. *Teachers College Record*, 113(7), 1361-1400.

Lave, J., Wenger, E., & Wenger, E. (1991). *Situated learning: Legitimate peripheral participation*. Cambridge: Cambridge university press.

Lavie, I., Steiner, A., & Sfard, A. (2018). Routines we live by: from ritual to exploration. *Educational Studies in Mathematics*, 1-24. https://doi.org/10.1007/s10649-018-9817-4

Lincoln, Y. S., & Guba, E. G. (1985). Establishing trustworthiness. *Naturalistic inquiry*, 289-331.

Lotan, R. A. (2003). Group-worthy tasks. *Educational Leadership*, 60(6), 72-75.

Schoenfeld, A. H. (2014). What makes for powerful classrooms, and how can we support teachers in creating them? A story of research and practice, productively intertwined. *Educational researcher*, 43(8), 404-412.

Smith, M. S., & Stein, M. K. (2011). *5 practices for orchestrating productive mathematics discussions*. Reston, National Council of Teachers of Mathematics.

Stein, M. K., Engle, R. A., Smith, M. S., & Hughes, E. K. (2008). Orchestrating productive mathematical discussions: Five practices for helping teachers move beyond show and tell. *Mathematical thinking and learning*, 10(4), 313-340.

Strauss, A., & Corbin, J. (1998). *Basics of qualitative research: Procedures and techniques for developing grounded theory*. Sage.

Webb, N. M., Franke, M. L., De, T., Chan, A. G., Freund, D., Shein, P., & Melkonian, D. K. (2009). ‘Explain to your partner’: teachers’ instructional practices and students’ dialogue in small groups. *Cambridge Journal of Education*, 39(1), 49-70.

## Acknowledgments

We are thankful to the participating teachers, to Nicole Louie and Anna Sfard who gave helpful feedback on earlier stages of work, to Avital for her help with the graphic representations, and to the research team: Patricia Buenrostro, Grace Chen, Brette Garner, Mariah Harmon, Lara Jasien, Samantha Marshall, Elizabeth Metts, and Katherine Schneeberger McGugan. This work has been supported by a grant from the National Science Foundation.
# Impacts of Augmented Reality on Collaborative Physics Learning, Leadership, and Knowledge Imbalance

Iulian Radu, Harvard University Graduate School of Education, iulian_radu@gse.harvard.edu
Bertrand Schneider, Harvard University Graduate School of Education, bertrand_schneider@g.harvard.edu

**Abstract:** Emerging technologies such as Augmented Reality (AR), have the potential to radically transform science education by making challenging concepts visible and accessible to novices. In this project, we designed a Hololens-based system in which participants learned about the physics involved in audio speakers (Radu & Schneider, 2019). We analyzed participant dyad where educational AR representations were present or not, focusing on the relationships between collaboration, learning, leadership, and knowledge imbalance. We find that, overall, AR representations improved time management, learning of structural concepts but reduced learning of physical concepts. The effects of leadership were mediated by the presence of AR: the presence of leaders in AR was linked to higher learning gains and better collaboration, whereas these effects were not present without AR. Finally, for groups with imbalanced knowledge, AR seemed to benefit participants in configurations where less-knowledgeable participants took the lead in discussions. These results indicate that AR can be beneficial for equalizing the effects of imbalanced collaboration. We discuss the implications of those results for the design of CSCL learning activities using augmented reality.

## Introduction

In this research we investigate the benefits and drawbacks of augmented reality for inquiry-based learning, specifically in relation to participant agency and leadership. We focus on a collaborative activity that allows pairs of students to explore concepts in electromagnetism. Electromagnetism is a topic that is often encountered in both maker spaces and traditional physics classrooms, and it is one of the most difficult topics to master because it combines understanding of physical objects (ex: magnets, wires) and abstract concepts (ex: magnetic field shapes, voltage, electricity) (Belcher and Bessette 2001; Ibáñez et al. 2014; Maloney et al. 2001).

We focus on how AR technology in this educational context relates to leadership and individual agency. A prevailing issue in classroom group collaborations is the effect of unequal participation and unequal knowledge among group members. Some people are naturally more dominant while others more passive, and this may be enhanced by the amount of each person's domain expertise (Salomon & Globerson, 1989). This imbalance can create negative effects whereby the more passive or less-knowledgeable students do not contribute as much as their peers, potentially leading to less effective problem solving, collaboration breakdowns, and "free rider" effects. Augmented reality environments have the potential to mediate the effects of participant leadership and knowledge imbalances. On one hand, AR environments can provide a plethora of holographic visualizations, which can be beneficial for participants by allowing passive or lower-knowledge participants to easily follow the communications of their peers; they allow participants to more easily ask for clarification or interrupt by pointing at representations; they allow more knowledgeable participants to explain by use of the available representations; and they can allow more equitable interaction with the experience, thus providing agency to participants who may naturally be more passive. On the other hand, AR environments require the use of specialized hardware, including the use of head-mounted devices which may cover the participants' eyes, thus making communication difficult by masking nonverbal cues such as facial expressions.

In this study we investigate these topics by studying dyads using head-mounted Microsoft Hololens AR devices, in conditions where educational AR representations are either present or not. We predict that the presence of AR information will balance collaboration because participants have shared access to information, thus one person will be less likely to dominate the experience; additionally, we expect the availability of shared representations to potentially increase the participation of more passive group members. For this reason, we predict that educational AR visualizations will have positive effects on collaborative processes. Finally, we observe how groups behave when more active participants (drivers) have high or low knowledge, and expect that groups with more knowledge imbalance will be negatively impacted in the condition where groups do not have access to educational AR information because a "free rider" effect (Salomon & Globerson, 1989).

## Related work

Augmented reality systems have been developed for a wide range of educational uses, such as for learning geometry (Radu et al. 2015), chemistry (Yu-Chien 2006), and history (Chang et al. 2015). Specifically for physics
education, systems have been built for visualizing electricity (Belcher and Bessette 2001) and magnetism (Maloney et al. 2001). Augmented reality systems can support student learning of spatial structures, improve performance on physical tasks, and have lasting effects on participant memory (for a comparative media review see Radu, 2014). While the effects of AR experiences on learning have been studied, there is relatively less research focused on understanding the impact of AR on the dynamics of co-located collaboration. Previous research projects have focused on designing AR infrastructures that support social interactions (Billinghurst, Clark, and Lee 2015), for example by allowing a remote expert to inhabit a physical space, allowing multiple users to annotate physical environments, and allowing users to have different views of the same environment. Research has also found that student groups using augmented reality can benefit from increased collaboration because they are more motivated to engage with the experience (Phon, Ali, & Halim, 2014) and because the experiences simulate real-world professional collaborations (Dunleavy & Dede, 2014). Furthermore, it has been argued that AR experiences can balance leadership behaviors of participants, since group members can have access to shared visualizations and one person is less likely to control the group resources (Morrison et al., 2009).

We contribute to this research agenda by specifically studying how collaboration aspects of leadership and knowledge imbalance are impacted by the presence of educational AR, presented as holograms on physical artifacts. A prevailing issue in group collaborations is the effect of unequal participation among group members. Unequal participation is caused by multiple factors including unequal knowledge relevant to the activity, unequal ability to control the activity, or unequal personal interest and initiative. In situations where team members do not (or cannot) contribute equally, this typically results in lower learning gains (Chen 2006). Previous research indicates that when a resource is limited among team members, this encourages one person to dominate the interaction and creates imbalanced participation (Church, Hazlewood & Rogers 2006). In such contexts, participants may simply follow along, leading to decreased learning gains and poor collaboration (Salomon and Globerson 1989). Shared interfaces such as tangible objects have the potential to balance participation as each user has access to the learning content (Church, Hazlewood & Rogers 2006), especially when such interfaces allow participants to have shared control and awareness of information available to the group (Yuill and Rogers 2012).

## Augmented reality system and research questions

In this section, we describe how prior work informed the design of our system, the study we conducted, our research questions and the methodology we used to answer them.

We have designed a tangible model of a sound-producing speaker, augmented with interactive educational holograms visible to two users through the Microsoft Hololens augmented reality headset. This educational system mixes physical content (physical wires, magnets, sounds, magnetic forces, movement) and virtual content (visualizations of electricity, magnetic fields, force directions). The participants can interact with the system by playing music or constant electric current in different directions; modify amplification; switch between different electromagnet coils; and slide the vibrating membrane. They also can use a compass to measure magnetic fields manually. The system is shown in Figure 1.

<figure><img src="image_74.png" /><figcaption>**Figure 1.** View of the system without educational AR representations (left) with components labeled; view of the system with educational AR showing magnetic fields, electricity and related representations (right – all holograms and labels are shown as visible in AR).</figcaption></figure>

In (Radu & Schneider, 2019), we have investigated how participants’ learning is influenced by augmented reality representations at the individual level (results are summarized in Fig. 3). Participants were randomly assigned to four experimental conditions which comprised a nested design with two factors (primary factor: presence of AR educational representations, sub-factor: type of technology features). All conditions had access to the physical system and were able interact by changing electricity, generate sound, move the speaker membrane, change the speaker coils, and change amplification; furthermore, all conditions had access to a
physical poster that explained electromagnetism concepts, had labels showing the function of pieces of the physical system. The groups which did not have educational AR representations (noEdAR) were split into 2 subgroups – dyads which did not wear the Hololens AR devices, and dyads who did but only saw minimal AR representations composed of holographic labels of major system components, and holograms of sound waves which represented the sound when the system played music. The participants who saw educational AR representations (EdAR) had access to the same information, but could also see interactive visualizations of magnetic fields, electron flows, and electricity graphs. These EdAR groups were split into 2 subgroups in which the presentation of AR visualizations was either presented all at once, or sequenced by a timer. All conditions performed the same study activities.

Our analysis of individual learning gains indicates that participants who used AR technology had significantly higher learning gains on specific concepts such as understanding spatial structures (ex: shapes of magnetic fields through questions, i.e. "Draw the magnetic field around a single wire"), higher ability to transfer knowledge to different situations (ex: transfer questions, i.e. "Is it possible to build a motor that is moved through electric signals? If yes, explain how."), but were significantly worse at understanding physical effects (ex: questions about relationship between physical movement and magnetic fields, i.e. "If the magnetic field is suddenly inverted, the speaker membrane: a) is pulled closer; b) is pushed away; c) does not move").

<figure><img src="image_75.png" /><figcaption>**Figure 2.** A dyad pair interacting with the system while seeing AR through Microsoft Hololens headsets.</figcaption></figure>

<figure><img src="image_76.png" /><figcaption>**Figure 3.** Individual differences in relative learning gains (Green: without educational AR; Red: with educational AR; whiskers: standard error).</figcaption></figure>

In the current research we focus on the effects of AR on collaboration and participant leadership and agency, and analyze the data aggregated at the group level. We, compare learning and collaborative behaviors between two conditions: participants who used the system and saw educational AR visualizations; and participants who used the system but did not see educational visualizations. We predict that, due to the increased availability of educational visualization, participants who see educational AR will exhibit less affected by imbalanced leadership and imbalanced knowledge in the group. Our research questions are as follows:

**RQ1:** Is overall collaboration and learning impacted by the presence of educational AR?

**RQ2a:** How does participant leadership imbalance impact learning, collaboration, and interaction?

**RQ2b:** How does leadership differ with the presence of AR?

**RQ3a:** How does leadership imbalance impact learning, collaboration, and interaction?

**RQ3b:** How does the effect of driver-follower imbalance differ with the presence of AR?

## Method

Participants were recruited from the study pool of a laboratory at a university in the northeastern United States. Participation required subjects to not know each other, have no significant prior physics knowledge, be born on/after 1976, speak English fluently, have at least a bachelor's degree, and wear no bifocal glasses. All participants first completed a pre-test, followed by a short written introduction to relevant physics concepts. Participants then worked on the speaker activity for 30 minutes under different experimental conditions (see Figure 3). During this period, all participants worked on a worksheet and saw a poster of printed physics knowledge on the wall. The study ended with a post-test and debriefing. The variables of interest are as follows:

## Independent variables

We looked at three different independent variables in our analyses: presence / absence of AR, leadership and knowledge imbalance. We provide our operationalization of those constructs below:

**Presence of Educational AR:** Groups were randomly assigned to conditions in which educational AR
was present (EdAR) or not present (NoEdAR), as described above.

**Leadership:** Groups were categorized under two conditions depending on participant leadership: dyads where a leader was present, and dyads where no leader was present. Leadership was considered to be present when both partners in a group do not initiate actions equally (i.e. when the maximum score was not recorded for the qualitatively-observed "reciprocal interaction" dimension, defined in Table 2 below).

**Driver Follower Knowledge:** We rated the participation of each person in a group as either being a "driver" (the participant who typically initiated actions overall) or "follower" (the participant who was more passive overall). We accounted for differences pretest knowledge, resulting in four group configurations: LL, LH, HL, HH where the first letter refers to the driver and the second letter refers to the passenger. For example, HL indicates a high-knowledge driver, and low-knowledge passenger; LH indicates low-knowledge driver and high-knowledge follower.

## Dependent variables

**Learning Metrics:** We measured participant learning through pre- and post-tests. Participants' learning was compared using relative learning gains, a measure of the relative improvement that occurred between pre-post test scores (Cuendet et. al. 2012). The learning test contained multiple-choice and open-ended questions measuring several aspects of conceptual knowledge. All learning metrics are listed in Figure 3 and described in detail in (Radu & Schneider, 2019).

**Collaboration Metrics:** Collaboration metrics were qualitatively coded for each pair of participants across several dimensions using a validated rating scheme described in Meier, Spada & Rummel (2007). The scale evaluates collaboratives processes through a 5-point scale on the following dimensions: sustained mutual understanding, dialogue management, information pooling, reaching consensus, task division, technical coordination, and reciprocal interaction. Examples are provided in Table 1.

**Interaction Metrics:** The backend of the tangible interface system recorded how much users interacted with different system components, such as changes in electricity direction, movement of speaker membrane, changes of speaker coils, and changes in amplification.

<table><caption>Table 1: The measured dimensions of collaboration, based on Meier, Spada & Rummel (2007)</caption><thead><tr><td>Collaboration Metrics</td><td>Example</td></tr></thead><tbody><tr><td>Sustained mutual understanding</td><td>Ensure partners are on same page. Speakers make their contributions understandable for their collaboration partner rather than ignoring each other's insight.</td></tr><tr><td>Dialogue management</td><td>Smooth flow of communication with little overlaps. Smooth volley of conversation with little interruptions; partners make sure to have each other's attention before transitioning to other topics</td></tr><tr><td>Information pooling</td><td>Ask questions to seek each other's perspective. Partners contribute their insight effectively or ask useful questions to seek opinions.</td></tr><tr><td>Reaching consensus</td><td>Coming to shared understanding / agreement. Both partners come to a shared conclusion; if there is disagreement, they resolve it through critical consideration</td></tr><tr><td>Task division</td><td>Task division is balanced, and tasks are explicitly distributed between partners through discussion.</td></tr><tr><td>Time management</td><td>Deadline met, detailed time planning. Partners monitor the time throughout their cooperation and make sure to finish the current subtask or topic with enough time to complete the remaining subtasks.</td></tr><tr><td>Technical coordination</td><td>All tools used, including physical compass. Partners use all technology at their disposal, including features such as physical magnetic compass, using different coils, referring to the physical poster, etc.</td></tr><tr><td>Reciprocal interaction</td><td>Partners hold equal status in working relationship; both take agency in leading the discussion instead of one partner dominating the working relationship.</td></tr><tr><td>Overall Scores</td><td>Overall Communication (avg. dimensions 1,2), Overall Joint Information Processing (avg. dimensions 3,4), Overall Coordination (avg. dimensions 5,6,7), Overall Collaboration (avg. dimensions 1-8)</td></tr></tbody></table>

## Results

In the original study, we recruited 30 participant pairs in the two primary conditions (N=60) and removed sessions where technical issues were encountered or sessions that contained outliers with exemplary pretest knowledge (2.5 standard deviations beyond the pretest mean score), resulting in N=56 sessions (N=28 in EdAR, N=28 in noEdAR).

For qualitative measurement of the degrees of collaboration and leadership, two researchers double-coded 20% of the video recordings and achieved interrater reliability of Cohen Kappa 0.67, indicating "good"
agreement. For statistical tests, normality assumptions were not met for continuous dependent variables, thus we
performed Wilcoxon Rank-Sum nonparametric tests for differences between conditions of Presence of
Educational AR, and for conditions of Leadership; and we performed Kruskal-Wallis H nonparametric tests
followed by post-hoc tests for differences between conditions of Driver-Follower Knowledge.

RQ1: Are overall collaboration and learning impacted by the presence of educational AR?

We tested for collaboration and learning differences between EdAR and noEdAR conditions. The EdAR groups were significantly better at **time management** (W=446, p=0.050), and significantly better at answering **near transfer questions** (W=568, p=0.003), but significantly worse at understanding relationship between **magnetic fields and movement** (W=248.5, p=0.019). These learning effects mirror our results of analysis at the individual level presented in (Radu & Schneider, 2019), where we found these and additional effects detectable in the larger sample when participants are considered individually (Figure 3). We found no other statistically significant effects when analyzing differences at the dyad level.

RQ2a: How does participant leadership imbalance impact learning, collaboration, and interaction?

Dyads were categorized into two groups: dyads where a leader was present (N=22 overall; N=11 in EdAR, N=11 in noEdAR), and where no leader was present (N=34 overall; N=17 in EdAR, N=17 in noEdAR).

We analyzed whether there is a significant effect of leadership on dependent variables across all groups.
We find that dyads with leadership generally had higher relative learning gains on **all transfer questions** (W=253.5, p=0.044); but had weaker collaboration in **sustained mutual understanding** (W=511, p=0.010), **reaching consensus** (W=584, p<0.001), **task division** (W=511, p=0.004), **overall joint information processing** (W=562, p=0.001), **overall communication** (W=492, p=0.033), and **overall collaboration** (W=584.5, p<0.001).
This indicates that, although leadership causes some learning gains, it also produces visible negative effects on collaboration.

<figure><img src="image_77.png" /><figcaption>Figure 4. Significant effects of driver-follower configurations in EdAR (left) and noEdAR (right). \*=sig. difference between leadership groups. Whiskers=standard error.</figcaption></figure>

RQ2b: How does leadership differ with the presence of AR?

Since AR seems to have an influence on group leadership, we investigated the effects of leadership in the presence and absence of educational AR. Within EdAR groups, we found that groups with leadership had higher gains on **transfer questions** (W=49.5, p=0.040) and understanding the relationship between **electricity vs. movement** (W=47.5, p=0.031); also, groups with leadership had lower scores on **reaching consensus** (W=147.5, p=0.001), **task division** (W=126, p=0.047), and **overall joint information processing** (W=137, p=0.027), compared to groups without leadership. In contrast, in the noEdAR condition, the presence of leadership had no statistical impact on learning gains, and had worse impact on collaboration and system interaction. For groups in the noEdAR condition, similar to EdAR condition, groups with leadership were worse on **reaching consensus** (W=144.5, p=0.001), **task division** (W=130, p=0.040), **joint information processing** (W=144, p=0.009), but had additional negative effects through lower scores on **interaction** with speaker membrane (W=82, p=0.039),
**sustained mutual understanding** (W=140.5, p=0.016), **overall communication** (W=138, p=0.029), and **overall collaboration** (W=162, p=0.001). These results indicate that within the EdAR experience, leadership imbalance has a stronger effect of emphasizing learning gains, and has less negative effects on collaboration and system interaction.

To understand the differences between these conditions in relation to leadership effects, we sampled video recordings from EdAR and noEdAR groups. One theme observed is that in situations where EdAR was present, the visual representations helped the more passive participant follow the more dominant participant's explanations: it allowed the passive participants to interrupt and ask for clarification by referencing to the AR representation. Table 2 illustrates an example in EdAR (left) where one participant is teaching the other by using magnetic field polarity representations. In contrast, in the Non-AR example (right) one participant is trying to teach but ends up dominating the discussion while the other participant is unable to keep up with the explanations due to the lack of shared representations.

<table><caption>Table 2: Quotes from participants in moments of teaching</caption><thead><tr><td>EdAR Group</td><td>Non EdAR Group</td></tr></thead><tbody><tr><td>P1: "The direction is, like, the south pole, like N S N S is the backward one."</td><td>P2: "So alright here's an interesting thing, when you play music, that's making little currents going forward and backwards, but if no forward current is allowed, it does this &lt;pushes button&gt; this just overpowers everything, and only gonna pull, and when you switch it to forward, only pushes, and when you switch it to music it does little tiny currents like really fast, and then that's making it vibrate, and as it vibrates it's shaking this cup, and as that vibrates it's shaking the air, and that air is shaking our ears, and shaking our brains."</td></tr><tr><td>P2: "Uh huh."</td><td>P1: "Really cool"</td></tr><tr><td>P1: "Can you see that?" &lt;pointing at AR&gt;</td><td>P2: "What do you think?"</td></tr><tr><td>P2: "Yeah."</td><td>P1: "Good story" [nervous laughter]</td></tr><tr><td>P1: "Starting from the left, N S -"</td><td>P2: "You like my story? Ok, so how does that help us get to the question"</td></tr><tr><td>P2: "Yeah."</td><td></td></tr><tr><td>P1: "-N S. And, for forward current, it's -"</td><td></td></tr><tr><td>P2: "Oh."</td><td></td></tr><tr><td>P1: "-S N N S."</td><td></td></tr><tr><td>P2: "S N. Ok. So, this is...so backward was [picks up pen]"</td><td></td></tr><tr><td>P1 [confirms, as P2 draws]: "Backward, N S !"</td><td></td></tr></tbody></table>

### RQ3a: How does group leadership imbalance impact learning, collaboration, and interaction?

Each group was categorized into one of the four driver-follower conditions. LL (EdAR N=9, noEdAR N=9), LH (EdAR N=6, noEdAR N=6), HL (EdAR N=9, noEdAR N=10), HH (EdAR N=4, noEdAR N=3). We analyzed the effects of different types of driver-follower configurations across all groups. No statistical differences were found for relative learning gains and interactions. There was a significant effect of driver-follower knowledge on the dimension of **overall coordination** (X²=8.9, p=0.031), and **overall collaboration** (X²=11.9, p=0.008); in all cases, descriptive LH groups scores higher than other groups. This indicates that overall, groups had better collaboration when novices were guiding the interaction.

### RQ3b: How does the effect of driver-following knowledge differ with the presence of AR?

Since the AR medium may influence how driver-follower configurations behave, we investigated the effects of this variable when educational AR was present or absent. Within the EdAR groups, there were significant effects of driver-follower configurations on the dimensions of **dialogue management** (X²=12.4, p=0.006), and **overall collaboration** (X²=10.3, p=0.016); in all cases, descriptive statistics show that the LH groups scored higher than other groups. Within the No-EdAR groups, there were no significant effect of driver-follower knowledge distribution. These results suggest that AR educational representations are beneficial to participants in the LH configurations (Fig. 5):

<figure><img src="image_78.png" /></figure>
**Figure 5.** Significant effects of driver-follower configurations in EdAR (left) and noEdAR (right). \*=sig. difference between driver-follower configuration conditions. Whiskers=standard error.

By qualitatively analyzing EdAR and noEdAR groups, we saw examples of participants with less knowledge taking initiative in guiding group discussion (Table 3). In groups where AR representations were present, participants with less knowledge could easily ask questions by pointing at hologram representations of the system, and participants with more knowledge had an easier time communicating their knowledge by referring to the visual representations. As an example, Table 3 illustrates how AR representations were useful for guiding participant discussion (left) and lack of such representations in Non-AR groups contributed to difficulties in understanding. In both examples, the participant with less knowledge is asking for clarification from the other participant; in the AR scenario the knowledge is provided by using the AR representations; in the Non-AR scenario, the person with more knowledge is unable to provide a clear explanation. These behaviors appeared to encourage communication within AR groups and may be a reason for increased benefits to dyads in the LH driver-follower configuration.

<table><caption>Table 3. Quotes from participants in the AR and non-AR conditions to clarify ambiguous interactions.</caption><thead><tr><td>EdAR Group</td><td>Non EdAR Group</td></tr></thead><tbody><tr><td>P2: "Is it because it's - is it hitting it? Is the -"</td><td>P1: "So, when it, when it's a backward current [illustrating shape of magnetic field using hands], it looks like it's pulling this way"</td></tr><tr><td>P1: "I can see the vibration, or I can see how it's moving in this AR [moving hand side to side]"</td><td>P2 [pointing toward cup]: "On this, this thing, right?"</td></tr><tr><td>P2: "In the AR?"</td><td>P1: "Yeah, so I wonder - I wonder if it's almost like, when it's backward, it's like that [illustrating on paper with fingers]."</td></tr><tr><td>P1: "AR - the augmented reality part, like it's just - [gesturing around area surrounding cup and coiled wire with hand]"</td><td>P2 [confused, handing pen over]: "Are you good at drawing?"</td></tr><tr><td>P2: "Yeah"</td><td>P1 [starting to draw on worksheet]: "Well, I would, um..."</td></tr><tr><td>P1: "The angle of the base is like [pointing to cup]"</td><td></td></tr><tr><td>P2: "Oh! Oh! Oh! Yeah, yeah, yeah, yeah, yeah, yeah! I like the - this thing [moving finger back and forth in front of area between cup and coiled wire]. Okay, so the alternating currents is pulling the magnet closer or further - further"</td><td></td></tr><tr><td>P1: "Mm-hm"</td><td></td></tr><tr><td>P2: "Okay"</td><td></td></tr></tbody></table>

## Discussion

Our original hypotheses predicted that the presence of educational AR representations would have an effect on overall collaboration and learning. For collaboration, we found differences in time management, with lower values for groups which did not have educational AR representations and tended to run out of time. This indicates that the AR representations were useful for completing the task, and groups which lacked these had a harder time understanding the system and communicating since they had to generate their own representations (for example by drawing). For learning gains, analyses at dyad level did not show the same effect as the individual analysis presented in (Radu & Schneider, 2019). This is likely due to the effect of averaging the learning between a participant pair, which reduced the sample size and decreased our statistical power.

Augmented reality was also helpful for groups where participants had knowledge imbalance, specifically when the participants with high knowledge took a back seat and allowed the low-knowledge participants to drive the interaction. When this occurred, metrics of group collaboration was higher in AR groups. This indicates that AR can potentially improve collaboration in groups where participants have unequal knowledge. This might be due to the availability of information to both participants, which can equalize the effect of knowledge imbalance by allowing less-knowledgeable participants to more easily communicate points of confusion by referring to the existing AR representation. Referring to the present representations might also allow the more knowledgeable participants to teach his/her peer, which can help passive participants be more engaged in the collaborative activity.

These results point at potential future work investigating the detrimental effects of augmented reality on learning concepts focused on physicality (ex: relationship between movement and magnetic fields), that possibly may be caused by learners being hyper-focused by highly visual nature of the AR medium and paying less attention to physical sensations. On the other hand, these results indicate that AR can be beneficial for equalizing the effects of imbalanced collaboration, and future work could investigate how this can benefit situations where participants tend to be imbalanced in agency (ex: in student team projects where some participants are more high-achieving than others) or in knowledge (ex: in teams of varied expertise levels, where participants may need to take on the role of teachers).
## Conclusions

In this study we analyzed collaboration, learning and interactions of dyad pairs as they experienced an AR system for learning about electromagnetism. We found that, in this context, augmented reality was generally beneficial for both learning and collaboration. Overall, AR representations improved time management, learning of structural concepts but reduced learning of physical concepts. The effects of leadership were mediated by the presence of AR: the presence of leaders in AR was linked to higher learning gains and better collaboration, whereas these effects were not present without AR. Finally, for groups with imbalanced knowledge, AR seemed to benefit participants in configurations where less-knowledgeable participants took the lead in discussions.

## References

Belcher, J. W., & Bessette, R. M. (2001). MIT educators share success-Using 3D animation in teaching introductory electromagnetism. COMPUTER GRAPHICS-US, 35(1), 18–21.

Billinghurst, M., Clark, A., & Lee, G. (2015). A survey of augmented reality. Foundations and Trends® in Human-Computer Interaction, 8(2–3), 73–272.

Chang, Y.-L., Hou, H.-T., Pan, C.-Y., Sung, Y.-T., & Chang, K.-E. (2015). Apply an augmented reality in a mobile guidance to increase sense of place for heritage places. Journal of Educational Technology & Society, 18(2).

Chen, Y.-C. (2006). A study of comparing the use of augmented reality and physical models in chemistry education. In Proceedings of the 2006 ACM international conference on Virtual reality continuum and its applications (pp. 369–372). ACM.

Church, T., Hazlewood, W. R., & Rogers, Y. (2006). Around the table: Studies in co-located collaboration. Austrian Computer Society.

Cuendet, S., Bumbacher, E., & Dillenbourg, P. (2012). Tangible vs. virtual representations: when tangibles benefit the training of spatial skills. In Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design (pp. 99–108). ACM.

Dunleavy, M., & Dede, C. (2014). Augmented reality teaching and learning. In Handbook of research on educational communications and technology (pp. 735–745). Springer.

Ibáñez, M. B., Di Serio, Á., Villarán, D., & Kloos, C. D. (2014). Experimenting with electromagnetism using augmented reality: Impact on flow student experience and educational effectiveness. Computers & Education, 71, 1–13.

Maloney, D. P., O’Kuma, T. L., Hieggelke, C. J., & Van Heuvelen, A. (2001). Surveying students’ conceptual knowledge of electricity and magnetism. American Journal of Physics, 69(S1), S12–S23.

Meier, A., Spada, H., & Rummel, N. (2007). A rating scheme for assessing the quality of computer-supported collaboration processes. International Journal of Computer-Supported Collaborative Learning, 2, 63–86.

Morrison, A., Oulasvirta, A., Peltonen, P., Lemmela, S., Jacucci, G., Reitmayr, G., ... Juustila, A. (2009). Like bees around the hive: a comparative study of a mobile augmented reality map. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1889–1898). ACM.

Phon, D. N. E., Ali, M. B., & Halim, N. D. A. (2014). Collaborative augmented reality in education: A review. In Teaching and Learning in Computing and Engineering (LaTiCE), 2014 International Conference on (pp. 78–83). IEEE.

Radu, I., Doherty, E., DiQuollo, K., McCarthy, B., & Tiu, M. (2015). Cyberchase shape quest: pushing geometry education boundaries with augmented reality. In Proceedings of the 14th International Conference on Interaction Design and Children (pp. 430–433). ACM.

Radu, I. & Schneider, B. (2019). What Can We Learn from Augmented Reality (AR)? Benefits and Drawbacks of AR for Inquiry-based Learning of Physics. In 2019 CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019), May 4–9, 2019, Glagsow, Scotland, UK. ACM.

Radu, I. (2014). Augmented reality in education: a meta-review and cross-media analysis. Personal and Ubiquitous Computing, 18(6), 1533–1543.

Salomon, G., & Globerson, T. (1989). When teams do not function the way they ought to. International Journal of Educational Research, 13(1), 89–99.

Yuill, N., & Rogers, Y. (2012). Mechanisms for collaboration: A design and evaluation framework for multi-user interfaces. ACM Transactions on Computer-Human Interaction (TOCHI), 19(1), 1.

## Acknowledgements

This material is based on work supported by the National Science Foundation, Grant No. 1748093. We thank all our participants, reviewers, the Harvard Decision Science Lab, and assistants who helped with data collection.
# Finding Common Ground: A Method for Measuring Recent Temporal Context in Analyses of Complex, Collaborative Thinking

Andrew R. Ruis, University of Wisconsin–Madison, arruis@wisc.edu

Amanda L. Siebert-Evenstone, University of Wisconsin–Madison, alevenstone@wisc.edu

Rebecca Pozen, University of Wisconsin–Madison, rnpozen@gmail.com

Brendan R. Eagan, University of Wisconsin–Madison, beagan@wisc.edu

David Williamson Shaffer, University of Wisconsin–Madison & Aalborg University, david.shaffer@wisc.edu

**Abstract:** Complex, collaborative thinking is often conceptualized as a process of developing cognitive connections among the contributions of different participants. A central problem in modeling collaboration in this way is thus determining, for any contribution to a discussion, the appropriate context for modeling the connections being made—that is, for determining the appropriate recent temporal context. Recent temporal context is typically defined using a moving window of fixed length. However, that length is dependent on the setting, and there are no existing methods for reliably determining an appropriate window length. This paper presents an empirical method for measuring recent temporal context, and thus for defining an appropriate window length to be used in analyses of complex, collaborative thinking. Importantly, the method we describe minimizes the need for human annotation while providing both qualitative and quantitative warrants for choosing a particular window length.

## Introduction

In the learning sciences, complex thinking is often conceptualized as a process of developing cognitive connections among concepts (DiSessa, 1988; Linn, Eylon, & Davis, 2004; Shaffer, 2012). In computer-supported collaborative learning (CSCL) contexts, individuals make such connections not only within their own contributions but also to the contributions of their collaborators (Garrison, Anderson, & Archer, 2001; Shaffer, 2017). A central problem in modeling complex, collaborative thinking in terms of cognitive connections is thus determining, for any contribution to a discussion, the appropriate temporal context for modeling the connections being made. Prior work in the learning sciences has approached this problem using *moving windows* (Dyke, Rohit Kumar, Hua, & Rosé, 2012; Siebert-Evenstone et al., 2016), where each turn of talk is associated with some prior segment of the discussion that forms its *recent temporal context* (Suthers & Desiato, 2012).

In such models, analysis of a given turn of talk accounts for both its own content and the content of its associated window. Because there are many variables that may affect the extent of the recent temporal context, including domain, topical complexity, age of the participants, and communication medium, it is important to identify an appropriate window length for each setting. However, there are no reliable methods for measuring the extent of recent temporal context, and studies have not been conducted that show the effects of window length on the features or interpretation of window-based learning analytic models. In this study, we present an empirical method for measuring the extent of recent temporal context. We then evaluate the method by analyzing conceptual connectivity in the same dataset using different window lengths to explore the effects of window length on the resulting models. The results suggest that an appropriate moving window length can be empirically determined with minimal effort, and that while window length can significantly affect model features and interpretation, the empirical method we describe produces relatively robust models of complex thinking.

## Background

Collaborative learning takes place both in face-to-face interactions and remotely, mediated by various communications technologies. In addition, computer simulations, intelligent tutoring systems, educational games, and other CSCL learning technologies create settings in which students work with one another, with educators or mentors, and with pedagogical agents to frame, investigate, and solve complex problems. Such settings foster the development of *communities of inquiry* (Garrison & Akyol, 2013; Seixas, 1993): groups of participants who interact with one another in a problem-solving context to facilitate critical discourse and establish shared meaning and understanding. A key goal of such collaborative interactions is to help participants make explicit connections among the contributions of different team members, which enables communities of inquiry to organize knowledge, identify and address misconceptions, facilitate reflection, and ultimately construct mutual understanding (Garrison et al., 2001).
Modeling complex, collaborative thinking, then, requires assessing not only the connections that individuals make among their own contributions but also the connections they make to the contributions of the other participants. This in turn requires identifying the appropriate relational context for any given turn of talk in a discussion (Arvaja, Salovaara, Häkkinen, & Järvelä, 2007). Researchers often establish relational context by looking at each turn of talk and identifying its *referents*: the prior turn or turns of talk that provide the information needed to understand the meaning of a given utterance (Shaffer, 2017).

## Illustrating the problem

Consider the following excerpt, in which a student project team working on a biomedical engineering design project is discussing with a mentor (Maria) how they determined which design prototypes to test (emphasis added).

1. Maria What role did the requests of the **internal consultants** play in deciding which prototypes to test?

2. Jill that's it though. then we have to pick a final one and make a poster presentation

3. Jill **Internal consultants** made it clear **reactivity** is important, but **cost** triumphs

4. Lesenia the **requests** gave a focus on what **factors** to pay attention to

5. Maria How did you design your device to address **patient** needs?

6. William We tried to make them **cheap** and **reliable** so they could last a long time and save the **patients** money

7. Brad Since we are trying to **make a bunch of people happy**, we wanted a wide **variety of results** to see which one **makes the most people happy**.

Maria asks the students a question in Line 1, which Jill and Lesenia answer in Lines 3 and 4. Maria then asks a second question in Line 5, to which William responds in Line 6. In Line 7, Brad's response references "people" twice, and we can identify who they are only by looking to previous utterances—that is, by identifying the referent(s) for Line 7. Yet how far back do we need to look? If the context for his utterance consists only of Lines 5 and 6, we may determine that Brad is referring to patients, the users of the device the team is designing. However, if all the lines are included as context, it is clear that by "people" Brad means both the patients *and* the internal consultants. Interpretation of Brad's utterance—and by extension, interpretation of the connections that Brad is contributing to the team's mutual understanding—depends on how we define his contribution's recent temporal context. That is, there is some window (in this case, seven turns of talk) that contains all of the context (i.e., all of the referents) needed to understand a given contribution to a discussion.

Ideally, the size of a given window would be determined by the content of the discussion, as the context needed for interpreting any given turn of talk will vary. In the excerpt above, understanding Brad's contribution requires a window of seven turns of talk, while understanding William's contribution in Line 6 requires only two: he is answering Maria's question in Line 5, while Brad is addressing both of Maria's questions (Lines 1 and 5) and building on the contributions of Jill, Lesenia, and William. Note that if a larger window were used to analyze William's contribution, he would be credited for making connections to the internal consultants and their concerns, which he does not seem to be doing. In other words, a larger window may produce more false positive connections. However, while humans can identify relational context with relative ease, manual annotation of discourse is not a scalable approach. As large volumes of rich process data are increasingly produced by a variety of CSCL environments, rapid and reliable procedures for measuring recent temporal context are necessary for analyses of complex, collaborative thinking at scale.

## Existing approaches

Research on natural language processing suggests that identifying changes in topic or turn-by-turn references to prior utterances cannot be reliably automated (Rosé et al., 2008). Even if automated processes were feasible, the excerpt above illustrates a key problem with segmentation by changes in topic. In Line 5, Maria *is* changing the topic—from the needs of the internal consultants to the needs of the patients—but Brad (Line 7) uses that as an opportunity to make a point about how, as engineers, the project team needs to consider the concerns of multiple different stakeholders when designing products. That is, he is making a connection between the two topics (Lines 1–4 and Lines 5–6). Moreover, even if a given utterance does not make explicit reference to any of the immediately preceding turns of talk, those utterances nonetheless form the *common ground* for that part of the conversation and are likely to influence what participants say and do (Clark, 1996).
Thus for both practical and theoretical reasons, researchers typically use a moving window of fixed length (e.g., a particular duration of time or number of turns of talk) to analyze collaborative connection-making in recent temporal context. Instead of creating summary values for all utterances in a conversation, a moving window analysis computes a value for each utterance, based on the content of that utterance and the content of preceding utterances that are contained within the window. This type of analysis has been used to explore intragroup interactions (Dyke et al., 2012) and to study connection-making in collaborative discourse (Csanadi, Eagan, Shaffer, Kollar, & Fischer, 2017; Quardokus Fisher, Hirshfield, Siebert-Evenstone, Arastoopour, & Koretsky, 2016; Siebert-Evenstone et al., 2016; Sullivan et al., 2018; Suthers & Desiato, 2012).

One technique that uses moving windows to model connections in recent temporal context is *epistemic network analysis* (ENA) (Shaffer, 2018; Shaffer, Collier, & Ruis, 2016; Shaffer & Ruis, 2017). ENA takes interaction data coded for elements of complex thinking and constructs weighted network models that represent the structure of connections made among those elements by each individual in the dataset. For each line in a conversation, ENA computes a network of connections within the recent temporal context as defined by the window length; that is, ENA identifies the co-occurrences of codes between a given utterance and the previous utterances in the conversation that fall within the window. Proceeding line-by-line through the data, ENA accumulates these networks for each individual to model the unique connections contributed by that person to the conversation. Critically, the length of the moving window—in ENA and in other window-based techniques for modeling conceptual connectivity—defines which *co-occurrences of codes* are hypothesized to represent *meaningful cognitive connections*.

## The challenge

A key challenge for window-based models of complex, collaborative thinking is thus to determine a fixed window length that is sufficiently long to capture the recent temporal context for most utterances but not so long as to overrepresent connections that are not meaningful. Because the length of the window determines which connections are included in the model, interpretation of the model may be significantly affected by the choice of window length (Gleicher, 2016). Researchers typically make such determinations on a case-by-case basis, usually without detailing the method used, and an extensive literature search revealed no scalable, objective techniques for measuring recent temporal context. In what follows, we present a novel method for determining window length that minimizes the need for human annotation and provides both qualitative and quantitative warrants for using a certain window length to analyze collaborative connection-making in the specified setting.

## Methods

### Setting and data

In this study, we analyzed the collaborative interactions of students in the engineering virtual internship *Nephrotex* (Arastoopour, Shaffer, Swiecki, Ruis, & Chesler, 2016; Chesler, Arastoopour, D'Angelo, Bagley, & Shaffer, 2013; Chesler et al., 2015). In *Nephrotex*, students work in project teams to design an ultrafiltration membrane for a hemodialysis system. The virtual internship is divided into a series of activities that simulate steps in the design process, including reviewing research reports; designing prototypes; discussing design choices with teammates and an engineering advisor; and addressing the needs of internal consultants and external clients. Students interact with their teams and with their engineering advisor through an online instant message program (chat), and the system automatically records all chat conversations for subsequent analysis. *Nephrotex* takes approximately 15 hours to complete.

Chat conversations (N = 54,896 chats) were collected from 20 runs of *Nephrotex* at five institutions in the United States. Participants (N = 652) were first- and second-year college students using *Nephrotex* as part of an engineering course. Students were randomly assigned to project teams of 4–5 members.

### Window length annotation

To measure recent temporal context in this setting, we randomly selected 200 utterances from the 54,896 chats in the *Nephrotex* dataset. For each chat, two independent raters identified all previous referents in the conversation. These annotations indicate, for each utterance, the window containing that utterance's recent temporal context, where window length is the number of chats from the referring utterance to the earliest referent, inclusive. In the excerpt above, for example, the window length for Brad's utterance is seven lines: the referring chat (Line 7); the furthest referent (Maria's chat in Line 1); and the intervening five chats.

To calculate agreement between the two independent raters, we computed Cohen's κ (kappa) for each window length. Kappa was calculated for each window size, *x*, by assigning a “1” to any utterance that a given rater determined to have window length *x* and a “0” to all other utterances. Kappa thus indicated the extent to
which the two raters agreed in their assessments of each utterance's recent temporal context. To determine whether kappa scores for the sample (200 chats) could be generalized to the population from which they were drawn (> 50,000 chats), we computed Shaffer's $\rho$ (rho) to estimate the expected Type I error rate of kappa given the sample size (Eagan, Rogers, Pozen, Marquart, & Shaffer, 2016; Shaffer, 2017).

## Effects of window length on models of connectivity

We tested our technique for empirically determining window length by analyzing a portion of the *Nephrotex* dataset using ENA (for a detailed description of ENA methodology, see Shaffer, Collier, & Ruis, 2016). Specifically, we used ENA to model data from two implementations of *Nephrotex* (48 students; 5,757 chats) at window length *x* for each *x* ∈ {1, 2, ..., 13}. At each window length, we compared the networks of (a) students using an engineering virtual internship for the first time (novices; *n* = 24), and (b) students using *Nephrotex* after using a different engineering virtual internship (relative experts; *n* = 24).

The data were coded using an automated coding algorithm for five elements of engineering design: (1) performance parameters: the functional attributes of a design; (2) design decisions: the process of making design choices, including prioritization and tradeoffs; (3) client requests: considering the concerns and needs of stakeholders; (4) data: considering technical or numeric information; and (5) collaboration: facilitating inclusivity and teamwork in the design process. Inter-rater reliability was computed for each code separately, and all codes had a kappa value greater than 0.75; each kappa value was statistically significant at $\alpha$ < 0.05 and $\kappa$ > 0.65: $N = 200, \kappa$ > 0.75, $\rho(0.65)$ < 0.05.

## Results

### Measurement of recent temporal context

Of the 200 chats examined, 49 (24.5%) made no reference to prior chats, and 51 (25.5%) referenced only the previous chat. Thus, a window length of two chats would capture the relevant connections for 50% of the utterances in the sample. However, as Table 1 shows, it is not until a window length of seven that the relevant connections were captured for more than 95% of the utterances. Subsequent increases in window length resulted in only very small improvements, and no utterance required a window length of more than 18 chats.

<table><caption>Table 1: Number and proportion of utterances with complete recent temporal context at each window length</caption><thead><tr><th>Window Length</th><th>Number (and Percentage) of Utterances with Complete RTC</th><th>Increase in Percentage of Utterances with Complete RTC over Previous Window Length</th><th>Cohen's κ</th><th>Shaffer's ρ (for κ > 0.65)</th></tr></thead><tbody><tr><td>1</td><td>49 (24.5%)</td><td></td><td>0.96</td><td><strong>&lt; 0.01**</strong></td></tr><tr><td>2</td><td>100 (50.0%)</td><td>+25.5%</td><td>0.95</td><td><strong>&lt; 0.01**</strong></td></tr><tr><td>3</td><td>131 (65.5%)</td><td>+15.5%</td><td>0.96</td><td><strong>&lt; 0.01**</strong></td></tr><tr><td>4</td><td>158 (79.0%)</td><td>+13.5%</td><td>0.97</td><td><strong>&lt; 0.01**</strong></td></tr><tr><td>5</td><td>170 (85.0%)</td><td>+6.0%</td><td>0.88</td><td><strong>&lt; 0.01**</strong></td></tr><tr><td>6</td><td>182 (91.0%)</td><td>+6.0%</td><td>0.84</td><td><strong>&lt; 0.01**</strong></td></tr><tr><td>7</td><td>192 (96.0%)</td><td>+5.0%</td><td>0.94</td><td><strong>&lt; 0.01**</strong></td></tr><tr><td>8</td><td>195 (97.5%)</td><td>+1.5%</td><td>0.91</td><td><strong>0.03*</strong></td></tr><tr><td>9</td><td>195 (97.5%)</td><td>+0.0%</td><td>0.91</td><td><strong>0.02*</strong></td></tr><tr><td>10</td><td>197 (98.5%)</td><td>+1.0%</td><td>0.85</td><td>0.11</td></tr><tr><td>11</td><td>197 (98.5%)</td><td>+0.0%</td><td>0.66</td><td>0.39</td></tr><tr><td>12</td><td>198 (99.0%)</td><td>+0.5%</td><td>0.80</td><td>0.22</td></tr><tr><td>13</td><td>198 (99.0%)</td><td>+0.0%</td><td>0.80</td><td>0.22</td></tr></tbody></table>

To calculate the level of agreement between the two raters and to determine whether the findings for the sample (200 chats) are generalizable to the population from which the chats were sampled (> 50,000 chats), we computed kappa and rho for each window length (see Table 1). For all window lengths up to nine, agreement between the two raters was statistically significant for $\kappa$ > 0.65 ($N = 200, \kappa$ ≥ 0.84, $\rho(0.65)$ < 0.05), which indicates that the level of agreement between the two raters would have been $\kappa$ > 0.65 for those window lengths had they evaluated the entire dataset. While the kappa scores obtained for window lengths greater than nine were not
statistically significant at a Type I error rate of $\alpha < 0.05$ (i.e., $\rho(0.65) > 0.05$), this is due to the extremely low number of utterances that refer to chats more than 8 turns earlier. Because the goal of this method is to identify the shortest window that captures the full recent temporal context for most utterances, a window length of seven appears to be optimal for this dataset.

## Effects of window length on model features and interpretation

Figure 1 shows the ENA models for moving window lengths one through four (MW1–4). The plotted points (top row), each of which represents the network of one student, show increasing discrimination between novices (red) and relative experts (blue) with increasing window length. The difference graphs (bottom row) compute the difference in connection strengths between the mean networks of the novices and relative experts, showing which connections were stronger in which group. Note that the network graph for MW1, which includes only connections made within individual utterances, is very different from the network graph for MW2, which includes both connections within a given utterance and connections to the immediately preceding utterance. With increasing window length, the network graphs become more stable, as does the interpretation of the model. As the difference graphs for MW3 and MW4 indicate, the novice students made stronger connections among *data, performance, and design decisions*, while the relative expert students made stronger connections between *client requests* and both *performance* and *design decisions*. In other words, the models for those window lengths suggest that novices were more likely to focus on the technical aspects of the design problem, while relative experts were more likely to focus on the needs of stakeholders in the design process.

<figure><img src="image_79.png" /><figcaption>**Figure 1.** ENA models of the same *Nephrotex* data at four moving window (MW) lengths. The plotted points (top row) show the network locations of the novices (red) and relative experts (blue), with the corresponding means (colored squares) and 95% confidence intervals (boxes). The difference graphs (bottom row) show the difference in connection strength between the means of the two groups.</figcaption></figure>

The summary statistics for the ENA models at all window lengths tested (MW1–13) are shown in Table 2. With the exception of MW5, all ENA models produced using a moving window larger than two indicate a significant difference ($p < 0.05$) between novices and relative experts with a moderate-to-large effect size ($d > 0.60$). However, at a window length of seven, which was hypothesized to be optimal based on our qualitative analysis, the difference between the two groups is significant at $p < 0.01$ for the first time, and the effect size increases almost 10% from the window length of six. The effect size increases further at MW8, but all window lengths from MW9 to MW13 produce summary statistics comparable to MW7.

Figure 2 shows the ENA model for MW7. Note that both the discrimination between the groups (left) and the structure of connections (right) are similar to the MW4 model shown in Figure 1. While the ability of ENA to discriminate between novices and relative experts is relatively robust to window length, interpretation of the model depends in part on the position of the network nodes in ENA space. The nodes are positioned based on the patterns of connectivity, which in turn are affected by changes in window size. Thus, differences in node
positions between models with different window sizes indicate potentially different interpretations of complex, collaborative thinking (1).

<table><caption>Table 2: ENA model discrimination between two study populations at each window length</caption><thead><tr><th>Window Length</th><th>t</th><th>p</th><th>Cohen's d</th></tr></thead><tbody><tr><td>1</td><td>0.91</td><td>0.37</td><td>0.26</td></tr><tr><td>2</td><td>1.58</td><td>0.12</td><td>0.45</td></tr><tr><td>3</td><td>2.15</td><td><strong>0.04*</strong></td><td>0.62</td></tr><tr><td>4</td><td>2.46</td><td><strong>0.02*</strong></td><td>0.72</td></tr><tr><td>5</td><td>1.55</td><td>0.13</td><td>0.45</td></tr><tr><td>6</td><td>2.44</td><td><strong>0.02*</strong></td><td>0.71</td></tr><tr><td>7</td><td>2.75</td><td><strong>&lt; 0.01**</strong></td><td>0.79</td></tr><tr><td>8</td><td>3.01</td><td><strong>&lt; 0.01**</strong></td><td>0.87</td></tr><tr><td>9</td><td>2.83</td><td><strong>&lt; 0.01**</strong></td><td>0.82</td></tr><tr><td>10</td><td>2.75</td><td><strong>&lt; 0.01**</strong></td><td>0.80</td></tr><tr><td>11</td><td>2.71</td><td><strong>&lt; 0.01**</strong></td><td>0.79</td></tr><tr><td>12</td><td>2.68</td><td><strong>0.01*</strong></td><td>0.78</td></tr><tr><td>13</td><td>2.70</td><td><strong>0.01*</strong></td><td>0.79</td></tr></tbody></table>

<figure><img src="image_80.png" /><figcaption>Figure 2. ENA model of the Nephrotex data at MW7.</figcaption></figure>

To assess at what window length the network node positions stabilize, we plotted the locations of the nodes at each window length for both the first (x) and second (y) dimensions of the ENA model (see Figure 3).

<figure><img src="image_81.png" /><figcaption>Figure 3. Node positions (x and y coordinates) in the ENA models at each window length.</figcaption></figure>

The x-coordinates of the nodes maintain the same relative order starting at MW2, and the relative spacing stabilizes starting at MW7. After MW7, there are no significant changes in either the relative order or the relative
spacing of the nodes. However, the relative order of the y-coordinates of the nodes does not begin to stabilize until MW7. *Data* (yellow) and *collaboration* (light blue) change position relative to one another between MW8 and MW9, but the difference is small and both nodes remain near the origin; thus, the change does not affect interpretation of model. A moving window length of seven is thus where the node positions in ENA space stabilize on both dimensions. Window lengths greater than seven do not add significant information to the model, and window lengths less than seven show different patterns of connectivity, even though models using smaller window lengths do discriminate between the novice and relative expert students.

Discussion

Our goal was to develop a method that (a) provides both qualitative and quantitative warrants for determining the optimal window length for use in moving window analyses of conceptual connectivity in CSCL contexts, while (b) minimizing the number of items requiring human evaluation. To assess this approach, two independent raters analyzed a random sample of 200 student chats (< 0.01% of the 54,896 chats in the dataset). This method identified MW7 as the optimal window length for analyzing these data. We then constructed ENA models of the data that differed only in the choice of window length. This analysis confirmed that a model with a window size of 7 both (a) provides statistical discrimination between groups known to exhibit different patterns of conceptual connectivity based on prior research (Chesler et al., 2015) and (b) provides a stable interpretation of the ENA model. This analysis suggests that statistical discrimination between two groups may be fairly robust to window length once some minimum length is reached (in this case, after MW3 with the exception of MW5), but that model features and interpretation may be more sensitive to window length (the features and interpretation of the ENA models do not fully stabilize until MW7).

As a result, we argue that annotating a subset of data for furthest referents makes it possible to analyze recent temporal context and thus determine an appropriate window length to be used in analyses of complex, collaborative thinking. Importantly, this method minimizes the need for human annotation while providing both qualitative and quantitative warrants for choosing a particular window length. While we describe this method by presenting results from one CSCL context (Nephrotex) and one learning analytic technique (ENA), we believe that approaches based on annotating a sample data for furthest referents will be compatible with different CSCL settings, different theories of collaborative discourse, and different methods for modeling conceptual connectivity using moving windows. Of course, future research should test our method by repeating this study using other data and other learning analytic models.

While there are many avenues for additional research, this study suggests that hand annotation of a relatively small number of utterances can be used to measure recent temporal context. Critically, this method provides a warrant for making generalizations to the population from which the hand-annotated sample was drawn, making it suitable for analyses of complex, collaborative thinking at scale.

Endnotes

(1) The goodness of fit measures are high (both Spearman's and Pearson's *r* > 0.95) for all 13 models except MW1 (*r* ≥ 0.85), which means that the relative positions of the network nodes provide good interpretations of the differences between networks in the model.

References

Arastoopour, G., Shaffer, D. W., Swiecki, Z., Ruis, A. R., & Chesler, N. C. (2016). Teaching and assessing engineering design thinking with virtual internships and epistemic network analysis. *International Journal of Engineering Education, 32*(3B), 1492–1501.

Arvaja, M., Salovaara, H., Häkkinen, P., & Järvelä, S. (2007). Combining individual and group-level perspectives for studying collaborative knowledge construction in context. *Learning and Instruction, 17*(4), 448–459.

Chesler, N. C., Arastoopour, G., D'Angelo, C. M., Bagley, E. A., & Shaffer, D. W. (2013). Design of a professional practice simulator for educating and motivating first-year engineering students. *Advances in Engineering Education, 3*(3), 1–29.

Chesler, N. C., Ruis, A. R., Collier, W., Swiecki, Z., Arastoopour, G., & Shaffer, D. W. (2015). A novel paradigm for engineering education: Virtual internships with individualized mentoring and assessment of engineering thinking. *Journal of Biomechanical Engineering, 137*(2), 024701:1–8.

Clark, H. H. (1996). *Using language*. Cambridge University Press.

Csanadi, A., Eagan, B., Shaffer, D. W., Kollar, I., & Fischer, F. (2017). Collaborative and individual scientific reasoning of pre-service teachers: New insights through epistemic network analysis (ENA). In B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (Eds.), *Making a difference: Prioritizing equity and access*
in CSCL: 12th International Conference on Computer-Supported Collaborative Learning (Vol. I, pp. 215-222).

DiSessa, A. A. (1988). Knowledge in pieces. In G. Forman & P. Pufall (Eds.), *Constructivism in the computer age* (pp. 47-70). Hillsdale, NJ: Erlbaum.

Dyke, G., Rohit Kumar, R., Hua, A., & Rose, C. P. (2012). Challenging assumptions: Using sliding window visualizations to reveal time-based irregularities in CSCL processes. In *Proceedings of the 10th International Conference of the Learning Sciences* (pp. 363-370).

Eagan, B. R., Rogers, B., Pozen, R., Marquart, C., & Shaffer, D. W. (2016). rhoR: Rho for inter rater reliability (Version 1.1.0). Retrieved from https://cran.r-project.org/web/packages/rhoR/index.html

Garrison, D. R., & Akyol, Z. (2013). The community of inquiry theoretical framework. In M. G. Moore (Ed.), *Handbook of distance education* (Vol. 3, pp. 104-120). Routledge.

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. *American Journal of Distance Education, 15*(1), 7-23.

Gleicher, M. (2016). A framework for considering comprehensibility in modeling. *Big Data, 4*(2), 75-88.

Linn, M. C., Eylon, B.-S., & Davis, E. A. (2004). The knowledge integration perspective on learning. In M. C. Linn, E. A. Davis, & P. Bell (Eds.), *Internet environments for science education* (pp. 29-46). Mahwah, NJ: Lawrence Erlbaum Associates.

Quardokus Fisher, K., Hirshfield, L., Siebert-Evenstone, A. L., Arastoopour, G., & Koretsky, M. (2016). Network analysis of interactions between students and an instructor during design meetings. In *Proceedings of the American Society for Engineering Education* (p. 17035). ASEE.

Rosé, C., Wang, Y.-C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., & Fischer, F. (2008). Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. *International Journal of Computer-Supported Collaborative Learning, 3*(3), 237-271.

Seixas, P. (1993). The community of inquiry as a basis for knowledge and learning: The case of history. *American Educational Research Journal, 30*(2), 305-324.

Shaffer, D. W. (2012). Models of situated action: Computer games and the problem of transfer. In C. Steinkuehler, K. D. Squire, & S. A. Barab (Eds.), *Games, learning, and society: Learning and meaning in the digital age* (pp. 403-431). Cambridge, UK: Cambridge University Press.

Shaffer, D. W. (2017). *Quantitative ethnography*. Madison, WI: Cathcart Press.

Shaffer, D. W. (2018). Epistemic network analysis: Understanding learning by using big data for thick description. In F. Fischer, C. E. Hmelo-Silver, S. R. Goldman, & P. Reimann (Eds.), *International handbook of the learning sciences* (pp. 520-531). New York, NY: Routledge.

Shaffer, D. W., Collier, W., & Ruis, A. R. (2016). A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. *Journal of Learning Analytics, 3*(3), 9-45.

Shaffer, D. W., & Ruis, A. R. (2017). Epistemic network analysis: A worked example of theory-based learning analytics. In C. Lang, G. Siemens, A. F. Wise, & D. Gasevic (Eds.), *Handbook of learning analytics* (pp. 175-187). Society for Learning Analytics Research.

Siebert-Evenstone, A. L., Arastoopour, G., Collier, W., Swiecki, Z., Ruis, A. R., & Shaffer, D. W. (2016). In search of conversational grain size: Modeling semantic structure using moving stanza windows. In C.-K. Looi, J. Polman, U. Cress, & P. Reimann (Eds.), *Transforming learning, empowering learners: The International Conference of the Learning Sciences (ICLS) 2016* (Vol. I, pp. 631-638).

Sullivan, S. A., Warner-Hillard, C., Eagan, B. R., Thompson, R., Ruis, A. R., Haines, K., ... Jung, H. S. (2018). Using epistemic network analysis to identify targets for educational interventions in trauma team communication. *Surgery, 163*(4), 938-943.

Suthers, D. D., & Desiato, C. (2012). Exposing chat features through analysis of uptake between contributions. In *45th Hawaii International Conference on System Science* (pp. 3368-3377). IEEE.

## Acknowledgments

This work was funded in part by the National Science Foundation (DRL-1661036, DRL-1713110), the Wisconsin Alumni Research Foundation, and the Office of the Vice Chancellor for Research and Graduate Education at the University of Wisconsin-Madison. The opinions, findings, and conclusions do not reflect the views of the funding agencies, cooperating institutions, or other individuals.
# Idea Thread Mapper: Designs for Sustaining Student-Driven Knowledge Building Across Classrooms

Jianwei Zhang and Mei-Hwa Chen

jzhang1@albany.edu, mchen@albany.edu

University at Albany, State University of New York

**Abstract:** This research project aims to address the intra- and inter-community challenges associated with student-driven knowledge building over longer terms and across classrooms, in which students take on collective responsibility for charting the course of sustained inquiry and dynamic interaction. We designed the Idea Thread Mapper (ITM) system to support knowledge building in each classroom community as well as idea interaction across communities. At a local classroom level, students engage in focused inquiry and knowledge-building discourse. As their inquiry proceeds, they carry out metacognitive conversations to frame/reframe their collective inquiry directions and processes and synthesize their journey of thinking over time. For cross-community interaction, they selectively share productive idea threads and syntheses in a cross-community space for idea contact with broader knowledge builders. Supported by ITM, we conducted design-based research in a network of Grades 3-6 classrooms. Findings and implications are discussed.

## Introduction

Research on computer-supported collaborative learning (CSCL) has made substantial progress in understanding the sociocultural and cognitive processes of collaborative discourse and knowledge building. However, as a field, we still face the challenge of how to enable sustained educational change and transformation in line with the ethos of CSCL (Chan, 2011; Stahl & Hesse, 2009; Wise & Schwarz, 2017). Tackling this grand challenge requires new research-based innovations to extend CSCL to longer timescales and larger social scales (Stahl, 2013), transcending the traditional boundaries associated with curriculum areas and school years. Aligned with these directions, this paper presents our research to create new learning infrastructures and designs for supporting student-driven knowledge building over long terms across classroom communities.

Design efforts to support student-driven, long-term knowledge building in each classroom need to address two competing demands: to provide the needed classroom guidance and, at the same time, to foster students' high-level agency over creative knowledge processes (Zhang et al., 2018). Extensive research has been conducted to scaffold collaborative learning using collaboration scripts, which are pre-designed to specify, sequence, and distribute various actions and procedures among learners in order to guide effective interactions (Kirschner & Erkens, 2013). Scripted collaboration tends to focus on short-term inquiry activities, in which students are given limited responsibility for charting the process of collaboration and inquiry (cf. Wise & Schwarz, 2017). Long-term, creative knowledge building involves highly dynamic interactions that are difficult to predict and pre-script (Sawyer, 2007). In proudcitve knowledge building communities, students need to take on collective responsibly for charting the course of sustained discourse and inquiry (Scardamalia, 2002). The inquiry process continually evolves and deepens through members' interactive input (Zhang et al., 2011). Deeper problems and knowledge goals are identified over time as new understandings are developed, driving sustained cycles of inquiry. To guide effective and broad classroom implementation of dynamic knowledge building, new research is needed to clarify how such student-driven, open-ended, interactive processes can become socially organized and pedagogically supported while addressing curriculum expectations and contextual constraints (Zhang et al., 2018).

Efforts to support long-term knowledge building in each classroom can further be supported by a larger social infrastructure that allows productive ideas to flow between communities. Such knowledge infrastructures are essential to the productivity of real-world knowledge communities that work as interconnected fields (Csikszentmihalyi, 1999). It is an unsolved challenge for CSCL research to design such accumulative infrastructures and make each community's knowledge progress accessible to other classroom communities. Existing online collaboration systems can easily create a permanent record of student discourse; nevertheless, the knowledge advances achieved in the discourse are not visible or easily interpretable to outsiders who were not part of the discussion. In practice, online discussion spaces are typically reset each academic year for the new student cohort, dumping out the ideas of the previous students. There are emerging efforts to share online discussion spaces between different communities for collaborative knowledge building (e.g. Laferriere, Law, & Montané, 2012). The dominant strategy is to have the different communities directly view each other's online
discussion posts and participate in the same discussion space. However, students often find it difficult to read other communities' online posts and understand their progress and contexts.

The design and research work presented herein aims to address the challenges of how student-driven, long-term knowledge building can be socially organized and how cross-classroom interaction can be supported. Drawing upon our previous research, we designed the Idea Thread Mapper (ITM) system, which was used to conduct design-based research in a network of Grades 3-6 classrooms.

## The design of Idea Thread Mapper

Zhang and colleagues have conducted a series of studies to examine how productive knowledge building communities operate for student-driven, continual idea improvement (e.g. Zhang et al., 2007; Zhang & Messina, 2010; Tao et al., 2015). The findings unveiled a socio-epistemic mechanism by which the student-driven, dynamic inquiry process is organized and scaffolded: emergent reflective structuration. Different from pre-scripted guidance structures, teachers and students engage in ongoing reflection on their inquiry and discourse and co-construct various inquiry structures and resources to support their collective work. The co-constructed structures include shared frames and representations about what the community should investigate, how their inquiry and discourse should be conducted, and who will work with whom in what areas (Tao & Zhang, 2018; Zhang et al., 2018). Guided by the co-constructed structures, students monitor their ongoing knowledge building practices and develop coordinated efforts to advance their collective agenda of inquiry without merely relying on the teacher to lead them through the whole process. Based on the insights gained, we designed a new collaboration system: Idea Thread Mapper (ITM, http://idea-thread.net), which interoperates with Knowledge Forum (Scardamalia & Bereiter, 2006) and potentially other collaborative learning platforms. ITM was originally created in 2012 (Chen, Zhang, & Lee, 2013; Zhang et al., 2012) and substantially redesigned and upgraded during 2017-2018. The new ITM incorporates systematic support for co-organizing long-term knowledge building in each classroom community as well as idea interaction across communities. At a local classroom level, students engage in focused inquiry and discourse in their own classroom's discourse space, carry out metacognitive conversations to reflect on shared focuses and insights emerged from the distributed discourse, and synthesize their journey of inquiry including the progress achieved and deeper problems to be addressed. For cross-community interaction, they selectively share productive idea threads and syntheses in a cross-community space for dynamic idea contact with their “buddy classrooms.” Across both social levels, students take on high-level responsibility for structuring their unfolding inquiry directions and processes, reviewing progress, and planning for productive collaboration with the support of embedded analytics. Below we summarize the design principles and features.

## Principle 1: Reflective structuration: Co-organize the journey of inquiry as it unfolds

Current collaborative online environments in the forms of online forums and chatting lack support for students to structure and monitor their unfolding, collective journey of inquiry. Students' online discourse tends to focus on teacher-assigned topics and tasks. Also, with students' diverse idea input recorded in distributed online postings, it is often difficult for students and their teacher to monitor the collective knowledge progress achieved in the online discourse (Hewitt, 2001; Suthers, Vatrapu, Medina, Joseph, & Dwyer, 2008; Zhang, 2009). Therefore, to better support students' reflective monitoring and co-organizing of their collective inquiry, ITM incorporates meta-layer representations of student-constructed inquiry structures on the top of the distributed online discourse. The meta-layer inquiry structures serve to capture students' shared directions of inquiry emerging from their interactive discourse, visualize the unfolding strands of inquiry, and document shared progress in each strand of inquiry to inform students' deepening work. The specific features include:

(a) **Co-organizing shared wondering areas and inquiry threads:** As a high-level structure to guide each knowledge building inquiry that may last over several months, students co-organize their shared wondering areas and specific threads of inquiry (Figure 1) as their work proceeds. Each wondering area is a major direction of inquiry (e.g., brain), which is identified by the classroom members based on their interests and questions. Under each wondering area, members develop one or more inquiry threads, each of which investigates a more specific problem or challenge (e.g. How does the brain work to control body movement?) related to one (or more) wondering area. The wondering areas and threads of inquiry are emergent, co-formed structures, which are similar to “desire lines” —as apposed to pre-planned paths—formed in public spaces (Zhang et al., 2018). In a knowledge building initiative, students begin with open exploration and discourse to develop initial ideas and questions. Through monitoring emerging inquiry interests and evolving needs, they identify high-potential wondering areas and set up idea thread focus, which guide their collaborative discourse and joint inquiry to advance their understandings while continually identifying deeper problems. Each student can select one or more wondering areas as his/her primary focus of inquiry, and adjust his/her focus as the inquiry unfolds.
Students with shared interests form into spontaneous flexible groups to conduct collaborative inquiry and advance their understandings.

<figure><img src="image_82.png" /><figcaption>**Figure 1.** Project dashboard with a visual organizer of the collective wondering areas and idea threads. The thickness of each branch shows the intensity of online contributions in each area.</figcaption></figure>

(b) **Visualizing the conceptual, temporal and social profiles of each idea thread:** Drawing upon the temporal inquiry-thread analysis developed in our prior work (Zhang et al., 2007), ITM organizes and visualizes students’ online discourse in each idea thread as theme-based, temporal interactions. Each idea thread represents a conceptual line of inquiry that involves a sequence of discourse entries—possibly involving several build-on trees—investigating a shared inquiry object over a time period. Figure 2 shows the discourse in an idea thread developed by a group of fifth-graders focusing on how the brain works. The discourse entries (notes) and build-on responses are displayed in a two-dimension space based on the timeline of note creation and the authors involved, showing the temporal and social profile of student contributions in this line of work.

<figure><img src="image_83.png" /><figcaption>**Figure 2.** An idea thread developed by a Grade 5 classroom focusing on how the brain works. Each dot represents a note (click to open), and a line connecting two notes shows a build-on connection. The notes are displayed based on the time of creation (x-axis) and author (y-axis).</figcaption></figure>

(c) **Reflective documentation of Journey of Thinking in each thread of inquiry:** As the conversation in each thread of inquiry proceeds giving rise to new insights as well as deeper issues, the contributors co-compose a Journey of Thinking (JoT) synthesis to deliberate their idea progress. JoT is a reflective “super note” that
includes three sections: problems/issues explored, “big ideas” learned so far, and deeper research needed. Scaffolds are provided in each section to guide student reflection. For example, for the synthesis of the “big ideas” learned, students frame their ideas using “We used to think…now we understand…” (see Table 1).

<table><caption>Table 1: A Journey of Thinking written by a group of 5th graders in the idea thread about breathing</caption><tr><td>Questions explored:</td><td>[We want to understand:] how do the tissues use the oxygen? How and why do we breathe?</td></tr><tr><td>Big ideas learned:</td><td>[We used to think:] That there was not a big process to this.<br/>[We now understand:] We get oxygen from air we breath in. The oxygen goes through the thin walls of the air sacs. With the help of hemoglobin, oxygen passes through the air sacs into the blood vessels throughout the body. As that happens CO2 goes out of your body.<br/>...</td></tr><tr><td>Deeper research needed:</td><td>[We need to do more learning] about how air get around the body.<br/>[We need better theories about] Where is hemoglobin exactly located in the body and does it do anything else besides carry oxygen to the blood, and bring CO2 back into the lungs.<br/>[We need to further understand] ...more about tissues.</td></tr></table>

(d) **Mapping collective progress and connections across different threads of inquiry:** At a higher level, the collective landscape of a whole knowledge building initiative is mapped out as clusters of idea threads that investigate interrelated issues through the contributions of all the members (Figure 3). The map of idea threads shows how the different threads of inquiry have evolved over time and further visualizes cross-thread connections, which show student efforts to build on and integrate ideas from the related areas to develop coherent understandings (see analysis in Zhang et al., 2018). The connections include (a) build-on links among the notes in the different idea threads; and (b) connective “bridging contributions,” each of which simultaneously investigates two or more topics focusing on the interrelated issues and cross-cutting ideas. For example, in Figure 3, there is a rich set of notes bridging thread #1 (heart and lungs), #3 (brain control), and #4 #4 (blood), as marked by the dashed vertical lines. Thus, with the visualizations and analytics embedded in the map of idea threads, students can reflect on the temporal process of their inquiry as well as their social participation (e.g. the number of notes and authors involved) and connectedness.

<figure><img src="image_84.png" /><figcaption>Figure 3. Mapping student discourse in the different idea threads developed in a Grade 5 classroom that studied the human body. Each color stripe represents an idea thread addressing a principal problem, spanning from the</figcaption></figure>
earliest to the latest note created. Each dot shows a note. A line connecting two notes indicates a build-on connection. Dashed vertical lines mark the notes belonging to multiple threads bridging the different lines of discussions.

## Principle 2: Boundary crossing between knowledge building communities

On the basis of the student-driven inquiry in each classroom community, ITM further incorporates supports for cross-community interaction, which is approached using a multi-level, boundary-crossing design (Zhang et al., 2017). As students engage in focused inquiry and conversations and make progress in their own classroom, they reflect on their progress of inquiry and generate JoT super notes, as synthetic knowledge artifacts. These artifacts are used as boundary objects (Star & Griesemer, 1989) to support higher-level interaction across classroom communities.

In the related literature, boundary objects are defined as artifacts (e.g. reports, tools, models) that can be used to bridge the boundaries (discontinuities) between different social worlds (Star & Griesemer, 1989). Objects from a community often have contextual meanings that are not easily accessible to other communities. What makes boundary objects effective for bridging different communities of practice is their interpretative flexibility as a “means of translation” (Star & Griesemer, 1989): they have a structure that is common enough to make them recognizable across the different social contexts and allow different communities to interact and work together. Through interacting with shared boundary objects, members from different communities can identify, understand, and reflect on their different practices, so they can develop better understandings and practices within their own community and potentially create new forms of practices at the intersection of the different communities (Akkerman & Bakker, 2011).

As noted previously, individual online discourse entries are hard to be used as boundary objects for cross-classroom sharing and interaction. Therefore, ITM uses high-level structural representations in the form of idea threads and JoT syntheses as boundary objects. These synthetic artifacts capture the journeys of inquiry taking place in different classrooms using a set of common scaffolds (see Table 1), so they can help the different communities to understand one another’s inquiry directions and knowledge progress. Specifically, ITM incorporates a cross-classroom sharing space for each knowledge building initiative (See Figure 1). The teacher can create “buddy connections” with other classrooms that are studying the same or related content areas, which may come from the same or a different school site. Students can then interact with peers in the buddy classrooms in the follow ways:

(a) **Ongoing access to the buddy classrooms' wondering areas and idea threads:** Students can see what the buddy classrooms are researching based on their wondering areas and idea threads (see Figure 1), and compare the wondering questions pursued by the different communities. For deeper information, they can also directly view the online discourse in any of the idea threads (in a read-only mode).

(b) **Ongoing sharing of the JoT syntheses:** Students can access the syntheses generated by the buddy classrooms, and find the most relevant syntheses based on the keyword-based index or using the search tool. As noted previously (Table 1), each JoT synthesizes the progress in a thread of inquiry including the questions explored, “big ideas” learned, and deeper research needed. Thus, by viewing other classrooms’ JoT syntheses, students can get a sense of their journey of inquiry and learn from their “big ideas” and deepening questions. For example, two upper elementary classrooms collaborated in their inquiry about human body systems. While both classrooms investigated how the heart works as part of their inquiry, one classroom’s inquiry questions focused more how the healthy heart functions while the other classroom researched problems about how holes in the heart affect its function. Through reading the heart-related JoT of the other classroom, students realized the different questions asked and discussed the two perspectives to develop coherent understandings (Zhang et al., 2017).

(c) **Live Super Talk across buddy classrooms:** Students can propose challenging issues as potential topics for cross-classroom discussion, which is called “Super Talk.” For example, in our research, four Grade 5 classrooms studying the human body worked as buddy classrooms. As students made progress in their own classroom understanding how the different body systems work, a few students in one of the classrooms were intrigued by the question of how the human body grows. They proposed this challenging issue as a Super Talk topic shared by the four classrooms. A total of 20 students from the four classrooms contributed 23 notes, looking into how people grow as related to the brain, spine, bones, muscles, skin, digestion, cells, and genetics.

## Principle 3: Embedded analytics to leverage student reflection and idea interaction

ITM integrates a set of analytics and visualizations in the local discourse space and cross-community space (see elements of analytics in Figures 1-3). Specific analytics embedded in the online discourse spaces aim to foster students’ reflective monitoring of the temporal, social, and conceptual profiles of their inquiry. A summary of
the key analytical information is also reported in the Activity Radar, with which students can trace their personal and collective contributions to the various wondering areas and idea threads in a selected time period. Additionally, in collaboration with analytics researchers we tested a set of automated tools to help students monitor inquiry directions, personal contributions, and collaborative connections. For example, to help students monitor emerging topics/areas of inquiry and review progress, ITM provides an automated tool for topical modeling. Specifically, it uses an augmented Latent Dirichlet Allocation (LDA) (Blei, 2012) method to retrieve conceptual topics from student online discourse in relation to topical structures detected from the relevant expert texts (e.g. Wikipedia article). For each topic, the tool further recommends the most relevant online posts, ranked based on semantic relevancy. The user can then review the posts as potential entries for setting up a new idea thread, and define the topic name based on the top keywords. To support student review of each idea thread, ITM further incorporates automated analysis enabled by LightSIDE (formerly known as TagHelper — see Rosé et al., 2008). LightSIDE learns from human-coded discourse data to classify student online posts based on contributions types, including questioning, referencing sources, theorizing, and using evidence (see Tao & Zhang, 2018 for the coding scheme). The automated classification is used to help students monitor the various online discourse moves in each thread of inquiry in order to make informed improvement. To facilitate cross-community interaction, we further tested analytics to gauge the semantic similarity between the idea threads of different communities. Students can find relevant inquiry work conducted by other classrooms, with which they may develop buddy connection and collaboration.

## Design-based research

ITM has been tested and refined through our multi-year design-based research conducted in a set of Grades 3-6 classrooms (Zhang et al, 2013, 2014, 2017, 2018; Zhang & Yuan, 2018). Students in each classroom investigated a science topic over multiple months with the support of ITM and Knowledge Forum. Their work integrated whole class knowledge building conversations, individual and group reading, experiments and observations, and so forth. Major questions, ideas, and findings generated through face-to-face activities were contributed online for continual discourse. With their teacher's support and using ITM, students engaged in metacognitive meetings to reflect on the emerging interests and ideas, form shared wondering areas and inquiry directions, and organize their collaborative roles to guide their work in the shared inquiry areas. The visualizations of the wondering areas and collaborative roles were used to guide their monitoring and planning of the ongoing participation and discourse. As the inquiry deepened within their own classroom community, students reviewed productive threads of ideas and composed JoT syntheses, which were shared with their buddy classrooms to stimulate cross-community interaction. Multiple sources of data were collected and analyzed including video records of metacognitive meetings in the classrooms, online discourse and JoT syntheses, individual reflective portfolios, and interviews with the teachers and students. We summarize the main findings below focusing on the within-community and cross-community processes.

## Co-constructing shared inquiry structures serves to inform students' ongoing discourse and enhance student understandings

The data analysis documented the processes by which students worked with their teacher to co-construct shared inquiry structures to guide their collective work, including shared frames of collective wondering areas and idea thread topics, mapping of cross-thread connections, and ongoing syntheses of JoT. Such structures served to inform students' ongoing contribution and interaction. With ITM reflection, students were able to conduct more productive and connected discourse: there was a higher proportion of notes identifying deeper questions, using evidence to examine explanations, and integrating different ideas to address challenging issues. Their discourse also became more connected with more build-on links and more connective notes that investigated interconnected inquiry themes and issues (Zhang et al., 2013; 2014; 2018). The enhancement to student inquiry and discourse contributed to improving students' knowledge outcomes. Students with ITM achieved more sophisticated understandings of the science topics as measured based on the content analysis of their reflective portfolios (Zhang et al., 2018). Each student also developed clearer awareness of their community's knowledge goals and progress beyond their personal interests, conducive to enhancing personal learning and collaborative connections (Zhang et al., 2014).

## Cross-classroom interaction leads to mutual learning and advancement

We further tested cross-classroom interaction for knowledge building (Zhang et al., 2017, 2018). For example, a design-based research study was conducted in two grades 5/6 classrooms that studied human body systems over a 10-week period (Zhang et al., 2017). As the students conducted focused inquiry and discourse within their
own community, they reviewed productive threads of ideas and posted JoT syntheses in a cross-community space. The syntheses were written by students using ITM's scaffolds for JoT reflection: *Our problem, We used to think...and now we understand..., We need deeper research*. A set of syntheses from the previous classrooms studying human body systems was also posted in the cross-community space. The analyses of the classroom discussions and student interviews suggest that the students engaged in active and substantial interactions with the JoT syntheses from other classrooms. They identified relevant and interesting inquiry topics from other classrooms and compared the different perspectives and ideas, which triggered students' deep reflection on their own inquiry. Further iterations of this design-based research expanded the cross-classroom interaction to more classrooms studying human body systems (Zhang & Yuan, 2018). Content analysis showed that the students' JoT syntheses had a high quality of reflection in capturing solid knowledge progress and deep questions. Social network analysis indicated that through reading the JoT syntheses from their buddy classrooms, students built extensive peer connections focusing on their own inquiry areas as well as the broader scientific concepts. They further identified challenging topics (e.g. how people grow) at the intersection of their interests to conduct cross-classroom collaboration and discourse, putting together their knowledge about the specific body parts to develop coherent explanations.

## Conclusions and implications

ITM and the related studies contribute new strategies to address the intra- and inter-community challenges of long-term knowledge building. Particularly, this research elaborates a reflective structuration approach to implementing long-term inquiry and knowledge building for educational transformation. In a knowledge building initiative (unit) that may extend over multiple weeks or months, the teacher can work with his/her students to co-structure their collective journey of inquiry over time without extensive pre-scripting of the inquiry directions and tasks. Through ITM-supported reflective processes, students can work with their teacher to co-structure the high-level issues about their inquiry, such as what to investigate, through what processes, by whom/with whom as their inquiry unfolds (Zhang et al., 2018; Tao & Zhang, 2018). ITM provides meta-layer representations and tools to capture the co-constructed directions and structures of inquiry, which are visualized to aid members' reflective monitoring and participation. The ITM-based research further elaborates a boundary crossing approach to cross-community interaction. While students engage in focused inquiry and discourse within their own classroom, they generate reflective Journey of Thinking syntheses, as boundary-crossing objects, and participate in the Super Talk to investigate challenging problems together. In sum, the ITM-based technology designs and classroom processes shed light on new designs of socio-technological infrastructures for enabling networks of knowledge building communities, each of which engage in continual idea advancement while building on the knowledge work of other communities. ITM's generic data exchange model supports data exchange with other online discourse environments besides Knowledge Forum. Building on the existing advances, we are currently designing analytics and feedback tools to help teachers and their students monitor emergent inquiry directions and progress in their classroom while discovering potential knowledge connection with other communities.

## References

Akkerman, S. F., & Bakker, A. (2011). Boundary crossing and boundary objects. *Review of Educational Research, 81*(2), 132-169.

Blei, D. M. (2012). Probabilistic topic models. *Communications of the ACM*, 55(4), 77-84.

Chan, C. K. K. (2011). Bridging research and practice: Implementing and sustaining knowledge building in Hong Kong classrooms. *International Journal of Computer-Supported Collaborative Learning, 6*, 147-186.

Chen, M.-H., Zhang, J. & Lee, J. (2013). Making collective progress visible for sustained knowledge building. In N. Rummel, M., Kapur, M. Nathan, & S. Puntambekar (Eds.), *To See the World and a Grain of Sand: Learning across Levels of Space, Time, and Scale: CSCL 2013 Conference Proceedings Volume 1* (pp.81-88). International Society of the Learning Sciences.

Csikszentmihalyi, M. (1999). Implications of a systems perspective for the study of creativity. In R. J. Sternberg (Ed.), *Handbook of creativity* (pp. 313-335). Cambridge, UK: Cambridge University Press.

Hewitt, J. (2001). Beyond threaded discourse. *International Journal of Educational Telecommunications, 7*(3), 207-221.

Kirschner, P. A., & Erkens, G. (2013). Toward a framework for CSCL research. *Educational Psychologist, 48*(1), 1-8.

Laferriere, T., Law, N., & Montané, M. (2012). An international knowledge building network for sustainable curriculum and pedagogical innovation. *International Education Studies, 5*, 148-160.
Rosé, C., Wang, Y.-C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., & Fischer, F. (2008). Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in CSCL. *International Journal of Computer-Supported Collaborative Learning, 3*(3), 237-271.

Sawyer, R. K. (2007). *Group genius: The creative power of collaboration*. New York: Basic Books.

Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith (Ed.), *Liberal education in a knowledge society* (pp. 67-98). Chicago, IL: Open Court.

Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In R. K. Sawyer (Ed.), *Cambridge handbook of the learning sciences* (pp. 97-115). New York: Cambridge University Press.

Stahl, G. (2013). Learning across levels. *International Journal of Computer-Supported Collaborative Learning, 8*(1), 1-12.

Stahl, G., & Hesse, F. (2009). Classical dialogs in CSCL. *International Journal of Computer-Supported Learning, 4*, 233-237.

Star, S. L., & Griesemer, J. R. (1989). Institutional ecology, “translations” and boundary objects: Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907–39. *Social Studies of Science, 19*, 387–420.

Suthers, D., Vatrapu, R., Medina, R., Joseph, S., & Dwyer, N. (2008). Beyond threaded discussion: Representational guidance in asynchronous collaborative learning environments. *Computers and Education, 50*, 1103-1127.

Tao, D., & Zhang, J. (2018). Forming shared inquiry structures to support knowledge building in a Grade 5 community. *Instructional Science, 46*(4), 563-592.

Wise, A. F., & Schwarz, B. B. (2017). Visions of CSCL: Eight provocations for the future of the field. *International Journal of Computer-Supported Collaborative Learning, 12*(4), 423–467.

Zhang, J. (2009). Comments on Greenhow, Robelia, and Hughes: Toward a creative social Web for learners and teachers. *Educational Researcher, 38*, 274-279.

Zhang, J., & Messina, R. (2010). Collaborative productivity as self-sustaining processes in a Grade 4 knowledge building community. In K. Gomez, J. Radinsky, & L. Lyons (Eds.), *Proceedings of the 9th International Conference of the Learning Sciences* (pp. 49-56). Chicago, IL: International Society of the Learning Sciences.

Zhang, J., & Yuan, G. (2018). Cross-Classroom Interaction for Knowledge Building: A Design Experiment in Four Grade 5 Science Classrooms. Paper presented at the 13th International Conference of the Learning Sciences (ICLS). London, UK: International Society of the Learning Sciences.

Zhang, J., Bogouslavsky, M., & Yuan, G. (2017). Cross-community interaction for knowledge building in two Grade 5/6 classrooms. In Smith, B. K., Borge, M., Mercier, E., and Lim, K. Y. (Eds.), *Making a Difference: Prioritizing Equity and Access in CSCL, 12th International Conference on Computer Supported Collaborative Learning (CSCL) 2017 (Vol. 1)*. Philadelphia, PA: International Society of the Learning Sciences.

Zhang, J., Chen, M.-C., Li, H., Zhao, Y., Chen, J., Rajbhandari, B. L., ... & Naqvi, S. (August, 2012). Making collective progress visible: The design and application of Idea Thread Mapper for sustained knowledge building. Knowledge Building Summer Institute, Toronto, ON, Canada.

Zhang, J., Hong, H.-Y., Scardamalia, M., Teo, C., & Morley, E. (2011). Sustaining knowledge building as a principle-based innovation at an elementary school. *Journal of the Learning Sciences, 20*, 262–307.

Zhang, J., Lee, J., & Chen, J. (2014). Deepening inquiry about human body systems through computer-supported metadiscourse. Paper presented at the Annual Meeting of American Educational Research Association, Philadelphia, PA.

Zhang, J., Scardamalia, M., Lamon, M., Messina, R., & Reeve, R. (2007). Socio-cognitive dynamics of knowledge building in the work of nine- and ten-year-olds. *Educational Technology Research and Development, 55*, 117–145.

Zhang, J., Tao, D., Chen, M.-H., Sun, Y., Judson, D., & Naqvi, S. (2018). Co-organizing the collective journey of inquiry with Idea Thread Mapper. *Journal of the Learning Sciences, 27*, 390-430.

## Acknowledgements

This research was sponsored by the National Science Foundation (#1122573, 1441479). We express our thanks to Drs. Feng Chen, Carolyn Rosé and Marlene Scardamalia for their intellectual input; to our collaborating teachers and their students for their creative classroom work enabling this research; and to the members of our project team who have contributed to the software development and classroom research.
# Location, Location, Location:
The Effects of *Place* in Place-Based Simulations

Amanda L. Siebert-Evenstone, University of Wisconsin-Madison, alevenstone@wisc.edu
David Williamson Shaffer, University of Wisconsin-Madison, Aalborg University, Copenhagen,
david.shaffer@wisc.edu

**Abstract:** Educators who develop and use place-based curricula argue that engaging students in authentic problem-based learning situated in students' own local place helps them understand the relevance of their academic work, which improves learning. However, while place-based curricula are localized, they are also authentic, problem-based experiences, which have been shown to be effective in their own right. Without controlling for the impact of authenticity, it is not clear whether localization itself improves student learning. In this study, we used a place-based simulation to engage students with authentic problem-based learning about a specific place: a city that is local for some students and not for others. We assessed differences in cognitive, affective, and behavioral outcomes between these two groups. Our results suggest that learning outcomes are better when students engage in a place-based simulation about their own locality, indicating that place does play a critical role in the success of place-based education.

## Introduction

According to theories of *situated learning* (Brown, Collins, & Duguid, 1989), all learning occurs within a specific context. However, learning scientists have also argued that we should also consider how learning is situated, not just that it is situated (Greeno & Middle School Mathematics Through Applications Project Group, 1998). One application of situated learning is *place-based education* in which students learn by engaging in authentic problem-based activities about issues in their local community. When creating place-based curriculum, educators identify content and create localized activities using particular place attributes including the geography, history, social issues, environmental issues, and other specifics of that place. One reason educators may use this method is that research has shown that when students engage in place-based curriculum they show higher increases in knowledge, interest, and pro-environmental behaviors when compared to students in a traditional curriculum. However, despite the centrality of place in environmental education, no studies have tested the effect of place itself, potentially because no existing method can separate the effect of place from the other beneficial pedagogical components associated with this practice.

One way to conduct a controlled study is through the use of simulation. In what follows, we use a *place-based simulation* to reproduce authentic problem-based activities about a real place. We evaluate the effect of place by comparing results between overlapping and non-overlapping physical and virtual locations to identify the effects of localizing curricula. Our results show that localized experiences improve student learning, and such a study could only be conducted using a place-based simulation that enables isolation of effects due to localization.

## Theory

### Place-based education

There are several different versions of place-based education—see, e.g., Gruenewald’s (2008) *critical pedagogy of place* and Smith and Sobel’s (2010) *place- and community-based education*. However different these approaches are in their specifics, theories of place-based learning generally share two key characteristics: (1) students engage in authentic problem-solving activities; and (2) the problems are situated in the students’ own local place—that is, the place where the students themselves live (Gruenewald, 2003; Semken, 2005; Smith & Sobel, 2010; Woodhouse & Knapp, 2000). Semken (2005), for example, describes a place-based curriculum in which students in Arizona use topographic maps to guide the collection of rock, fossil, and soil samples from near their school in order to understand how tectonic forces shaped the geology and resources of their community.

Place-based educators argue that curricula that situate learning in the context of students’ own communities help students understand the real-world relevance of their academic work. Unlike more traditional academic approaches which focus on abstract principles, isolated facts, and otherwise decontextualized knowledge, place-based theorists argue that pedagogies that focus on studying issues where students actually live lets them work on concrete, real-world problems in a context they know and care about (Semken et al., 2009).
While in principle place-based education can be developed and applied in any domain, it is most often applied in the context of environmental education. Proponents of place-based environmental education argue that situating learning in students' own local place increases civic and community engagement, interest in environmental issues, and science learning (Powers, 2004; Sobel, 2004). For example, Lieberman and Hoody (1998) conducted a 3-year study of 40 schools across the country, comparing schools using the *Environment as an Integrating Context* curriculum with traditional decontextualized programs. They found that instructional activities based on the local environment around a school increased school achievement across subjects, and increased student engagement and enthusiasm. While these studies did not control for confounding variables, they nevertheless suggest that place-based education can improve teachers' pedagogy and increase student achievement, engagement, and motivation. Based on such studies, Gruenewald (2003) and others (e.g., Smith & Sobel, 2010) claim that place-based learning is the most effective form of environmental education.

## Mechanisms of learning in place-based education

Theorists of place-based learning propose two mechanisms through which place-based learning occurs. One is that place-based curricula offer students multiple options for *authentic experiences* that mirror components of the real-world. Barab and colleagues (2009) suggest that when students work in a setting where they can make consequential decisions, they are more engaged in the learning activities because they believe their actions and decisions matter. Place-based curricula can offer students such experiences a real-world context and a real-world problem to solve, each of which can provide personal and community value. In other words, this argument suggests place-based curricula are effective because they provide consequential and realistic problem-solving activities (Smith and Sobel, 2010). Research in Computer Supported Collaborative Learning more generally has shown authentic experiences (Järvelä, Häkkinen, Arvaja, & Leinonen, 2004) and problem-based activities are beneficial for learning (Hmelo-Silver, 2004). Arastoopour and colleagues (2014) have demonstrated that authentic problem-based activities lead to greater gains in confidence and commitment than students engaging in traditional instruction, and that this method was particularly effective for women.

A second key mechanism of place-based learning is that students are familiar with the context in which their actions are taking place. The first step in many place-based theories calls for *localization* of the curriculum, which refocuses learning to be about the particular place attributes of nearby or regional locations (Woodhouse & Knapp, 2000). By choosing places that students know about, localization provides the opportunity for students to connect to past experiences and envision future experiences in the setting (Smith & Sobel, 2010). Therefore, place-based education provides both a particular kind of pedagogy—authentic, problem-based learning experiences—and a particular kind of setting—the students’ own local context. In other words, place-based education is a *pedagogy of place* that combines authenticity and localization.

## Defining local place

While the concept of localization is a key component of place-based education, theorists have struggled to define what constitutes students' own local place and to demonstrate that working in their own local place contributes significantly to students' learning.

Place-based educators emphasize the importance of a local place and often specify features to incorporate into the curriculum from that local place, but rarely identify what constitutes a local place. For example, Semken (2005) defines places as “spatial localities given meaning by human experiences in them” while Gruenewald (2003) and Casey (1997) argue that a local place is one that people live in and know directly. In each of these definitions, nearness is a core component, yet the bounds of this physical distance are never defined. In their seminal book on place-based education, Smith and Sobel (2010) summarize five of the most popular definitions of place-based education, including their own, and outline seven other antecedents to this pedagogy yet never define or specify what constitutes a local place or community. In response to Gruenewald and others, Stevenson (2008) questions the concepts of local and place, asking what educators and theorists actually mean when they advocate for learning about a local place.

There is a body of research, primarily in sociology and urban planning, on *attachment* to a place: that is, the extent to which people feel that they belong in or to a place, and their perception that the place is familiar and "their own" (Devine-Wright, 2013). More than 75% of place attachment studies focus on the neighborhood scale (Lewicka, 2011), which Devine-Wright (2013) and others (Feitelson, 1991) argue limits understanding of the different scales of place and that place attachments may occur at multiple scales. To address this issue, Laczko (2005) collected data from 24 countries and found that after their high affinity to their country, United States citizens felt closest to their state community rather than their neighborhood or city.

## Testing a pedagogy of place
Although place-based education may sound promising and beneficial, a fundamental assumption of this pedagogy has not been tested: the effect of the *place* itself. As described above, extant research comparing place-based education with traditional curricula has not controlled for the simultaneous effects of *authentic experiences* and *localization*, both of which are hypothesized as key components of a pedagogy of place. When compared with traditional classrooms, do students in place-based curricula do better because of localization, or because activities are authentic, or merely that they leave the classroom? Or is there some synergy between these elements of a pedagogy of place that make it more effective than either localization or authentic experiences alone?

In other words, while place-based curricula are localized, they are also authentic, problem-based learning experiences, which have been shown to be effective in their own right. Without controlling for the impact of authenticity it is not clear whether localization improves student outcomes.

One way to disambiguate the impact of localization from the other features of place-based learning would be to compare the effects of local and non-local experiences of the same place-based curriculum. Such a study would require a curriculum that could present students with authentic, problem-based issues situated in an *existing* place, but in one that is not the student's own local place. So, for example, students would have to use Semken's (2005) place-based geology curriculum (described above) based on the specific rock formations near the Colorado Plateau even though they did not live in Arizona. However, the curriculum has students use local topographic maps to identify places to collect rock, fossil, and soil samples to learn about local rocks and local ground and surface water resources. To adapt this place-based curriculum to another part of the country would require finding a different local land formation for collecting samples that had similar petrologic and hydrologic properties – or rewriting the curriculum to focus on different geologic issues. In other words, changing the location of the curriculum potentially requires a substantial reworking of the original curriculum.

An alternative way to address this issue would be to simulate the two main components of place-based education: *authentic problem-based learning* and *specific local contexts*.

Simulations are technological environments that are designed to reproduce events, places, experiments, and processes from the real world (Dawley & Dede, 2014). In such environments, designers can replicate problem-solving contexts, real-world activities, professional tools, common social interactions, and realistic work products. For example, Chesler and colleagues (2015) created *Nephrotex* where students can role-play as engineering interns at a biotechnology company designing filtration membranes for kidney dialysis. Simulations such as *Nephrotex* offer students the ability to engage in *authentic problem-based learning* about core disciplinary ideas in realistic settings.

## Current study

In this study, we explore the impact of a *place-based simulation* (PBS): a simulation that engages students with authentic problem-based learning *about a real location*. One example of a PBS is the virtual internship, *Land Science*, in which students assume the role of interns at a fictitious urban planning firm. Students engage in realistic professional work in a meaningful real-world context (Chesler et al., 2015) by performing the kinds of tasks that urban planners do in their training: they receive materials that urban planners use, such as research reports, ecological impact surveys, and communications from concerned citizens, which provide information about revenue, water pollution, housing, and other issues. Students engage in these authentic problem-based activities to develop and justify land-use plans that meet the needs of competing stakeholders. Additionally, *Land Science* simulates these activities in the specific context of an urban planner working in Lowell, MA, using content, history, stakeholders, environmental indicators, and maps that are place-specific to that city. Through participation in *Land Science*, students learn about complex eco-social systems (Bagley & Shaffer, 2011) and learn to think like urban planners in the context of a real city (Beckett & Shaffer, 2005). In this sense, *Land Science* is a curriculum that is *based on a real place* and replicates core elements of place-based pedagogy.

PBSs like *Land Science* have several advantages. Typically, in order to adapt a place-based curricula to a new location, educators would need to adapt existing materials to ensure that the curriculum is both specific to a place and uses authentic practices. Because PBSs are based on a real place and use authentic activities, students can interact with the same curriculum from anywhere. As a result, the experience could be local for some students and not for others depending on the geographic location of the student. In other words, we can use *Land Science* to hold authenticity constant and vary the localization of the curriculum. We can, therefore, test differences in student performance between students who might consider Lowell, MA their own *local* place against students in other areas across the country who are engaging in a *non-local* experience.

Of course, in order to test the effect of place, meaningful measures for environmental education must be used to identify differences. Commonly, environmental educators are interested in cognitive, affective, and behavioral changes. For example, the North American Association of Environmental Education (2013) identified the goal of environmental education as creating environmentally literate students, where literacy is contingent
upon changing skills, values, and behaviors instead of solely increasing content knowledge. Therefore, we assessed differences between local and non-local experience using common cognitive, affective, and behavioral outcomes. To address cognitive changes, we assessed students' ability to identify an example of a scientific model—a model that resembles, represents, and/or summarizes the functionality of an object or phenomenon by making a particular feature of the world easier to understand, define, quantify, visualize, or simulate (Lehrer & Schauble, 2006)—because modeling is a critical practice of science in general and of environmental science in particular (Bagley & Shaffer, 2011). Because the virtual internship addresses learning about urban planning, we assessed interest in cities and the environment as an affective outcome. Finally, we assessed changes in future behavior by adapting a civic engagement measure about knowledge and ability to engage in community problems (Flanagan, Syvertsen, & Stout, 2007) that was adapted for middle and high school students and included school as a possible community.

In this study, our primary research questions examine the relationship between online place and location of play across three different outcomes. In this study, we ask:

1. Do students who engaged in a local PBS have *higher ability to identify a science model* after the simulation than students who engaged in a non-local PBS?

2. Do students who engaged in a local PBS have *higher changes in interest* than students who engaged in a non-local PBS?

3. Do students who engaged in a local PBS have *higher changes in civic engagement* than students who engaged in a non-local PBS?

# Methods

## Land Science virtual internship

In the virtual internship *Land Science*, students explore the environmental and socio-economic impacts of land-use decisions. Students role-play as urban planning interns at Regional Design Associates, an urban planning firm developing a land-use plan for the city of Lowell, Massachusetts. Students work individually and in teams to develop a rezoning plan for Lowell that addresses the demands of various community groups who advocate for environmental and socio-economic issues such as wildlife protection, job creation, housing, and controlling air and water pollution. Students use iPlan, a geographic information system model, to evaluate the impacts of land use. Each zoning plan cannot address all of the stakeholder's concerns, so students must make and justify decisions about which demands to meet and how to meet them. The virtual internship takes ten to fifteen hours to complete.

## Local and non-local simulations

Following Laczko (2005) as discussed above, we categorized students who engaged in the *Land Science* PBS about the state they live in and know about as students who had a *local* experience of a place-based curriculum. We categorized students who engaged in the *Land Science* PBS about a state where they do not live as students who had a *non-local* experience of a place-based curriculum. Because simulations can be played from many sites about endless simulated locations, the *Land Science* PBS provided a platform to test the differences between local and non-local experiences of a place-based curriculum.

## Participants

We collected pre- and posttest responses from 94 middle and high school students who participated in one of 10 different implementations of *Land Science* (6 local and 4 non-local). All students participated in *Land Science* as part of an informal science learning experience run by a teacher in a place-based science education center. We examined data from 68 students who engaged in a local PBS and 26 students who engaged in a non-local PBS. To be included in analyses, students must have completed the internship and answered all three sets of questions.

## Measures

Students completed an online survey before (pretest) and after (posttest) participating in *Land Science*. The survey had three components: (1) *Scientific Modeling*, (2) *Interest*, and (3) *Civic Engagement*.

### Cognitive measure: Scientific modeling score

The survey asked students to write an example of a scientific model. Answers were scored with 1 point if they provided an accurate example of a scientific model and 0 points otherwise. For example, some students incorrectly identified elements of the scientific method, such as "in a scientific model, you need to do basic research and to have a hypothesis." While correct responses would include accurate examples of a model such as a "globe" or
“double helix” model of DNA. Two raters achieved acceptable interrater reliability on a subset of the data (n = 126, Cohen’s κ = 0.91, Shaffer’s ρ(0.65) < 0.01), and a single rater coded the rest of the data (please see Shaffer (2017) for more details).

### Affective measure: Interest score

The survey asked students to respond to an open-ended question about their interest in cities and the environment. Table 1 lists the five codes that identify varying types of interest. Two raters coded a subset of 120 responses. We compared their rate of agreement and all five scores had acceptable agreements (Cohen’s κ < 0.65, ρ(0.65) < 0.01) resulting in a weighted Kappa of 0.72. A single rater scored the rest of the data.

<table><caption>Table 1: Categories to code student interest in the cities and/or the environment</caption><thead><tr><td>Category</td><td>Points</td><td>Example</td></tr></thead><tbody><tr><td>No Interest</td><td>0</td><td>“I have no interest in cities or the environment”</td></tr><tr><td>Interest in <i>either</i> a specific city or cities or the a specific environmental issue</td><td>25</td><td>“I’m interested to learn about how cities work”</td></tr><tr><td>Interest in <i>both</i> a specific city or cities and the a specific environmental issue or issues</td><td>50</td><td>“I want to learn about plans that help out cities and the environment.”</td></tr><tr><td>Interest in the <i>relationship</i> between cities and environment generally</td><td>75</td><td>“I’m interested in learning how we affect the environment and what we can do to reduce to effect our cities have on the environment.”</td></tr><tr><td>Interest in a <i>specific relationship or relationships</i> between cities and the environment</td><td>100</td><td>“I’m interested in making room for the entire population of the city, without overcrowding, while keeping the environment intact.”</td></tr></tbody></table>

### Behavioral measure: Civic engagement score

The survey asked students to self-report their knowledge and ability to engage in school and community problems by answering 11 four-point Likert Scale questions adapted from Flanagan, Syversten, and Stout (2007), questions such as: “I know ways of addressing community problems” and “I would be able to find and examine research related to the issue.” We summed the student scores for each of the 11 questions and rescaled the result to create a *Civic Engagement Score* from 0 to 100 for each student at two time points (pretest and posttest), where a score of zero indicated strong disagreement on all questions and 100 indicated strong agreement on all questions.

### Analyses

We constructed a series of nested multiple regression analyses to predict the change in outcome for students’ *Interest* and *Civic Engagement Scores*. Because *Scientific Modeling* was a dichotomous variable we constructed a series of nested logistic regressions to predict this outcome. In each analysis, we tested the following predictors: location, pretest Civic Engagement Score, pretest Interest Score, and pretest Scientific Modeling Score. Pretest measures were included in each set to control for differences in students’ academic ability levels across the 10 implementations included in the study. We computed the Bayesian Information Criterion (BIC) to determine which of the models had the best balance between model fit and model simplicity. For each set of models, we chose the model with the lowest BIC and reported the associated tests for that model.

## Results

### Change in scientific modeling score

The optimal nested logistic regression for Scientific Modeling included location and scientific modeling pretest as predictors which had a significant regression equation, $X^2(2) = 21.25$, $p < 0.001$ (see Table 2). Students who were able to identify a scientific modeling example at the beginning of the game had a 3.09 times higher relative chance of identifying a model after the internship than students who were unable to identify models at the beginning of the internship. For students in Massachusetts, the relative chance of identifying a model is approximately 2.34 times higher than students who played *Land Science* in a different state. There was no effect of civic engagement pretest and interest pretest.

<table><caption>Table 2: Logistic regression results predicting change in Scientific Modeling Score</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Predictors</th><th colspan="2">Model Comparison</th></tr><tr><th>Intercept</th><th>Modeling</th><th>Location:</th><th>Civic Egmt</th><th>Interest</th><th>Pseudo BIC</th></tr></thead></table>
<table><thead><tr><th></th><th></th><th>Pretest</th><th>MA</th><th>Pretest</th><th>Pretest</th><th>R<sup>2</sup></th><th></th></tr></thead><tbody><tr><td>Null</td><td>0.22<br>(0.14) 1.25</td><td></td><td></td><td></td><td></td><td>--</td><td>295.35</td></tr><tr><td>I</td><td>-0.24<br>(0.19) 0.79</td><td>1.07***<br>(0.29) 2.90</td><td></td><td></td><td></td><td>0.05</td><td>286.72</td></tr><tr><td>IIA</td><td>-0.01<br>(0.17) 0.99</td><td></td><td>0.75*<br>(0.31) 2.12</td><td></td><td></td><td>0.02</td><td>294.55</td></tr><tr><td>IIB</td><td>-0.36<br>(0.60) 0.70</td><td></td><td></td><td>0.01<br>(0.01) 1.01</td><td></td><td>0.00</td><td>299.72</td></tr><tr><td>IIC</td><td>0.29<br>(0.29) 1.33</td><td></td><td></td><td></td><td>-0.00<br>(0.00) 1.00</td><td>0.00</td><td>300.63</td></tr><tr><td>III</td><td>-0.53*<br>(0.22) 0.59</td><td>1.13***<br>(0.30) 3.09</td><td>0.85**<br>(0.32) 2.34</td><td></td><td></td><td>0.07</td><td>284.80</td></tr><tr><td>IV</td><td>-1.71*<br>(0.76) 0.18</td><td>1.21***<br>(0.31) 3.34</td><td>0.90**<br>(0.33) 2.46</td><td>0.02<br>(0.01) 1.02</td><td>0.00<br>(0.01) 1.00</td><td>0.08</td><td>292.55</td></tr></tbody></table>

Standard errors are reported in parentheses. *p < 0.05, **p < 0.01, and ***p < 0.001. n = 94.

## Change in interest score

The optimal nested regression model for Interest Score based on BIC included location and interest as predictors which had a significant regression equation (F(2,91) = 13.26, *p* < 0.001), with an adjusted R² of 0.21 (see Table 3). Participant change in interest scores decreased by 0.51 percent for each 1-point increase in the pretest score indicating that students with lower interest at the start of the game had higher increases in interest. On average, students in Massachusetts increased their interest score by 21.41 percentage points more than the scores for students in a different state increased. Because we scaled these tests to have a maximum value of 100, this difference in scores between local and non-local students, controlling for their pretest score, would be the equivalent of increasing two letter grades. There was no effect of civic engagement pretest or example pretest.

<table><caption>Table 3: Regression results predicting change in Interest Score</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Intercept</th><th colspan="4">Predictors</th><th colspan="2">Model Comparison</th></tr><tr><th>Interest Pretest</th><th>Location: MA</th><th>Civic Egmt Pretest</th><th>Modeling Pretest</th><th>Adjusted R<sup>2</sup></th><th>BIC</th></tr></thead><tbody><tr><td>I</td><td>28.13***<br>(5.52)</td><td>-0.49***<br>(0.13)</td><td></td><td></td><td></td><td>0.12</td><td>914.34</td></tr><tr><td>IIA</td><td>-3.85<br>(5.93)</td><td></td><td>20.39**<br>(6.97)</td><td></td><td></td><td>0.08</td><td>919.29</td></tr><tr><td>IIB</td><td>19.18<br>(13.07)</td><td></td><td></td><td>-0.12<br>(0.19)</td><td></td><td>-0.01</td><td>927.21</td></tr><tr><td>IIC</td><td>9.88*<br>(4.82)</td><td></td><td></td><td></td><td>-0.02<br>(0.07)</td><td>-0.01</td><td>927.56</td></tr><tr><td>III</td><td>13.21<br>(6.91)</td><td>-0.51***<br>(0.13)</td><td>21.41**<br>(6.46)</td><td></td><td></td><td>0.21</td><td>908.15</td></tr><tr><td>IV</td><td>6.79<br>(13.73)</td><td>-0.53***<br>(0.13)</td><td>22.58**<br>(6.70)</td><td>0.12<br>(0.18)</td><td>0.03<br>(0.06)</td><td>0.20</td><td>916.66</td></tr></tbody></table>

Standard errors are reported in parentheses. *p < 0.05, **p < 0.01, and ***p < 0.001. n = 94.

## Change in civic engagement score

The optimal nested regression model for Civic Engagement Score based on BIC included location and civic engagement pretest score as predictors which had a significant regression equation (F(2,91) = 15.83, *p* < 0.001), with an adjusted R² of 0.24 (see Table 4). Participant civic engagement scores decreased by 0.34 percent for each 1-point increase in the pretest score indicating that students with lower pretest scores showed greater change in civic engagement score. On average, students in Massachusetts had a change in civic engagement score that was 9.74 percent higher than the change in score for students playing *Land Science* in a different state. Because we scaled these tests to have a maximum value of 100, the difference in post-test scores between local and non-local students, controlling for their pretest score, is the equivalent change of one full letter grade. There was no effect of example pretest or interest pretest.

<table><caption>Table 4: Regression results predicting change in Civic Engagement Score</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Intercept</th><th colspan="4">Predictors</th><th colspan="2">Model Comparison</th></tr><tr><th>Civic Egmt Pretest</th><th>Location: MA</th><th>Modeling Pretest</th><th>Interest Pretest</th><th>Adjusted R<sup>2</sup></th><th>BIC</th></tr></thead><tbody><tr><td>I</td><td>32.86***</td><td>-0.39***</td><td></td><td></td><td></td><td>0.17</td><td>776.96</td></tr></tbody></table>
<table><tbody><tr><td>IIA</td><td>(5.88)</td><td>(0.09)</td><td>12.10***</td><td>0.11</td><td>783.61</td></tr><tr><td></td><td>-1.63</td><td></td><td>(3.39)</td><td></td><td></td></tr><tr><td>IIB</td><td>10.22***</td><td></td><td>0.06</td><td>0.02</td><td>792.62</td></tr><tr><td></td><td>(2.35)</td><td></td><td>(0.03)</td><td></td><td></td></tr><tr><td>IIC</td><td>7.04*</td><td></td><td>0.00</td><td>-0.01</td><td>795.82</td></tr><tr><td></td><td>(2.94)</td><td></td><td>(0.07)</td><td></td><td></td></tr><tr><td>III</td><td>22.74***</td><td>-0.34***</td><td>9.74**</td><td>0.24</td><td>772.30</td></tr><tr><td></td><td>(6.53)</td><td>(0.08)</td><td>(3.18)</td><td></td><td></td></tr><tr><td>IV</td><td>17.92*</td><td>-0.32***</td><td>10.36**</td><td>0.25</td><td>777.93</td></tr><tr><td></td><td>(7.00)</td><td>(0.09)</td><td>(3.20)</td><td></td><td></td></tr><tr><td></td><td></td><td></td><td>(2.87)</td><td></td><td></td></tr><tr><td></td><td></td><td></td><td>(0.06)</td><td></td><td></td></tr></tbody></table>

Standard errors are reported in parentheses. *p < 0.05, **p < 0.01, and ***p < 0.001. n = 94.

Discussion

Our results suggest that curricula situated in a specific, real place are more effective when that place is local to the learners. To predict change in each outcome, we included the location, relevant pretest, as well as the other pretests to see which variables might explain differences in the score. For each of the three outcome variables, however, the nested regressions showed that only the relevant pretest and location were significant predictors of the change in outcome. Models that included both the relevant pretest and location best explained the change in outcome and that this model was more parsimonious than models that included all four predictors. These results suggest that even though all students engaged in the same authentic activities and problem-based learning about a real place, we found that students showed different changes in outcomes based on their location. When students engaged in a simulated place-based curriculum about their state, they had higher changes in civic engagement, interest, and ability to identify a scientific model than students in other states.

These results have implications both for the study of place-based pedagogies and for the use of simulations in education more generally.

While the importance of local place has been assumed in place-based education research, this is the first work we are aware of that provides empirical evidence to support this claim. It may be that previous studies have made this assumption because of the physical constraints that limit the places students can experience in an engaging and immersive way. Our results suggest that when students experience a simulation of authentic and problem-based learning about a specific place, it makes a difference if it is their own local place. Thus, as Gruenewald (2003) and other place-based theorists argue, environmental education may be more effective when it engages students in the places where they live.

More broadly, though, researchers who study educational simulations argue that one benefit of these tools is that educators can create curricula that can be accessed from locations anywhere in the country or world (Dawley & Dede, 2014). This means that students are not limited in their access to sound pedagogies based on the resources of their own communities. However, this study suggests that local context has a powerful influence on how students perceive and experience an online curriculum. There are surely important ideas that are best understood through the lens of distant or imagined worlds, but it may be that students are more swayed by Thoreau's (1881) argument that "it is not worth the while to go round the world to count the cats in Zanzibar." As a result, teachers may need the ability to customize one-size-fits-all simulations—or perhaps more accurately, one place fits all simulations—to use them more effectively in environmental science classrooms.

This study has several limitations. First, it describes a small number of students in a small number of locations where students could not be randomly assigned to a locality. However, this reestablishes the problems in measuring the effect of place. While we were not able to use randomization and the sample size was not large enough to account for differences in the implementations such as length or background work by teachers, we would be quick to point out that we have no reason to believe that those are systematically related to the place in which they were implemented. As a result, although we are currently working to expand this analysis to a larger sample size with more locations, the implications of this study should be generalized with caution. Second, this study tested only one definition of local place and does not explore all of the complex issues involved with conceptions of space or place. Future analyses could examine the impact of different spatial scales for locality to better understand the relationship between localization and learning. Finally, this study focused on three limited outcomes measured in a pretest-posttest design. As a result, it provides no information on the mechanisms by which locality impacted learning in this environmental education simulation, and future studies are needed to better understand the effect of locality on science content and practice learning within place-based simulations.

Despite these limitations, these results suggest that students have higher changes in civic engagement, interest, and ability to identify a scientific model when they engage in a place-based simulation about their own local place—and that advocates for place-based education are correct in arguing that situating curriculum in students local place has a positive impact on student outcomes. Place does matter.
## References

Arastoopour, G., Chesler, N., & Shaffer, D. (2014). Epistemic persistence: A simulation-based approach to increasing participation of women in engineering. *J.of Women and Minorities in Sci. and Eng, 20*(3), 211–234.

Bagley, E. A. S., & Shaffer, D. W. (2011). Promoting civic thinking through epistemic game play. In R. Ferdig (Ed.), Discoveries in gaming and computer-mediated simulations: New interdisciplinary applications (pp. 111–127). Hershey, PA: IGI Global.

Barab, S. A., Gresalfi, M., & Arici, A. (2009). Why educators should care about games. *Ed. Leadership, 67*(1), 76–80.

Beckett, K. L., & Shaffer, D. W. (2005). Augmented By Reality: the Pedagogical Praxis of Urban Planning As a Pathway To Ecological Thinking. *Journal of Educational Computing Research, 33*(1), 31–52.

Brown, J., Collins, A., & Duguid, P. (1989). Situated Learning and the Culture of Learning. *Ed. Research, 18*(1), 32–42.

Casey, E. S. (1997). *The Fate of Place: A Philosophical History*. Berkeley, CA: University of California Press.

Chesler, N. C., Ruis, A. R., Collier, W., Swiecki, Z., Arastoopour, G., & Shaffer, D. W. (2015). A novel paradigm for engineering education: Virtual internships with individualized mentoring and assessment of engineering thinking. *Journal of Biomechanical Engineering, 137*(2), 1–8.

Dawley, L., & Dede, C. (2014). Situated learning in virtual worlds and immersive simulations. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), *Handbook of research on educational communications and technology* (4th ed., pp. 723–734). New York.

Devine-Wright, P. (2013). Think global, act local? The relevance of place attachments and place identities in a climate changed world. *Global Environmental Change, 23*(1), 61–69.

Feitelson, E. (1991). Sharing the globe: The role of attachment to place. *Global Env. Change, 1*, 396–406.

Flanagan, C. A., Syvertsen, A. K., & Stout, M. D. (2007). Civic Measurement Models: Tapping Adolescents' Civic Engagement. *The Center for Information & Research on Civic Learning & Engagement, May (May)*, 40.

Greeno, J. G., & Middle School Mathematics Through Applications Project Group. (1998). The Situativity of Knowing, Learning, and Research. *American Psychologist January 1998, 53*(1), 5–26.

Gruenewald, D. A. (2003). Foundations of place: A multidisciplinary framework for place-conscious education. *American Educational Research Journal, 40*(3), 619–654.

Gruenewald, D. A. (2008). The best of both worlds: a critical pedagogy of place. *Env. Ed. Research, 14*(3), 308–324.

Hmelo-Silver, C.E. (2004). Problem-Based Learning: What and How Do Students Learn? *Educational Psychology Review, 16*(3), 235–266.

Järvela, S., Häkkinen, P., Arvaja, M., & Leinonen, P. (2004). Instructional Support in CSCL. In J. W. Strijbos, P. A. Kirschner, & R. L. Martens (Eds.), *What We Know about CSCL* (pp. 115–119). Kluwer Academic Publishers.

Laczko, L. S. (2005). National and Local Attachments in a Changing World System: Evidence from an International Survey. *International Review of Sociology, 15*(3), 517–528.

Lehrer, R., & Schauble, L. (2006). Cultivating model-based reasoning in science education. In K. Sawyer (Ed.), *Cambridge Handbook of the Learning Sciences* (pp. 371–388). Cambridge, MA: Cambridge University Press.

Lewicka, M. (2011). Place attachment: How far have we come in the last 40 years? *J. of Env. Psych., 31*(3), 207–230.

Lieberman, G. A., & Hoody, L. L. (1998). *Closing the achievement gap: Using the environment as an integrating context for learning. Results of a nationwide study*. Poway, CA: Science Wizards.

NAAEE. (2013). North American Association for Environmental Education, http://www.naaee.net/.

Powers, A. L. (2004). An evaluation of four place-based education programs. *Journal of Env. Education, 35*(4), 17–32.

Semken, S. (2005). Sense of Place and Place-Based Introductory Geoscience Teaching for American Indian and Alaska Native Undergraduates. *Journal of Geoscience Education, 53*, 149–157.

Semken, S., Freeman, C., Watts, N., Neakrase, J., Dial, R., & Baker, D. (2009). Factors that influence sense of place as a learning outcome and assessment measure of place-based geoscience teaching. *Elect. J. of Sci. Ed., 13*(2), 136–159.

Shaffer, D. W. (2017). *Quantitative Ethnography*. Madison, WI: Cathcart Press.

Smith, G. A., & Sobel, D. (2010). *Place- and community-based education in schools*. New York, NY: Routledge.

Sobel, D. (2004). Place-based Education: Connecting Classroom and Community. *The Orion Society*.

Stevenson, R. (2008). A critical pedagogy of place and the critical place(s) of pedagogy. *Env. Ed. Res., 14*(3), 353–360.

Thoreau, H. D. (1981). *Walden and other writings*. Bantam Classics.

Woodhouse, J. L., & Knapp, C. E. (2000). Place-Based Curriculum and Instruction: Outdoor and Environmental Education Approaches. *ERIC Clearinghouse on Rural Education and Small Schools, 1–8*.

## Acknowledgments

This work was funded in part by the National Science Foundation (DRL-1661036, DRL-1713110), the Wisconsin Alumni Research Foundation, and the Office of the Vice Chancellor for Research and Graduate Education at the University of Wisconsin-Madison. The opinions, findings, and conclusions do not reflect the views of the funding agencies, cooperating institutions, or other individuals.
# The Role of Funds of Knowledge in Online Search and Brokering

Wendy Roldan, Paola Vanegas, Laura Pina, Carmen Gonzalez, and Jason Yip
wr4@uw.edu, pvdt@uw.edu, lpina@uw.edu, cmgonzal@uw.edu, jyip@uw.edu
University of Washington

**Abstract:** Lower-socioeconomic status, immigrant parents who are English language learners often work collaboratively with their children to search the internet. Family members rely on each other's language and digital literacy skills in this collaborative information problem solving process known as online search and brokering (OSB). While previous work has identified the ecological factors that impact OSB, research has not yet distilled the specific learning processes behind such collaborations. From a case study analysis of three families, this work explores the funds of knowledge that children and parents rely on as they engage in collaborative learning experiences through OSB. We demonstrate how in-home computer supported collaborative processes are often informal, collaborative, social, and highly relevant to solving real-life information challenges. Our work shows how parents and children draw on their funds of knowledge when they search collaboratively, with and for their family members, to build their collective knowledge of technology and problem-solving.

## Introduction

Approximately 8 million U.S. children have at least one immigrant parent who is an English language learner (Zong, Batalova, & Hallock, 2018). Lower-socioeconomic status immigrant parents often rely on their children's language and digital literacy skills to address family needs (Eksner & Orellana, 2012). In these families, adults and children work with each other to search the internet, in a collaborative information problem-solving process we call *online search and brokering* (OSB) (Pina et al., 2018; Yip, Gonzalez, & Katz, 2016). For these families, children's responsibilities extend beyond traditional chores to holding the role as the family's primary problem-solver for critical family needs through online information searches (Pina et al., 2018). While previous work has identified online search and brokering practices (Yip et al., 2016) and the ecological factors that influence online search and brokering (Pina et al., 2018), research has not yet distilled the specific learning processes behind such family collaborations. While online information problem-solving models (Brand-Gruwel, Wopereis, & Walraven, 2009) contribute to our understanding of digital learning and information processing, they do not fully explain the sociocultural, collaborative learning processes that occur as families engage in online search and brokering.

Our research focuses on Latin American families, the fastest growing population in the U.S. (Pew Research Center, 2018). Latino children are projected to make up about a third of the K-12 enrollment by 2023 and with more than half of them living in immigrant families (Foxen & Mather, 2016), many of them are searching for critical information online for their English language learning family members. Education researchers engaged in sociocultural historical approaches with children from non-dominant backgrounds explain that learning is an ongoing process that is not divided into separate characteristics of individuals and contexts (Gutiérrez & Rogoff, 2003). Therefore, we believe it is important to understand the learning that occurs in intergenerational, bilingual, information problem-solving family collaborations. Previous research with families whose households have been traditionally viewed as low-resourced, applies a funds of knowledge framework to understand and nurture the strategic knowledge and skills that are often overlooked (Moll, Amanti, Neff, & Gonzalez, 1992). Using a funds of knowledge framework (Moll et al., 1992) this work explores the learning processes that occur in Latino families, when family members collaboratively search for information online. We examine a case study of three focal Latin American, lower-socioeconomic status, English language learner families as they engaged in collaborative, computer supported, online search and brokering. Our research questions are: **RQ1**. What funds of knowledge do bilingual children and their English language learner parents rely on when attempting to solve information problems using the Internet?; and **RQ2**. What are the collaborative learning processes around online information searches in English language learner families as they work together?

## Background

This work builds on literature in individual and collaborative search processes, the phenomenon of search and brokering, joint media engagement, and sociocultural historical theory on learning as a cultural, ongoing activity. We draw on research that investigates how individuals search for information online to make sense of the learning that happens within individual search processes. However, in applying individual problem-solving models for online search we find gaps with respect to understanding how learning occurs in intergenerational and bilingual collaborative searches. Thus, we turn to the literature on language brokering that focuses on intergenerational and
bilingual problem-solving between youth and their parents to help us fill the gaps. To strengthen our literature review on online search and brokering with families, we find that joint media engagement is a particularly useful framework to examine the learning that is happening as families use media together and in the home.

## Individual and collaborative search processes

A rich body of work explores how individuals search for information online. Models like *Information Problem-Solving for the Internet* offer insights into the set of skills individuals need to search for information online. These skills include: defining, searching, scanning, processing, and organizing information (Brand-Gruwel et al., 2009). While we know much about the learning that happens within individual search processes, less is known about the process of how people collaborate together to solve information problems and the learning processes behind it (Stahl, Koschmann, & Suthers, 2006). Research on online searching as a collaborative process between individuals (Morris, 2013), shows the opportunity to reinforce and learn search skills from exposure to others' strategies (Foss et al., 2012). But much of this work has focused on skilled peer adults searching remotely together using online tools (Morris, 2013). Further, research with families engaged in informal collaborative relationships with technology suggests that participation in these activities can nurture knowledge of managing information and promoting self-direction of one's learning (Jenkins, 2006). This strand of research provides insights into the relationship between collaboration and learning but focuses on collaborative engagement in technologically mediated activities that are not information problems. From our literature review, we conclude that research on intergenerational, bilingual, collaborative search processes and learning is limited thus we turn to the communications literature that explores language brokering with youth in the home.

## Online search and brokering (OSB)

Recent literature explores how parents and children work collaboratively to address family information needs using digital resources, in a process known as online search and brokering (Pina et al., 2018). Research on immigrant youth shows how language brokering is part of everyday life as children and parents work together to address family needs (Orellana, Dorner, & Pulido, 2003). Bilingual youth work as translators and interpreters for their immigrant parents, which opens the families' access to resources and information on education, health, and finances. As families collaborate to address family needs in everyday language-brokering events, different levels of skills and expertise are leveraged, and knowledge becomes co-constructed (Eksner & Orellana, 2012). As the digital divide narrows, scholars have also explored how children become *technology brokers* to introduce and teach their parents new technologies (Nelissen & Van den Bulck, 2018). There are significant differences in how children help their parents with technology across families from different socio-economic status (Brown, Bakken, Nguyen, & Von Bank, 2007). In high-socioeconomic status families, children drive the adoption of mobile applications, technology for entertainment, and educational purposes. On the other hand, in families from a lower-socioeconomic status, children help connect their adult family members to critical information needs (e.g., finances, health, well-being) (Nelissen & Van den Bulck, 2018). Collaborative family engagement with technology also differs significantly across families. To further contextualize our work as parents and children use technology together, we draw on literature from joint media engagement.

## Joint media engagement (JME)

The nuanced interactions that occur as parents and children engage in online search and brokering is a form of joint media engagement that has not been previously studied (Pina et al., 2018). The phenomenon of joint media engagement helps us understand the experiences of people using media together as they view, play, contribute, search, and create with both traditional and digital media (Sobel et al., 2017; Takeuchi & Stevens, 2011). Previous research with families defines joint media engagement as the process of learning between children and parents as they co-create meaningful connections among interests, experiences, and representations using all forms of media and technologies that are present in children's lives (Takeuchi & Stevens, 2011). However, much of the research on joint media engagement emphasizes learning together through gaming, entertainment, and education (Gee, Takeuchi, & Wartella, 2018). Our work builds on prior research with families and technology by identifying the funds of knowledge that children and their parents draw on to problem-solve and co-create meaningful connections that extend beyond play and family fun to solve critical family needs.

## Theoretical framework – funds of knowledge

We apply a funds of knowledge framework to examine the collaborative, intergenerational, bilingual, information problem-solving learning processes that occur during online search and brokering with an asset-based perspective. A funds of knowledge framework allows us to understand the historically accumulated and culturally developed bodies of knowledge that are essential for household and individual well-being (Moll et al., 1992). We posit that a funds of knowledge approach allows us to capture the tacit knowledge parents and children draw on when
engaging in online search and brokering, knowledge that is not necessarily taught but critical to family well-being. This framework allows us to push the conversation away from deficit models of learning for non-dominant students towards honoring the learning that occurs in the household and everyday activities of Latino youth and adults as they search and broker for information online collaboratively. Previous research with families whose households have been traditionally viewed as low-resourced uses funds of knowledge to refer to the strategic knowledge and skills that exist within these families (Moll et al., 1992). A person's funds of knowledge can be described as their accumulated life experiences, the skills and knowledge they use to navigate everyday life, and their cultural-historical academic and personal background knowledge. Our work builds on prior funds of knowledge research in learning and education, by identifying the knowledge and skills found in households that members use to solve information problems as a computer supported collaborative learning process.

As parents and children engage in everyday online search and brokering, we explore the learning processes that occur through a sociocultural lens by applying a funds of knowledge framework. Sociocultural scholars propose that the educational movement toward equity will occur, “as we create learning environments that connect in deep ways to the life experiences of all students” (Nasir, Rosebery, Warren, & Lee, 2006). A sociocultural lens provides new insights on issues of race, culture, and learning (Nasir et al., 2006). Learning, as a cultural process of engagement in repertoires of practices, is a process in which individual development is understood in cultural and historical contexts (Gutiérrez & Rogoff, 2003). Building on this body of work, we recognize understanding learning requires a focus on how individuals engage and participate in particular every day activities and how they draw on artifacts, tools, and others to solve problems (Nasir & Hand, 2006).

## Methods

For this study, we adhered to the standards and practices of a case study examination (Merriam & Tisdell, 2015). We focused on three Latin American lower-socioeconomic status, English language learner families in the Pacific Northwest and their collaborative learning experiences through engagement in online search and brokering.

This exploratory, qualitative study with three focal families is a part of a larger qualitative study. Between July 2016 and June 2017, we visited 23 families in an urban area of the Pacific Northwest, U.S., within a 32-kilometer radius of our research institution. Our participants included parents, grandparents, and children aged 10-17. Most of the adults that were a part of our study were born in Mexico, did not have a college-degree, worked in service industries, and represented a lower-socioeconomic population. At the time of this study, a tense political climate existed for Latino families therefore we relied on community center networks, local cultural events, and a paid community liaison as a part of our recruitment strategies. We conducted two separate in-home family visits per family. In-home visits allowed participants to feel comfortable in their usual search and brokering environment and allowed researchers to observe search practices within the home's digital infrastructure. In our first in-home visit (V1), we conducted separate adult and youth retrospective interviews to contextualize where and how they search. In V1 we gathered family members' perspective on their practices, strategies, and challenges when searching for information online. The interview protocol was adapted from previous work on in-home media studies (Katz & Gonzalez, 2016). Each interview (45-60 mins) was audio recorded and transcribed. Parent interviews were in Spanish and youth interviews in English. For our second in-home visit (V2), we focused on directed internet search tasks between adult-youth dyads. Visit 2 was audio recorded and screen interactions were video recorded. In V2, participants engaged in a set of imposed search tasks prompted by researchers and historical tasks of prior online searches. We take a deep dive into three specific families from this larger data set (Table 1).

<table><caption>Table 1: Demographics of the families</caption><thead><tr><td>ID</td><td>Relationship (Age)</td><td>Occupation</td><td>Adult birthplace</td><td>Grade completed by adult</td></tr></thead><tbody><tr><td>2</td><td>Mother (39), Daughter (10)</td><td>Hospitality</td><td>Mexico</td><td>Primary, 4<sup>th</sup> grade</td></tr><tr><td>17</td><td>Grandfather (62), Grandson (12)</td><td>Food Industry</td><td>Mexico</td><td>Secondary, 7<sup>th</sup> grade</td></tr><tr><td>22</td><td>Mother (42), Daughter (15)</td><td>Housekeeping</td><td>Mexico</td><td>Secondary, 10<sup>th</sup> grade</td></tr></tbody></table>

Our selection criteria for these three families are the following. (1) Each family offered insights into different household structures that influence the online search and brokering process. One family was a mother-daughter dyad within a single-parent home, another was a grandparent-grandson dyad within a seven-person home, and the third was a mother-daughter dyad within a four-person family home. (2) In a comparative analysis each family allowed us to see how despite the differences across family structures, we observed themes in the funds of knowledge used and the learning processes occurring during online search and brokering. (3) Each family offers insights into different types of devices used in the home to search for information.

For this paper, we analyzed data from V1 and V2 with three families to explore our research questions. To analyze the data, two researchers used open coding with constant comparative analysis (Merriam & Tisdell, 2015). This iterative open-coding was done for both parent and child interviews and for an analysis of V2 video
data. Two researchers open-coded the data independently for themes such as technology and language brokering, information problem-solving, relationships, perceptions of learning, and strategies and challenges to searching. We coded and compared the themes to develop further categories for analysis and then systematically compared and contrasted the themes between researchers. Following open-coding, we used axial coding to make connections between funds of knowledge categories and subcategories. We performed a constant sorting and comparative analysis until theoretical saturation was reached and no new codes were generated. Finally, we looked at these themes and interpreted them using a funds of knowledge analytical lens. Data on search tasks from our second visit allowed us to triangulate interview data from our first visit to provide a rich analysis of our data.

## Findings

For each family, we provide a case description for context and describe a search task in-depth. We then discuss each case using a funds of knowledge analytical lens (Moll et al., 1992) to help us document the learning and knowledge in practice that is a part of online search and brokering as a daily household routine (Gutiérrez, Morales, & Martinez, 2009). We highlight the funds of knowledge that each family member brings into the collaborative information problem search process from their home and community lives (Moll et al., 1992). Each family engages in online search and brokering differently with devices, set-ups, and strategies. All quotes are translated from Spanish. We use [ ] to indicate typing on digital devices. All names used in the cases are pseudonyms, with Name^A for adult, and Name^C for child.

### Family 2. Norma^A and Mia^C

*Case Description.* Mia^C is Norma^A's 10-year-old youngest daughter who helps her mom with searches on a more regular basis compared to her older sisters. Mia^C is a developing yet savvy researcher, often struggling with spelling search words but consistently recognizing the need for descriptive terms to narrow her search. At the time of the study, Norma^A indicated she did not feel confident with technology but was aware of technology resources through her local community networks. In V1, we learned Norma^A and Mia^C engaged in collaborative online search and brokering when Norma^A discovered someone had taken money out of her bank account fraudulently. The woman from the bank taught Mia^C how to access Norma^A's online banking information for her. During V2, we saw this financial search information problem revisited. For one of the tasks, Norma^A wanted to find the closest bank near her home. Mia^C clicked the search bar (Figure 1, left), and began to type using the tablet screen keyboard and her two index fingers, [*the cosest bank*]. Mia^C paused as the search engine suggestions came up for *the closest bank of america* and *the closest bank of america to me*. Mia^C continued to type into her search [*the cosest bank of america*] and pressed the search button to generate results. Mia^C scanned the results and clicked on the Google visual map (Figure 2, right). Mia^C hovered her finger over the options and explained to her mom what the results meant. Mia^C noted Option A is the closest because it appeared first.

<figure><img src="image_85.png" /><figcaption>Figure 1. Norma and Mia searching for the closest bank using Google Maps.</figcaption></figure>

## Case analysis. Knowledge of linguistic translational practices

In this case, we notice how Norma^A and Mia^C use their funds of knowledge about each other's assets to complete their financial information search task of finding the bank nearest to them. Mia^C built on her cultural repertoires of linguistic practice as she (a) translated her mom's inquiry about finding the closest bank from Spanish to English; (b) searched using key descriptive terms in English that would narrow her search for banks nearest her; (c) scanned the information presented in English; (d) relied on visuals and maps to find answers; and (e) translated the results back to Norma^A. After the search was completed, Mia^C explained she did not understand maps, but Norma^A used her knowledge of spatial geography and relied on her lived experiences of navigating this area to understand where the bank was. Norma^A and Mia^C helped each other process, translate, and increase their collective understanding of finding and reading a map. We see knowledge of linguistic translational practices used in their intergenerational, bilingual, online search and brokering practices. In this example, we see Mia^C engaging in a range of practices that challenge deficit notions of students' repertoires developed across non-school settings (Gutiérrez et al., 2009). As Mia^C collaborative searches with and for her mom, she uses linguistic knowledge, problem-solving and search knowledge, and translation knowledge. Each of these funds of knowledge Mia^C uses to search could become future resources for her learning across settings and practices, inside and outside the classroom (Gutiérrez et al., 2009).
## Family 17. Carmelo<sup>A</sup> and Mateo<sup>C</sup>

*Case Description.* Mateo<sup>C</sup> is Carmelo<sup>A</sup>'s 12-year-old youngest grandson who helps Carmelo<sup>A</sup> with most of his searches. Carmelo<sup>A</sup> owns a restaurant and is a lifelong learner who desires to learn more about technology. Mateo<sup>C</sup> is a visual and audio searcher who feels comfortable with technology and uses google voice to search for information. Similar to Mia<sup>C</sup>, Mateo<sup>C</sup> struggles with spelling and instead prefers to say rather than type a word. In V1, Carmelo<sup>A</sup> told us that he searches the internet to buy things for his restaurant and preferred images to make sense of information online. In V2, we observed Carmelo<sup>A</sup> and Mateo<sup>C</sup> search together for new industrial stoves knobs for Carmelo<sup>A</sup>'s restaurant. Carmelo<sup>A</sup> wanted to buy goods on Amazon for his restaurant. Mateo<sup>C</sup> clicked on the Amazon app on a smartphone and clicked on the search bar (Figure 2, left). He moved his fingers from the search bar to the keyboard. Mateo<sup>C</sup> showed Carmelo<sup>A</sup> how to go to the search bar. Carmelo took the smartphone and typed [*buttons for*] in Spanish. Mateo<sup>C</sup> interrupted and asked if this was for clothing, but Carmelo<sup>A</sup> explained this was for stoves. When Carmelo finished typing, the Amazon app returned images of what Carmelo was looking for. As the discussion continued, there was a disconnect between what Mateo<sup>C</sup> thought they were looking for. Mateo<sup>C</sup> typed word [*butten*] into the Amazon search bar (Figure 2, center). Mateo<sup>C</sup> said the search was not giving him what he wanted. Carmelo<sup>A</sup> took back the phone, left the Amazon app, clicked the Google app, and typed [*how do you spell buttons in English*]. Carmelo<sup>A</sup> said, “There it is! This is how I do it. But it takes me a long time.” At this point Carmelo<sup>A</sup> asked Mateo<sup>C</sup> to teach him how to search using voice to be more agile. Mateo<sup>C</sup> clicked on the voice icon on Google search (Figure 2, right) and told him to just talk. Carmelo<sup>A</sup> took the phone up to close to his mouth and said, “I want to look for buttons for the stove,” but the search did not work. Mateo<sup>C</sup> explained, “You have to push this button and then go. You push it and go.” Carmelo<sup>A</sup> used the voice assistant again, “I want to look for buttons for the stove.” Carmelo<sup>A</sup> looked and smiled at the search result images of the knobs he was looking for, having learned a new technology skill (Figure 2, right).

<figure><img src="image_86.png" /><figcaption>**Figure 2.** Carmelo and Mateo searching for stove knobs using the Amazon app and Google voice assistant.</figcaption></figure>

### Case analysis. Knowledge of problem-solving strategies

In this case, we believe Carmelo<sup>A</sup> and Mateo<sup>C</sup> rely on each other's funds of knowledge to explore different search strategies when they face the challenge of not knowing the translation between buttons and knobs in Spanish. Carmelo relies on his lived experiences running the restaurant to explain to Mateo<sup>C</sup> what he was looking for. Mateo<sup>C</sup> sets the search up for Carmelo<sup>A</sup> by (a) previously configuring the Amazon app settings to Spanish; (b) showing him where to enter his search terms; and (c) clarifying what it is they are looking for to make sure the language translation is correct. When they get stuck, Carmelo<sup>A</sup> uses his previous knowledge related to persistence to show Mateo<sup>C</sup> how he uses the Internet to problem-solve translational challenges. Further, this example shows how each other's funds of knowledge are taken up as they explore alternative search strategies moving from the Amazon app, to Google search, to Google search voice assistant. Carmelo<sup>A</sup> observes Mateo<sup>C</sup> using the Google voice assistant and learns how to do it himself through Mateo<sup>C</sup>'s scaffolds. The two help each other conceptualize the problem, explore different internet resources to push away from challenges, and learn new language and technology skills. We see knowledge of alternative strategy exploration being taken up by Carmelo<sup>A</sup> and Mateo<sup>C</sup> as they expand their knowledge of technology to uncover the shared meaning of knobs buttons in their multisensory (auditory and visual), intergenerational, bilingual, and online search and brokering practices.

## Family 22. Romelia<sup>A</sup> and Amy<sup>C</sup>

*Case Description.* Amy<sup>C</sup> is Romelia<sup>A</sup>'s 15-year-old daughter. Romelia<sup>A</sup> is a stay-at-home mom and relies on Amy<sup>C</sup> to search for information related to the needs of others in the home. Romelia<sup>A</sup> watches videos on YouTube to learn how to do things like braid hair or find new recipes. Amy<sup>C</sup> is a searcher who uses multiple strategies in action as she searches for her mom using her knowledge about viruses, advertisements, and web browsing. When the two engage in collaborative online search and brokering together they often sit side-by-side, co-viewing on the family desktop computer. During our first visit, we learned Romelia<sup>A</sup> wished she knew how to help her husband search for construction jobs. In V2, we observed how Amy<sup>C</sup> and Romelia<sup>A</sup> search for construction jobs for the father. Amy<sup>C</sup> asked, “What jobs do you want to help find for papa?” (Figure 3, left). Romelia<sup>A</sup> responded, “Like companies, like companies that have jobs.” Amy<sup>C</sup> typed, [*construction companies hiring near me*] and clicked on the second search result, Monster.com. Romelia<sup>A</sup> noted, “Make sure the search results are local.” Amy<sup>C</sup> narrowed the search by selecting the U.S. state and typed [*construction*] in the Monster.com search box. The search results come up and Amy<sup>C</sup> translated the job descriptions from English to Spanish for her mom. As Amy<sup>C</sup>
simultaneously interpreted, translated, and scanned the results, she also explained why she was not clicking on job postings that require a technical or engineering degree. However, AmyC noted she was stuck. RomeliaA suggested adding the word *siding* and Amy typed in [*constructing citing* (sic)], which did not result in what they needed. AmyC asked, “*citing*, right?” They stepped away from the computer and look for other resources to refine their search terms. RomeliaA pointed to the father’s contracting certification that showed how to spell *siding* the correct way (Figure 3, right). Amy typed, [*construction companies hiring siding installers in (U.S. state)*]. She translated the information to her mom. RomeliaA asked, “How much do those jobs pay?” Amy noted it was about $20 an hour. AmyC asked, “Do you want jobs for him or for his company?” RomeliaA stated, “I want to look for jobs for a contractor, not jobs that hire by the hour, I want to look for big jobs because he has his company.”

<figure><img src="image_87.png" /><figcaption>Figure 3. Romelia and Amy searching for construction siding jobs using online and physical resources.</figcaption></figure>

## Case analysis. Knowledge of search resources

RomeliaA and AmyC discuss, contribute, and move from online resources to resources in the home to move past challenges. When AmyC began the search, RomeliaA used her geospatial knowledge and lived experiences of finding a job to ask Amy to narrow the search for local jobs. AmyC used her knowledge to (a) click on a job search engine; (b) scan the job postings; (c) translate linguistically the search results; and (d) explain her interpretation of the job descriptions. When the two got stuck, RomeliaA offered new search terms (siding) while AmyC continued to scan and interpret website search results. Using knowledge of alternative resources available to them beyond the Internet they collectively (a) step away from the computer; (b) find the father's contracting license; and (c) correct their spelling of their search terms. They build on each other's funds of knowledge to conceptualize the problem, find alternative resources to spell the search term, and explain what type of job they are looking for based on the father's qualifications. We argue knowledge of search resources moves fluidly between the two as they translate across language, across physical and online, and across technical linguistic practices.

## Discussion

*Funds of Knowledge in Online Search and Brokering.* In this study, we found that each individual family member contributes their own knowledge to collectively build household knowledge resources. Each child is a part of a larger social structure within the home where they rely and share their knowledge with siblings, parents, and grandparents. Across all families, we observed how MiaC, MateoC, and AmyC's role as the primary researcher in their home carried "the trace of prior social relations," building on their prior actions while co-creating new knowledge with adults (Bakhtin, 1981; Vossoughi, Jackson, Chen, & Roldan, Under Review; Vygotsky, 1978). The adults usually have knowledge for the search conceptualization such as geospatial understanding, spelling clarifications, or knowledge of family resources. The children usually have critical technology proficiency and linguistic translational knowledge to search, scan, process, gather, and translate online information. During collaborative searches, each family member continually builds on the knowledge of each other's assets. This knowledge informs the ways in which children and adults set up future learning opportunities for each other within online search and brokering practices.

Further, we highlight how family members developed resilience, as they faced structural and systemic challenges to searching for information (language, access to social resources, schooling). Resilience was reinforced through funds of knowledge and resource sharing as family members relied on each other's knowledge, strategies, and skills when met with a challenge during their search process (Berkes & Ross, 2013). Across all families, we observed resilience at the level of the individual and the household as interrelated, with family members actively developing their shared resilience through capacity building and collaborative learning while engaged in online search and brokering (Berkes & Ross, 2013). The families persisted and relied on their funds of knowledge (translational practices, problem-solving, search resources) collaboratively to solve their information problems when they got stuck or tried to problem-solve through the unknown. For the majority of the adults in this study, the United States is not their home country. With this information, we can see the funds of knowledge related to resilience, problem-solving through new situations, and strategies to move beyond challenging situations while engaging in online search and brokering in our data. Building on sociocultural theories of learning, we offer this analysis as a way to help us construct classrooms that draw on the funds of knowledge that students bring from their home and everyday experiences to promote holistic and equitable development and learning (Gutiérrez & Rogoff, 2003).

*Collaborative Learning Processes in Online Search and Brokering.* In contrast to highly individualized
instructional systems (e.g., formal school), we demonstrate learning processes in-the-home are often informal,
collaborative, highly social, and highly relevant to solving real-life information access challenges. Through our
thematic analysis of the funds of knowledge that adults and children draw on while they engage in collaborative
online search and brokering with their families, we offer a nuanced understanding of computer supported
collaborative learning processes happening within the home. We contribute to studies of joint media engagement
(Takeuchi & Stevens, 2011) by identifying the funds of knowledge Latino families draw on to problem-solve and
co-create solutions that extend beyond play to solve critical family needs. Joint media engagement is not just
about learning together through gaming, entertainment, or educational technologies. Instead, we highlight the
ways that family members work together as a group to solve information problems using online connected
technologies. Work within CSCL notes that group cognition forms as a result of collaborative knowledge building
in which meaning is created across the utterances of different people (Stahl, 2006). In our cases, all of the adults
and youth act in a joint activity to problem-solve for their family needs. The shared construction of meaning
occurs as both adult and child engage through intersubjectivity of language, technological interactions,
information problem-solving, and family funds of knowledge. For many of these families, the group remains at
the synchronous level, in which adult and child work together around an interactive device (e.g., desktop, tablet,
laptop, smartphone) to collaborative solve their family information needs. We believe our research in this area
contributes to CSCL as there is a need to identify how technology and online information problem-solving for
family needs supports how people learn, and uncovering what challenges lie ahead.

*Implications for Educators.* As online search and brokering plays a major part of family practices, our work highlights the funds of knowledge that are evident as parents and children search together, which educators can utilize in the classroom to design instructional materials that are relevant to students’ family responsibilities. Overall, while our findings focus on a specific group of Latin American families with English language learner parents from a lower-socioeconomic status, family information search and collaboration happen across all contexts and ages. For example, other immigrant English language learner families might search for health information for elderly family members together. Individuals must use similar processes of finding information together, making sense of that information, and translating that information by closely drawing on their funds of knowledge at each step of the information problem-solving model. By examining a) the funds of knowledge families use in online search and brokering and b) the computer supported collaborative learning processes in online information search, we are better able to provide instructional design implications to help schools, libraries, and community centers attend to culture in understanding students’ learning. Our work opens up new opportunities to bridge home and school through the computer supported collaborative learning process that are embedded in children’s every day practices. Our findings can help educators consider family roles when designing curricula. Future work could examine the unique challenges of English language learning students and how their search practices might fluctuate across formal and informal contexts.

Conclusion

We argue that an intergenerational online search and brokering process is different compared to collaborative
online information problem-solving that happens between classroom peers or between co-workers. Our research
shows how both parents and children draw on their funds of knowledge when they search collaboratively, with
and for their family members, to build their collective knowledge of technology and problem-solving. Through
the metaphor of a jigsaw puzzle, we can begin to understand how collaborative problem-solving takes first, the
identification of knowledge and skills and secondly, the learning processes behind figuring out how to can arrange,
turn, and shift the puzzle pieces, in this case the funds of knowledge, to solve the information problem at hand.
The CSCL community can benefit from an understanding of how online search and brokering is a form of
collaborative learning around technologies, given it is the daily reality for millions of bilingual children.

References

Bakhtin, M. M. (1981). The dialogic imagination: Four essays by mm bakhtin (m. holquist, ed.; c. emerson & m. holquist, trans.). Austin: University of Texas Press.

Berkes, F., & Ross, H. (2013). Community Resilience: Toward an Integrated Approach. Society & Natural Resources, 26(1), 5–20. https://doi.org/10.1080/08941920.2012.736605

Brand-Gruwel, S., Wopereis, I., & Walraven, A. (2009). A descriptive model of information problem solving while using internet. *Computers & Education*, *53*(4), 1207–1217.

Brown, B. B., Bakken, J. P., Nguyen, J., & Von Bank, H. G. (2007). Sharing information about peer relations: Parent and adolescent opinions and behaviors in Hmong and African American families. *New Directions for Child and Adolescent Development*, *2007*(116), 67–82.

Eksner, H. J., & Orellana, M. F. (2012). Shifting in the zone: Latina/o child language brokers and the co-
construction of knowledge. Ethos, 40(2), 196–220.

Foss, E., Druin, A., Brewer, R., Lo, P., Sanchez, L., Golub, E., & Hutchinson, H. (2012). Children’s search roles at home: Implications for designers, researchers, educators, and parents. Journal of the American Society for Information Science and Technology, 63(3), 558–573.

Foxen, P., & Mather, M. (2016). Toward A More Equitable Future: The Trends and Challenges Facing America's Latino Children. Retrieved from http://publications.nclr.org/handle/123456789/1627

Gee, E., Takeuchi, L., & Wartella, E. (Eds.). (2018). Children and families in the digital age: learning together in a media saturated culture. New York: Routledge, Taylor & Francis Group.

Gutiérrez, K. D., Morales, P. Z., & Martinez, D. C. (2009). Re-mediating literacy: Culture, difference, and learning for students from nondominant communities. Review of Research in Education, 33, 212–245.

Gutiérrez, K. D., & Rogoff, B. (2003). Cultural ways of learning: Individual traits or repertoires of practice. Educational Researcher, 32(5), 19–25.

Jenkins, H. (2006). Confronting the challenges of participatory culture: Media education for the 21st century. An occasional paper on digital media and learning. John D. and Catherine T. MacArthur Foundation.

Katz, V. S., & Gonzalez, C. (2016). Toward meaningful connectivity: Using multilevel communication research to reframe digital inequality. Journal of Communication, 66(2), 236–249.

Merriam, S. B., & Tisdell, E. J. (2015). Qualitative research: A guide to design and implementation. John Wiley & Sons.

Moll, L. C., Amanti, C., Neff, D., & Gonzalez, N. (1992). Funds of knowledge for teaching: Using a qualitative approach to connect homes and classrooms. Theory into Practice, 31(2), 132–141.

Morris, M. R. (2013). Collaborative search revisited. Proceedings of the conference on computer supported cooperative work. ACM, 2013.

Nasir, N. S., & Hand, V. M. (2006). Exploring Sociocultural Perspectives on Race, Culture, and Learning. Review of Educational Research, 76(4), 449–475.

Nasir, N. S., Rosebery, A. S., Warren, B., & Lee, C. D. (2006). Learning as a Cultural Process: Achieving Equity Through Diversity. In The Cambridge handbook of: The learning sciences. (pp. 489–504). New York, NY, US: Cambridge University Press.

Nelissen, S., & Van den Bulck, J. (2018). When digital natives instruct digital immigrants: Active guidance of parental media use by children and conflict in the family. Information, Communication & Society, 21(3), 375–387.

Orellana, M. F., Dorner, L., & Pulido, L. (2003). Accessing assets: Immigrant youth's work as family translators or para-phrasers. Social Problems, 50(4), 505–524.

Pew Research Center. (2018). Hispanic/Latino Demographics. Retrieved from http://www.pewresearch.org/topics/hispaniclatino-demography/

Pina, L. R., Gonzalez, C., Nieto, C., Roldan, W., Onofre, E., & Yip, J. C. (2018). How Latino Children in the US Engage in Collaborative Online Information Problem Solving with their Families. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), 140.

Sobel, K., Bhattacharya, A., Hiniker, A., Lee, J. H., Kientz, J. A., & Yip, J. C. (2017). It wasn't really about the Pokémon: Parents' Perspectives on a Location-Based Mobile Game (pp. 1483–1496). ACM Press.

Stahl, G. (2006). Group cognition: Computer support for building collaborative knowledge. Cambridge, Mass: MIT Press.

Stahl, G., Koschmann, T. D., & Suthers, D. D. (2006). Computer-supported collaborative learning.

Takeuchi, L., & Stevens, R. (2011). The new coviewing: Designing for learning through joint media engagement. Presented at the New York, NY: The Joan Ganz Cooney Center at Sesame Workshop.

Vossoughi, S., Jackson, A., Chen, S., & Roldan, W. (Under Review). Embodied pathways and ethical trails: Studying learning in and through the choreography of social relations.

Vygotsky, L. (1978). Interaction between learning and development. Readings on the Development of Children, 23(3), 34–41.

Yip, J.C., Gonzalez, C., & Katz, V. (2016). The learning experiences of youth online information brokers. In C. K. Looi, J.L. Polman, U. Cress, & P. Reimann, P. (Eds.). Transforming Learning, Empowering Learners: The International Conference of the Learning Sciences (ICLS) (Vol. 1, pp. 362–369).

Zong, J., Batalova, J., & Hallock, J. (2018). Frequently Requested Statistics on Immigrants and Immigration in the United States. Migration Policy Institute.

## Acknowledgments

We thank the families that allowed us into their homes and the individuals that contributed to this research.