# On a Making-&-Tinkering Approach to Learning Mathematics in Formal Education: Knowledge Gains, Attitudes, and 21st-Century Skills

Stella Timotheou, Cyprus Interaction Lab, Department of Multimedia and Graphic Arts, Cyprus University of Technology, stella@cyprusinteractionlab.com

Andri Ioannou, Cyprus Interaction Lab, Department of Multimedia and Graphic Arts, Cyprus University of Technology, Research Center on Interactive Media, Smart Systems and Emerging Technologies (RISE), andri@cyprusinteractionlab.com

**Abstract:** Making and tinkering, as learning practices, have gained a lot of attention during the last decade, especially in STEAM education. Despite the significant interest in making-&-tinkering activities, most of the research has focused on implementations in non-formal and informal educational settings, with learners of older age (i.e., students in secondary school). The present study sought to investigate young students’ knowledge gains, attitudes towards a making-&-tinkering approach to learning mathematics in formal education and the development of 21st-century skills as they engaged in a collaborative making-&-tinkering project using a variety of arts, crafts, and technological tools such as a physical robot. The making-&-tinkering approach involved making, tinkering, programming, and play in a group project integrated into the formal mathematics curriculum. Findings from the study suggest that young students can greatly benefit from such an approach. The study has demonstrated the applicability and value of such methods for young learners in formal educational contexts, with implications for future research and practice in the field.

## Introduction and theoretical background

There is a lot of recent discourse on learning in STEAM (Science, Technology, Engineering, Arts, and Mathematics) and how it is promoted through making, tinkering, programming, and play activities in various contexts and settings, including CSCL ones. STEAM education has been given a lot of attention during the last decade, and it has become an emphasis in many curriculums around the world (Vossoughi & Bevan, 2014). Increased pressure has been placed on teachers to improve performance in the above disciplines (Vossoughi & Bevan, 2014). This improvement can be achieved through new ways of teaching. Students who have been taught through traditional methods may have limited critical thinking ability as these methods emphasize only on the right answer to the problem through a determined process (Rode et al., 2015). In contrast to traditional methods, newer ones that support making, tinkering and play, set the significance of the process (and not only the end-result) and provide better ways to attract students with diverse thinking styles (Rode et al., 2015). In the present investigation, we adopt a making-&-tinkering approach to learning Mathematics (M), together with Science (S), Technology (T), Arts (A) and Engineering (E), in formal education with young students in elementary school, aged 8 to 9 years old.

Making is a process of creating something (Hsu, Baldwin, & Ching, 2017), and specifically, it is “the act of creating tangible artifacts” (Rode et al., 2015, p. 8). The movement of making, called “the maker movement”, as an active process of building, designing, and innovating with tools and materials to produce shareable artifacts, has gained enormous momentum during the last years (Papavlasopoulou, Giannakos & Jaccheri, 2017). Making is a learner-driven educative practice that supports learning, participation, and understanding (Vossoughi & Bevan, 2014). Tinkering as a part of making (Vossoughi & Bevan, 2014), is a problem-solving technique and learning strategy, which promotes a practice of improvement, and it is associated with experimentation and “trial and error” methods (Krieger, Allen, & Rawn, 2015). As Martinez and Stager (2013) indicate, making and tinkering involve a playful approach to solving problems through direct experience, experimentation, and discovery. Programming and physical computing are very often making and tinkering activities (Hsu et al., 2017), as students can build and rebuild their robot, make the program design, code and debug. Both in programming and in making and tinkering activities, the play is diffused. Play in this work is defined as a dynamic, active and constructive behaviour, and it’s contiguous with the gamification of learning (Deterding, 2011; Ioannou, 2018). The connection between all the above concepts and learning in CSCL, including students’ attitudes, are examined in this paper.

Both making and tinkering in education are not new ideas. They have their theoretical roots in Papert’s constructionism, which is built upon Piaget’s constructivism (Papavlasopoulou et al., 2017). Making is also noted by educators such as Froebel, Montessori and Dewey when they advanced practical, physical and playful learning
(Vossoughi & Bevan, 2014). As Vossoughi and Bevan (2014) indicated, the theories of Vygotsky, Lave and
Wenger are related to making, too, in the notion that making can support learning and development.

The benefits of making in the learning process have been identified for many decades, thus during the last years a growing interest has been shown on making or “maker culture” as a philosophy or phenomenon (Papavlasopoulou et al., 2017), which should be examined more thoroughly through more empirical studies (Hsu et al., 2017). Making can change learners’ role from passive recipients to active ones, taking control over their own knowledge (Papavlasopoulou et al., 2017), which it is a vital characteristic of learner-driven pedagogies that are compounded in curriculums. Learning by doing gives learners the opportunity to engage in problem-solving, self-directed and collaborative work (Hsu et al., 2017) through the development of interests, identity and knowledge, as it may involve the use of physical materials in traditional crafts or hobby techniques (such as woodworking, cooking, etc.) or the use of technology. During making, learners are engaged in tinkering, which also builds on inquiry-based pedagogy and exploits learner-centered learning (Papavlasopoulou et al., 2017). According to Sheridan et al. (2014), tinkering supports learning and promotes 21st-century skills. As programming involves tinkering, educational robotics as learning tool can also benefits teaching and learning in STEAM education. Making, tinkering and programming are closely related to playful and gameful learning, as experimental and hands-on activities are involved. Robots can be used as a game in students’ hands and alongside with gameful design (Deterding, 2011; Ioannou, 2018), the learning outcomes can be beneficial.

Although a few researchers focus on the value of making, tinkering, programming, and play in learning (Krieger et al., 2015; Hsu et al., 2017; Martinez & Stager, 2013), studies which document the child's learning experience in making-based activities are currently lacking. A growing number of efforts in making-enhanced activities is observed in non-formal or informal learning environments, such as workshops, libraries, museums, summer-school and after-school programs rather than in formal education. Yet, most of these efforts are practice-oriented in contrast to only few research-oriented studies (Chu, Angello, Saenz, & Quek, 2017). Moreover, typically such activities are done with students of older age e.g., secondary education, 12-18 years old (Papavlasopoulou et al., 2017).

In the present investigation, we adopted a making-&-tinkering approach to learning Mathematics (M), together with Arts (A) and Engineering (E), in formal education with young learners. The study sought to investigate young learners' knowledge, attitudes towards a tinkering-&-making approach to learning mathematic and their development of 21st-century skills, as they engaged in a group project using a variety of arts, crafts, and technological tools. The study sought to answer the following research questions:

*RQ1: What kinds of learning gains do young students experience during making-&-tinkering activities?*

*RQ2: What are their attitudes towards this approach of learning in formal (math) education?*

*RQ3: How this approach seems to enact the development of 21st-century skills?*

## Methodology

The study adopted a mixed methods research design, relying on both quantitative and qualitative data collection and analysis to answer the research questions under investigation.

## Participants

The sample was composed of 18, 3rd-grade students (aged 8 to 9 years old), 12 girls and 6 boys. The students came from a small public primary school in a rural area in northeastern Mediterranean. The majority of the students came from families of middle socioeconomic status and education. The school was selected by convenience sampling, as the researcher was teaching at the school.

## Procedures

The research work took place in the participants' regular classroom. The desks were set in a way so as students could work in six groups of three. The group formation constituted children from different cognitive, emotional and social behavior levels. Two groups used the classroom's computers and four groups used tablets. On each group's table, there were pencils, rubbers, rulers, variety of reused materials and an Edison robot. The 5-sessions intervention lasted four weeks as summarized in Table 1 (see also pictures in Figures 1, 2).

<table><caption>Table 1: The 5-sessions intervention</caption><thead><tr><td>Sessions</td><td>Activity</td><td>Description</td></tr></thead><tbody><tr><td>1st</td><td>craft</td><td>-the problem is given<br/>-brainstorming about how they could make their craft and materials they could use</td></tr></tbody></table>
<table><tr><td></td><td></td><td>-students in groups create their craft</td></tr><tr><td>2<sup>nd</sup></td><td>math goals</td><td>-small exhibition in the classroom of their “monster robot vacuum cleaners”<br/>-students write a four-digit price for their monster<br/>-they are taught about rounding to the nearest hundred<br/>-they choose six of their monsters’ prices (the number six was given to limit the task) and try to round them to the nearest hundred</td></tr><tr><td>3<sup>rd</sup></td><td>finish the artefacts and program the robot</td><td>-students work in three-members’ group<br/>-they choose which monster is suitable to be set on the robot<br/>-they find a way to put the monster on the robot<br/>-they program their artefacts to move continuously</td></tr><tr><td>4<sup>th</sup></td><td>improve the programming</td><td>-groups look for ways to improve their programming, by making their robot moving continuously without getting out of the borders</td></tr><tr><td>5<sup>th</sup></td><td>prepare posters</td><td>- groups prepare their poster and organize their presentation</td></tr></table>

## Data collection and analysis

Data collection included knowledge pre- and post- tests, questionnaire on students' attitudes, video data and focus groups data.

### Knowledge test

A knowledge test was used pre and post intervention to measure students' acquisition of mathematical knowledge. The test included two exercises on the content of the math lesson. The same test was given a month later (i.e., as delayed-test) to measure the retention of knowledge. The tests were examined for face validity by an experienced teacher-colleague at the school. Another teacher-colleague examined and ensured that the lesson was relevant and in-line with the National Curriculum.

<figure><img src="image_154.png" /><figcaption>Figure 1. Making-&-tinkering activities to set the craft onto the robot.</figcaption></figure>

### Attitudes questionnaire

The questionnaire used is known as the My Class Activities (MCA), an instrument developed by Gentry and Gable (2001) to measure students' attitudes toward their class activities, specifically their level of: (a) interest e.g., "What I do in my class fits my interests", "I have an opportunity to work on things in my class that interest me", (b) challenge e.g., "The activities I do in my class are challenging", "I have to think to solve problems in my class", (c) choice e.g., "I can choose to work in a group", "I can choose materials to work with in class", and (d) enjoyment e.g., "I look forward to my class", "I have fun in my class". The questionnaire has previously reported good psychometric properties (Gentry & Gable, 2001). The questionnaire was translated from its English version
to Greek, following the standard test adaptation guidelines (Chapman & Carter, 1979). Initially the MCA items were translated by the researcher, who is a native speaker of Greek and then the items were translated back to English by an English teacher, and finally they were compared. Two more experts reviewed the translated MCA items, to ensure the content validity of the items. Then it was piloted with four students of the same school age, coming from another school of the same district. The MCA questionnaire was administered at the end of the 5th session of the intervention.

## Video data and analysis

Observational data were collected via video analysis, which was held in two rounds. In the first round, which was based on the Critical Incident Technique (Flanagan, 1954), general but important episodes and themes were noted (e.g. “Making and tinkering was detected during the phase of the craft’s construction”, “Tinkering helped students to improve their craft and their programming, to remember the result, and to feel satisfaction about their achievement”, “Collaboration during the making and tinkering activities helped them to improve their artifact and to find the best programming solution”). These themes helped in the organization of the focus groups’ protocol. The second round of video-analysis was conducted using the Manifest Content Approach (FitzGerald, 2012) on the episodes identified during the first round; the episodes were coded based on the Tinkering Dimensions Framework (Bevan et al., 2015) which was used as per Table 2. Both, the first and second video-analysis rounds were conducted separately by the researcher and another expert and inter-rater reliability (present agreement) was assessed to 87% and 89% respectively. Upon percent agreement computation, the raters discussed and resolved all disagreements.

<table><caption>Table 2: Learning Dimensions Framework developed by the Tinkering Studio (Bevan et al., 2015)</caption><thead><tr><td>Learning Dimension</td><td>Indicators</td></tr></thead><tbody><tr><td>Engagement</td><td><ul><li>Spending time in tinkering activities</li><li>Displaying motivation or investment through effect or behavior</li></ul></td></tr><tr><td>Initiative & Intentionality</td><td><ul><li>Setting one's own goals</li><li>Seeking and responding to feedback</li><li>Persisting to achieve goals in the problem space</li><li>Taking intellectual risks or showing intellectual courage</li></ul></td></tr><tr><td>Social Scaffolding</td><td><ul><li>Requesting or offering help in solving problems</li><li>Inspiring new ideas or approaches</li><li>Physically connecting to others' works</li></ul></td></tr><tr><td>Development of understanding</td><td><ul><li>Expressing a realization through effect or utterances</li><li>Offering explanations for a strategy, tool or outcome</li><li>Applying knowledge</li><li>Striving to understand</li></ul></td></tr></tbody></table>

## Focus groups

Semi-structured focus-group interviews with students (3 focus groups with 6 students each, 40 minutes each) were conducted at the end of the experience. The focus-group interviews were organized based on the framing of the study, the themes derived from the video analysis, and the research questions of the study (e.g., “What did you learn from the experience?”, “How do you feel about this approach to learning mathemat at school?”). The focus-group interview data were video-recorded and transcribed for subsequent data analysis. Data were transcribed and qualitatively analyzed using a thematic analysis approach (Attride-Stirling, 2001).

## Findings

### Knowledge gains

A Wilcoxon Signed Rank Test demonstrated statistically significant gains in students' knowledge from pre- to post- testing (W(18) = 2, z = -2.18, p < .05). Also, a Wilcoxon Signed Rank Test was used to examine knowledge retention (i.e., delayed test data); results showed that knowledge gains were preserved since there was no statistically significant difference from post to delayed testing (W(18) = 1, z = -1.34, p > .05).
Consistent with the quantitative data, during the focus group discussions, almost all the children (n=15 out of 18) mentioned that they could explain how to round a four-digit number to the nearest hundred. Specifically, a girl noted:

> I really liked the session where we gave four-digit prices to our monster-hoovers and organized a small shop. And it was nice when we walked around the classroom -to the shop-desks- to choose 6 prices and try to round them to the nearest hundred... We didn't know how to round the numbers and we were trying [...] Only John was right and when he explained to us, I understood and now I know! [#12, Girl]

## Attitudes towards the tinkering-&-making approach

Descriptive statistics on the attitude subscale mean-scores showed that the students appreciated and enjoyed the tinkering-&-making approach to formal (math) learning, particularly the opportunity to choose the way they could work in the project and to face challenges during the project, which triggered their interest (Table 3).

<table><caption>Table 3: Subscale mean-scores for attitudes</caption><thead><tr><td>Subscale</td><td>Mean</td><td>SD</td></tr></thead><tbody><tr><td>Interest</td><td>3.30</td><td>0.61</td></tr><tr><td>Challenge</td><td>2.88</td><td>0.66</td></tr><tr><td>Choice</td><td>1.90</td><td>0.77</td></tr><tr><td>Enjoyment</td><td>3.50</td><td>0.61</td></tr></tbody></table>

Findings from the video-analysis supported the results from questionnaire data on attitudes, as 27% of the incidents concerned the presence of the dimension of “Engagement” (see Figure 2), based in the framework by (Bevan et al., 2015).

<figure><img src="image_155.png" /><figcaption>Figure 2. Results according to the "Learning Dimension Framework".</figcaption></figure>

The focus-group interviews provided additional evidence of students' attitudes, in fact, demonstrating a positive shift in attitudes over time, e.g.

> At the beginning, we couldn't work [...] by the end, I liked it!
> 
> When you gave us the programming challenge, we thought it was something difficult, but after we tried it was easier[...] I liked the project because we programmed the robots through the computers! [#1, Girl]

> I was enthusiastic! Basically, we felt happiness! [...] All the sadness we felt at the beginning had gone away by the end as we were happy about our achievements! [...] That time I felt I was a scientist who invented something very great. And we learned a thousand of things we didn't know before! And using all those materials you gave us, we managed to do something we never did before[...] Will we make again a lesson with robots? [#4, Girl]
## 21st-century skills

Most of the results derived from the focus-group interview data referred to the 21st-century skills. The data were grouped into three categories -- Learning & Innovation Skills, Information Media & Technology Skills, and Life & Career Skills – per framework by the Partnership for the 21st Century Learning (P21, 2015). Findings concerning the “Learning & Innovation Skills” are organized below in categories.

### Creativity and innovation

The focus-group data revealed that creativity and innovation skills were worked throughout the project. The students used their imagination to construct their craft and design and to prepare their poster. During the project, the students improved their crafts and their programming, invented new ideas, improved or changed their initial ideas and found ways to fasten their craft onto the robot, e.g.

> At the beginning, I had an idea, later I got another idea, and finally, I got another idea...
> We put our imagination to make our monsters and posters!
> I mostly liked the craft part and when I used my imagination!
> [...] and we managed to fasten our monster on the robot without any lego or anything else! We just set it on the robot and it wouldn't fall.
> My team found another way [...] we set the monster onto the robot, but when we realized that there was no space to put our fingers to press the robot's buttons, we used the small wood chips that were set on the monster to press the buttons. So, we turned the monster in a way that the yellow wood chip could press the circular button, the green wood chip could press the square button, and the other wood chip could press the triangle!!! [#13, Girl]

In most groups, the students worked creatively with others and they implemented or suggested innovations, e.g.

> I liked the moment we invented new things.
> That time I felt I was a scientist who invented something very important!
> When you are engaged in such a kind of activities you feel like you are Einstein!
> Instead of the monster-hoover robot, we could have made the caveman robot related to our history [...]
> Yes! Or to use the idea in Gymnastic where the robots could be our bodies!"
> And invent more outdoor activities, not only indoor activities.
> [...] the teacher could use more puzzle challenges leading us to a treasure, for more mystery and fun!
> Oh! We could also write the programming commands on a paper and later when we find the treasure, we could test them on the tablet. [#18, Girl]

### Critical thinking and problem-solving

In students' reporting we could easily identify strategies related to critical thinking and problem solving e.g.

> We learned how to describe step by step the process we followed from the beginning to the end
> [...] we planned with all the steps until we could solve the problem and reached our goal!
> [...] we made questions to find other solutions [...]
> We should be careful, and we should observe what we were doing to avoid making mistakes...
> We knew why we made each step, and we were able to explain it...
> At the beginning, we couldn't find the right programming solution, but later we understood how to do it! I know how to do it!" (talking about rounding to the nearest hundred)
> I realized that we must finish something before we review and make judgments [...] it may not be so difficult [...]
> why the other group found the programming and we did not?
> [...] I saw the other posters full of colours and I felt nervous. [#12, Boy]

### Communication and collaboration

It became apparent in the focus-group interview data that the students communicated clearly with others via a variety of communicative ways, ranging from body movements and discussions to the usage of descriptions, posters and narrative stories. They also learned to collaborate with others especially their groupmates, e.g.
innovation, communication and collaboration. Similar results were found in previous works (Bevan, Gutwill, Petrich & Wilkinson, 2015; Harnett, Tretter, & Philipp, 2015).

Although the study, confirms to a large extent, findings already reported in previous works, the field is still in maturing and therefore replicability helps to enhance the validity of such methods and practices in education. The present study is one of very few with young children of this age (8 to 9 years old), engaging in making-&-tinkering activities in the context of formal mathematics curriculum and therefore, has a unique merit in the research literature with implications for researchers and practitioners in the area. The small (N=18) sample of convenience in the present investigation does not allow for generalizability of findings, yet the rich description of our methods and procedures should enable other researchers and practitioners to transfer these findings into similar contexts and settings. Overall, a better understanding of how making-&-tinkering occurs in formal settings with young learners will enable researchers and practitioners to nourish learning environments which can serve the development of the 21st-century skills.

## Acknowledgments

This work is part of a project that has received funding from the European's Horizon 2020 research and innovation programme under grant agreement N°739578 and the government of the Republic of Cyprus through the Directorate General for European Programmes, Coordination and Development.

## References

Attride-Stirling, J. (2001). Thematic networks: an analytic tool for qualitative research. Qualitative research, 1(3), 385-405.

Bevan, B., Gutwill, J. P., Petrich, M., & Wilkinson, K. (2015). Learning through stem-rich tinkering: Findings from a jointly negotiated research project taken up in practice. Science Education, 99(1), 98-120.

Chapman, D. W., & Carter, J. F. (1979). Translation procedures for the cross-cultural use of measurement instruments. Educational Evaluation and Policy Analysis, 1(3), 71-76.

Chu, S. L., Angello, G., Saenz, M., & Quek, F. (2017). Fun in Making: Understanding the experience of fun and learning through curriculum-based Making in the elementary school classroom. Entertainment Computing, 18, 31-40.

Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011, September). From game design elements to gamefulness: defining gamification. In *Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments* (pp. 9-15). ACM.

Ioannou, A. (2018). A model of gameful design for learning using interactive tabletops: enactment and evaluation in the socio-emotional education classroom. Educational Technology Research and Development, 1-26.

Flanagan, J. C. (1954). The critical incident technique. Psychological bulletin, 51(4), 327.Gentry, M., & Gable, R. K. (2001). From the Student's Perspective—My Class Activities: An Instrument for Use in Research and Evaluation. Journal for the Education of the Gifted, 24(4), 322-343.

Harnett, C. K., Tretter, T. R., & Philipp, S. B. (2014, October). Hackerspaces and engineering education. In Frontiers in Education Conference (FIE), 2014 IEEE (pp. 1-8). IEEE.

Hsu, Y. C., Baldwin, S., & Ching, Y. H. (2017). Learning through making and maker education. TechTrends, 61(6), 589-594.

Krieger, S., Allen, M., & Rawn, C. (2015, February). Are females disinclined to tinker in computer science? In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (pp. 102-107). ACM.

Martinez, S. L., & Stager, G. (2013). Invent to learn: Making, tinkering, and engineering in the classroom (pp. 31-41). Torrance, CA: Constructing modern knowledge press.

Papavlasopoulou, S., Giannakos, M. N., & Jaccheri, L. (2017). Empirical studies on the Maker Movement, a promising approach to learning: A literature review. Entertainment Computing, 18, 57-78.

P21- Partnership for 21st Century Learning. (2015). P21 Partnership for 21st Century Learning. Partnership for 21st Century Learning, 9. Retrieved from http://www.p21.org/documents/P21_Framework_Definitions.pdf

Rode, J. A., Weibert, A., Marshall, A., Aal, K., von Rekowski, T., El Mimouni, H., & Booker, J. (2015, September). From computational thinking to computational making. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 239-250). ACM.

Sheridan, K., Halverson, E. R., Litts, B., Brahms, L., Jacobs-Priebe, L., & Owens, T. (2014). Learning in the making: A comparative case study of three makerspaces. Harvard Educational Review, 84(4), 505-531.

Vossoughi, S., & Bevan, B. (2014). Making and tinkering: A review of the literature. National Research Council Committee on Out of School Time STEM, 1-55.
# Parent and Educator Perspectives on the Benefits of an Online
Space to Promote Offline Program Collaborative Learning

Susan K Walker, Cheuk Hei Cheng, Alize Rattenni, Caroline Reynolds, and Sam Lapham
skwalker@umn.edu, cheng539@umn.edu, arattenn@umn.edu, reyno574@umn.edu, lapha018@umn.edu
University of Minnesota

**Abstract.** This multiple perspective, qualitative investigation with parents and teachers/teaching assistants provided insight on values of a virtual platform to aid parent collaborative learning in a dual focus early childhood program. Benefits were viewed as complementary or supplementary to face-to-face program components and context. Parent access to information about their children not available to them during program participation may deepen understanding and strengthen feelings of connectedness, particularly important to parent-teacher relationships. While there was synergy on most platform benefits, teachers did not observe parents' value in maintaining the program climate of trust and safety, and extending feelings of community. These may be essential for the platform to represent the learning supports aligned with the offline program experience. Study findings lead to hypotheses on the role of an online environment in parents' mental models of programs for collaborative learning, and in social presence that strengthen collaborative learning.

Virtual environments and interactive digital applications hold powerful potential for extending learning beyond face to face settings. Efforts to hybridize formal K-16 education with complementary technologies show positive impacts on student collaborative learning outcomes (e.g., Heflin, Shewmaker, Nguyen, 2017). Less well studied is technology-enhanced learning in nonformal education and with adult learners, specifically parents of young children who participate in community-based education programs. These learners and settings are critically important to the wider study of computer-supported collaborative learning. There is obvious societal value to promoting parenting role competence. And nonformal education group-based programs for parents prove successful at facilitating domain and practice learning through collaborative inquiry and reflection (Campbell & Palm, 2018). The conscientious integration of technology in ways that complement nonformal program aims for parent learning is rare (Strickroth & Pinkwart, 2013). The current study extends the authors' work employing design-based implementation research (DBIR) (Penuel, Fishman, Cheng & Sabelli, 2011) to explore technology integration in a dual focus early learning program (Walker, 2017).

## Theoretical and empirical background

Socioconstructivist perspectives on parent learning view the acquisition of knowledge about parenting and the self in the parenting role as a continuous process, involving reflection of experience and observation of and interaction with others who reinforce norms and validate experiences (Azar, 2003). As facilitated in group parenting education, collaborative inquiry occurs in discussion, critical self-reflection and exploration of the role through the sharing of experiences. This encourages the development of mental models, or conceptual schemes that guide parent action (Lam & Kwong, 2012; Marineau & Segal, 2006). Facilitators foster group cohesion and social capital through a climate of respect, trust and security (Campbell & Palm, 2018).

Research on technology's role in parent social learning has largely focused on individual participation in what Henri and Pudlelko (2003) label 'communities of interest' (e.g., specialized groups on social media) (Nieuwboer, Fukkink, & Hermanns, 2013; Zero to Three, 2016). Parents seek others online for information and support in ways that complement the personal and professional sources in their lives (e.g., family, friends, the pediatrician) (Zero to Three, 2016). And their investment in these online contacts varies with their technology comfort, and access. To a lesser extent, parents may participate in structured online parent 'learner communities' and research suggests benefits similar to offline collaborative settings (Gray, 2004). Online formats may also blend effectively to meet offline learning program aims. For example, Guldberg and Pilkington (2006) observed that, as facilitated by a trained peer as part of a credential program, a complementary online community for the caregivers fostered context (creation of a safe space, defining common values) and process (emergence of challenging questions, resolution-focused goal attainment) dimensions of collaborative learning. Theoretically then, if parents in a nonformal education program benefit from a collaborative, constructivist format offline and experience an online complementary experience, learning gains would be greater. This was the premise for our platform design and implementation research, as discussed below.
## Social learning technology and the early childhood family education program

Minnesota's Early Childhood Family Education (ECFE) is a parenting education program that adheres to constructivist principles. Operating through local school districts, ECFE offers weekly no/low cost two-hour classes and access to community resources to parents of children from birth through age five (ECFE.info). During the classes, parents and children interact together for the first hour, guided by licensed early childhood and parenting educators, then separate for the second (Figure 1). Unlike other group parent education, ECFE does not use an established curriculum; educators tailor learning content to individual groups, guided by field guidelines of early childhood program quality (Minnesota Department of Education, 2001). ECFE sites generally offer multiple class sections per site and hold several site-wide, all-family events per year. ECFE research has demonstrated processes and contexts of learning and instruction consistent with collaborative learning principles (Campbell & Palm, 2018). Weekly discussion time with other parents, as facilitated by parenting educators, fosters learning through information exchange, reflection on experiences and joint problem solving. Attention is given to a climate and context of support and respect. The first authors' own research indicates that parent relationships fostered in ECFE endure beyond the program and are supports throughout their child's life (Walker, 2018b).

<figure><img src="image_156.png" /><figcaption>Figure 1. Elements and Program Structure of ECFE. (PE = parent educator; EC = early childhood educator).</figcaption></figure>

Partnership with an ECFE site in an urban school district resulted in the development of an online platform to support ECFE's learning objectives (Walker, 2017). Participatory design research revealed parent preference for between-class social connectivity, information about parenting and about their child's learning; employing technology that was easy to use, mobile and ensured privacy (e.g., restricted access to class pages). The developed platform features a Facebook-like news feed for posts and discussion, and area for announcements, a shared calendar, dynamic photo album, external links, and members visual display (Figure 2). A user dashboard provides access to class pages, private messaging and notifications. Posts can be directed to the full site or to selected classes. Launch of the completed platform occurred in early 2017. Google Analytics tracking to the end of the school year revealed steady use by repeated visitors, with approximately 2/3 of sessions (i.e., group of site interactions per visit) by parents. Expanded platform use by additional ECFE sites in the subsequent school year showed similar proportionate use by parents (Walker, 2018a).

To date, investigations of the virtual platform show promise regarding usability, perceived usefulness by parents and staff and integration into program operations (Walker, 2018a). Contributing to new learning theory (Penuel, et al, 2011), requires deeper examination into cross-environment collaborative learning by parents inclusive of the roles played by teachers. Parenting educators are key to the parents' meaning making process (Lam & Kwong, 2012). And in early childhood settings that also serve parents, the perspectives of teachers and parents represent their adult roles (teacher; learner, parent) and are proxy for the children who also participate (Katz, 1993). Multiple perspective research allows a more complex and nuanced understanding of the shared experience [so that] views can be situated within the social relationships wherein they are constructed (Vogl, Zartler, Schmidt & Rieder, 2018, p.179).

## Method

In-depth interviews with multiple perspective qualitative analysis were employed to understand shared and unique perspectives from teachers, teaching aides, and parents on technology's role in promoting parent collaborative learning in ECFE.
<figure><img src="image_157.png" /><figcaption>Figure 2. Desktop view of ECFE virtual platform.</figcaption></figure>

## Sample

The sample consisted of 25 parents (19 women, six men) and six ECFE staff members (five women, one man) from the original ECFE design site. That site and parents for the study were selected for their demographic diversity and representativeness of families who attend ECFE in the school district. Parents were also sampled to represent all 8 weekly site classes. They ranged in age from 20 to 57 years (M = 35.3, S.D. = 1.5), and reported from one to six children (M = 2.7, S.D. = 1.7). Five parents reported caring for children with disabilities. Six parents came from the two Spanish-speaking classes. Parents were fairly evenly distributed by length of program participation (first year to more than 3 years). Staff held positions as licensed teachers (2 parenting educators, 1 early childhood teacher; all female) and as classroom assistants (3; 1 male). Their years of working in ECFE ranged from three to 35 years (M = 21.3). Staff ages ranged from 39 to 56 years (M = 48).

## Data collection and analysis

Semi-structured interviews explored participation or instructional experiences with the ECFE program and the use of technology in ECFE. Each interview lasted on average 1 hour (range: 45 minutes to 3 hours, 30 minutes). Audio-recorded interviews were conducted at the ECFE site by one of the three trained interviewers. Informed consent was obtained from participants prior interviews. Interviews were transcribed to text, then audited for accuracy by a second transcriber, and pseudonyms applied to all names identified for confidentiality.

An informed constructivist grounded theory approach (Charmaz, 2014; Thornburg, 2012) guided coding and analysis. Additional considerations in coding and analysis were deployed to compare within and across groups consistent with multiple perspective analysis (Vogl, et al 2018). The coding process with memoing consisted of three phases, initial coding, focused coding, and theoretical coding. Coders were assigned in pairs to analyze parent and staff interviews separately and together to enhance trustworthiness (Denzin, & Lincoln, 2018). During initial coding, transcripts were read and coded with the incident-by-incident approach, informed by sensitizing concepts from the literature on collaborative learning in parenting education. Second, focused codes emerged after reviewing initial codes. Themes identified within one relational unit (e.g. parents, teaching staff) were compared by coding pairs to generate central theoretical codes by unit. Finally, theoretical codes emerging from each relational unit were compared with each other. The themes emerged from staff and parents were compared across pairs of coders to understand the meanings of the perspectives, and shared and differing aspects of the perspectives.

## Results

A summary of parent and staff perspectives on collaborative learning as supported by the virtual platform is provided in Table 1. Learning supports were identified as process (e.g., aiding the acquisition or interpretation of content) or context (e.g., climate, tone) and as being supplementary (in addition to) or complementary (an extension of) the face-to-face program experience.

<caption>Table 1. Perspectives on technology benefits to collaborative learning in ECFE</caption>
<table><thead><tr><th>Construct</th><th>Evidence</th></tr></thead><tbody><tr><td>Information about Children<br>Process/Supplementary</td><td>Parents<ul><li>Positive emotion at seeing pictures of children in classroom while away</li><li>Feel taken care of, thought of by staff</li><li>Understand more about child and children</li></ul>Staff<ul><li>Opportunity to add information that parents don't get while in parenting education</li><li>Reflect interest by parents to learn more about children in classroom</li><li>Strengthen relationships between parents and teachers:</li></ul></td></tr><tr><td>Engagement with parenting content<br>Process/Complementary (main)/<br>Supplementary (secondary)</td><td>Parents<ul><li>Access information if can't attend (supplementary)</li><li>Share information with co-parent who can't attend (supplementary)</li><li>Gather information from parents in other class supplementary)</li><li>Read or view before or after class (complementary)</li><li>Get advice and information from parents and teachers (complementary)</li></ul>Staff<ul><li>Content to stimulate thinking before or after class (complementary)</li><li>Additional materials provided in handouts, drive links, URLs to deepen learning (complementary/supplementary)</li></ul></td></tr><tr><td>Relational/supportive engagement with families<br>Process/Complementary</td><td>Parents<ul><li>Support to parents from other parents, from staff</li><li>May use with other methods to connect with family and friends for support</li></ul>Staff<ul><li>Continue to provide 1-1 communication of support to parents.</li><li>Sharing images and videos with other children and families helps retain connections beyond class, over summer.</li></ul></td></tr><tr><td>Program communication and information<br>Process/Complementary</td><td>Parents<ul><li>Easy access to program reminders, events, 1-1 communication</li><li>Maintains 'idea' of program continually availability</li></ul>Staff<ul><li>Easy provision of program reminders, events, 1-1 communication</li><li>Makes work more efficient; easier to complete administrative tasks</li></ul></td></tr><tr><td>Climate, tone, safety, privacy and trust<br>Context/Complementary</td><td>Parents<ul><li>Feels like a safe space to post and view pictures of my child</li><li>Tone in discussion similar to what is experienced face to face</li><li>Appreciate member only access for privacy</li></ul></td></tr><tr><td>Maintain Community<br>Context/Complementary</td><td>Parents<ul><li>Able to maintain connections to others that extends community</li><li>Able to connect and see families who attend other classes. Represent diversity in community.</li></ul></td></tr></tbody></table>

## Supplementary information access: Child

Of all identified, this was the dominant subtheme expressed by parents and by staff. Both observed how teachers and occasionally, other parents posted pictures or videos of children. At times they augmented the images with text to identify developmental gains or processes. Parents expressed appreciation for the chance to learn more about their child, and what their child was doing in the classroom when they were in parenting education. Parents spoke with pride at seeing their child's activity or accomplishment; some also spoke of learning more about the child in context with other children. They also felt cared for by staff. Mother of 2, Mia said, "So it's nice to, like, have them vested in, like, 'hey the parents probably want to see their kids doing this'." For early
childhood staff who have limited interaction time with parents, it was the opportunity to help parents learn more about the child and child development. Staff also shared positive emotions about the exchange: *Teacher Jade:*

> "You know, I think that, that's a priceless tool, because in the moment I can have that [image of what a child is doing] with me, and I can look back to it; I can share it with the parent. I can say that I'm noticing something changing. We can look for patterns together. For both parents and early childhood staff, the opportunity and action in posting and reading/seeing information about the child appeared to strengthen the sense of connection.

## Supplementary information access: Parenting

Parent access to information about parenting when they weren't in class due to absence or inability to attend (as is the case of many co-parents) was less frequently but consistently mentioned across parents and the two parenting educators. The parenting educators felt that this helped them better reach all members of the class learning community. Parents appreciated feeling caught up. Supplementary access also meant learning from someone in a different class. Stella noted, "She had a special needs child ... and she put an article online that we could all read.... that's very cool and I think we need to know more about that...[there are] cool things that I can access now and I can learn more and I can become a better parent now because I get to see what everybody's doing"

## Parenting information complementary to classroom discussions

Parenting educators appreciated being able to post a video, reading or webpage to stimulate thinking before the weekly class or extend discussion afterward. Although parents were generally positive about having access to this information (e.g. "I think that would be really valuable because there's stuff that we're all interested in and I'm sure there's information and resources that it's being drawn from and it might be interesting to spend you know rather than flipping through Facebook, one night."), few spoke about continuing discussions outside of class. As mother of 6 Bella observed,

> "You know and that's kind of sad but it's the truth because our lives are so busy and fast moving. And then for me, I don't want to scroll through and see like what questions someone asked today, because they're not really my questions. I don't know. So if I have a burning question or something that's really is on my mind, I'm not going to probably take the time to like post it and then wait to see if anyone responds. Because it's not immediate as a text would be or a discussion with someone face to face."

## Continued access to program information

Parents and teachers liked easy access to information for registration, events, activities and reminders. The platform was a characterized as a one stop location to coordinate and find site information. Parents remarked that this helped simplify their busy lives; staff liked the efficiency of reaching more parents, more quickly. Reduced time in email and phoning, freed up time for teaching or other program responsibilities.

## Access to support from parents and teachers

Use of the platform as a vehicle to continue supportive connections between parents, and from staff to parents was a common theme across parents and teachers, especially the parenting educators and one of the early childhood aides (who'd worked in ECFE for over 20 years). Three parents from one class spoke of using the platform to organize meals for an ECFE family who'd just given birth to triplets. Parents also relayed a sense of relief or gratitude to extend 1-1 access to the parenting educator. In turn, staff commented that the platform complemented other virtual and real time means for providing support to parents. Parenting educator Everly said; '[...] you know I give parents my phone for texting, e-mail, [the platform] is great for that of course. And parents will call me, parents will text me. They'll reach out and...and not just me. I'm saying that sometimes it's a starter place. But I encourage them also to reach out to each other. So that's where the platform is helpful for that as well."

## Continuity of community and climate (parents)

A difference between parents and staff related to the platform extension of feelings of community, and program climate and tone. Parents observed this; staff did not. This quote from 21 year old Brooklyn conveys the extension of community:

> "And just being able to kind of like someone's picture of their kids you know kind of makes you feel like you're friends with that person and so it has that very Facebooky feel to it and that
> creates I think a kind of that community. And maybe give you an idea, you know, 'oh I saw that picture of you guys doing this and that looks fun' and it gives you maybe something to talk about when you see them in class...I have used it to connect with people outside of class and say like 'hey let's have a playdate go to the park' kind of thing. So that's a nice way to keep building that community."

Parents were also vocal about the maintained tone of civility and the sense of safety that they felt from the private, ECFE member only access. As Lily stated:

> "...having like a respectful place that we had just communicate with each other that you still feel like, ECFE feels like a safe space to me and I feel like Parentopia is kind of like, also it offers that as well, because they know that there is an administrator and there's not going to be any kind of negative thoughts or comments. I mean everyone's really supportive. Yeah and I think it's a place that I can kind of share my ideas too and with everyone and they can do the same"

## Conclusion and implications

This study offers preliminary support that parents and staff alike find shared and unique values to an online platform as an extension of the face to face weekly program, and identify a variety of benefits to collaborative learning. Platform interactions allow for continuity of connection (important for parents with infants, multiple children and single parents, and for co-parents and other caregivers who cannot attend). It extends access to parenting content, emotional and informational support, and to program events and activities. And significantly, it serves as a means to receive information about the child that the parent can't acquire due to the program structure that separates parents and children. Staff see the platform as improving the quality of their teaching and outreach, it may make their work easier and more efficient, and this may bring a sense of emotional satisfaction. While there is potential evidence of the platform's practical value to collaborative learning, insights from this study also suggest possible theoretical or conceptual advancements.

## Strengthening parent-teacher partnerships and educator presence

In the traditional ECFE program, parents separate from their 2 ½ -5 year old children during parenting education time and leave their children in the capable hands of the early childhood staff. While they look forward to time to themselves and the interaction with other adults for reflection, sharing and discussion, many wonder about their children's activity and happiness. The platform allows staff to post information, videos and text-augmented photos of the child(ren). Because of the community (class) and site (families) nature of the program, pictures of the child with other familiar children can help reinforce the parent's sense of the child in a wider social context. Parents expressed a range of positive emotions when voicing appreciation for the information about their child provided by the staff. Many conveyed sentiments of feeling closer to staff. In turn, early childhood staff who cannot interact with parents as much during the all family parent-child interaction time appreciate the ability to share information in other ways. Parents' sense of learning community membership is not exclusive to peers, but includes their children and ECFE staff (Walker, 2018b). Children are the reason that parents join ECFE, and raising healthy children is the value that all adults – parents and teachers – share in the program. Supplementary interactions may strengthen the sense of trust and connection between parents and professional staff who teach and care for the child (Lauluvein, 2010). Stronger parent-teacher relationships in early childhood bode well for children's learning outcomes and set the stage for partnerships in later school years (Froiland, Peterson & Davison, 2012).

This aspect of collaborative learning by parents facilitated online by teachers' actions may also have an emotional component. Sharing and receiving information about the child at the center of the teacher and parent's interest in ways that bring satisfaction to both may deepen the learning experience and offer a reciprocal interaction that sustains learning online and offline. Teachers like Everly observed the benefits to relationships: "*the more the parents see the teachers posting and they see their name up there and they see their face and they see what they posted they see a photo. Parents are bridging- they're connecting more. They're building stronger relationships.* ") Lehman's (2006) framework of presence in online learning suggests that addressing emotional considerations in interaction activities can help learners' confidence, focus and excitement to learn collaboratively.

## The importance of community and climate: Lessons for professionals
Whereas parents appreciated how the platform mimics the climate, tone, privacy and feelings of safety of the face to face site, and extended community and connection to others who share a parenting and program identity, staff didn't speak to these. The difference may be due to the 4 to 1 ratio of parents to professionals in the study, affording more observations by the former. The difference however, may be due to differences in perspectives of *being* a community member vs one whose *job* it is to foster learning as community. Staff were more articulate about the specific technologies they use for instruction and communication, and pedagogical purposes related to sharing information, communication, extending class content and lending support. A focus on technology for teaching over the social experience of the learner may limit teachers from their seeing themselves as participants in discourse or as members of parents' learning community (Lauluvein, 2010). They may focus on providing content when a busy parent with other content resources may more highly value their presence and affirmation.

Staff may need guidance to extend community in ways that maintain the visual membership and emotional tone of the face to face program. This means encouraging families to post information about themselves, pictures from neighborhood events, and posting in a site-wide forum. It can also mean monitoring discourse to maintain exchanges in the civil, positive and nonjudgmental tone that led parents in this study to make favorable comparisons to the platform over their less then positive experiences with Facebook. Encouraging a respectful online climate and sense of community can help parents make connections across classes and build bridging social capital valuable to learning (Wenger, White & Smith, 2009).

## The platform as representation of whole program participation experience

Parents, teachers and teaching assistants identified a range of ways in which using the platform served to complement the weekly face to face learning experience. These included exposure to parenting content curated by professionals, provision of up to date program information, access to supportive others, outlets for ongoing communication to peers and professionals, and continued visual and text reminders of membership in a learning community. As these informational, interactive and relational elements align with constructivist, collaborative inquiry approaches in parenting education (Campbell & Palm, 2018; Lam & Kwong, 2012) and represent the values and operation of ECFE, they may indicate the potential of a virtual platform as useful to the multiple process and context mechanisms that foster parent collaborative learning.

Theoretically however, the platform may also facilitate parent learning from a conceptual standpoint. That the platform serves many learning and engagement functions complementary to parents' ECFE experience may support parents' mental image of themselves as participating in a learning community beyond the weekly session structure. The notion is expanded as they are also “in the children's classroom” through the supplementary images and information about their children. This may give them a sense of participating in all components of the weekly program depicted in Figure 1. And as ECFE is a place that is positive, as busy, stressed parents, the platform sends the message of ECFE being continually present, reliable and responsive, and this may bring them comfort. This includes the consistently strong presence of staff who for some, function as significant emotional supports. So, the platform's ability to extend the 'idea' or mental model the program that parents affiliate with as a supportive space for learning with others, including staff, may be another complementary value of the platform for parents' collaborative learning.

This study is, to be sure, preliminary in its findings. It depended on the voices of parents and staff at a single ECFE site, about their experiences with a newly designed platform early in its implementation. Yet, there is promise in the findings to buoy continued platform implementation in the school district program that will allow use by larger numbers of parents and staff, at more ECFE sites. And it encourages further investigations in the program-platform coordinated role in parent learning. To date the platform is being used by eight ECFE programs in four school districts and reaching over 450 parents and teachers (Walker 2018a). The current findings validate employing design-based implementation research when identifying innovation for a long-standing face-to-face parenting education program. Attention to the needs of the program and of the participants and staff allowed development of a useful and possibly critical innovation to face to face parent education and extending adult learning benefits. Continued research on implementation of the platform will suggest avenues for its contribution to learner success and for sustainability within nonformal, community-based parent education settings.

## References

* Azar, S. T. (2003). Adult development and parenthood: A social-cognitive perspective. In J. Demick & C. Andreoletti (Eds.), Handbook of adult development (pp. 391-415). New York: Springer.

* Campbell, D. and Palm, G. (2018). *Parent education: Working with groups and individuals* (1st ed.). San Francisco, CA: Jossey-Bass.

* Charmaz, K. (2014). *Constructing grounded theory*. London; Thousand Oaks, Calif.: Sage.
Denzin, N. K. and Lincoln, Y. S., Editors. (2018). *The SAGE handbook of qualitative research*, 5th edition.
Thousand Oaks, CA: Sage.

Froiland, J. M., Peterson, A., & Davison, M. L. (2012). The long-term effects of early parent involvement and parent expectation in the USA. *School Psychology International*, 34(1), 33-50. doi: 10.1177/0143034312454361

Gray, B. (2004). Informal learning in an online community of practice. *Journal of Distance Education, 19*, No 1, 20-35.

Guldberg, K. & Pilkington, (2006). A community of practice approach to the development of non-traditional learners through networked learning. *Journal of Computer Assisted Learning* 22, 159-171.

Heflin, H., Shewmaker, J., & Nguyen, J. (2017). Impact of mobile technology on student attitudes, engagement, and learning. *Computers & Education, 107*, 91-99.

Henri, F. and Pudelko, B. (2003). Understanding and analyzing activity and learning in virtual communities. *Journal of Computer Assisted Learning, 19*, 474-487.

Katz, L. G. (1993). Multiple perspectives on the quality of early childhood programs. *European Early Childhood Education Research Journal, 1*(2), 5-9.

Lam, C. M., & Kwong, W. M. (2012). The “paradox of empowerment” in parent education: a reflexive examination of Parents' pedagogical expectations. *Family Relations, 61*(1), 65-74.

Laluvein, J. (2010). Parents, Teachers and the "Community of Practice". The Qualitative Report, 15(1), 176-195.

Lehman, R. (2006). The role of emotion in creating instructor and learner presence in the distance education experience. *Journal of Cognitive Affective Learning, 2*(2), 12-26.

Lincoln, Y. S., & Guba, E. G. (1985). *Naturalistic inquiry* (Vol. 75). Sage.

Marienau, C. & Segal, J. (2006). Parents as developing adult learners. *Child welfare, 85*(5), 768-784.

Minnesota Department of Education (2001). Early childhood quality indicators of progress. St Paul, MN.
https://education.mn.gov/MDE/dse/early/ind/

Nieuwboer, C. C., Fukkink, R. G., & Hermanns, J. M. (2013). Online programs as tools to improve parenting: A meta-analytic review. *Children and Youth Services Review, 35*(11), 1823-1829.

Penuel, W. R., Fishman, B. J., Cheng, B., & Sabelli, N. (2011). Developing the area of design-based implementation research. *Educational Researcher*. Menlo Park, CA: SRI International.

Strickroth, S., & Pinkwart, N. (2013). An Approach for Supporting Hybrid Learning Communities: The Case of a Regional Parent Community. In *Proceedings of the CSCL*.

Thornberg, R. (2012). Informed grounded theory. *Scandinavian Journal of Educational Research*, (56), 3, 243-259.

Vogl, S., Zartler, U., Schmidt, E. M., & Rieder, I. (2018). Developing an analytical framework for multiple perspective, qualitative longitudinal interviews (MPQLI). *International Journal of Social Research Methodology, 21*(2), 177-190.

Walker, S. (2018a). Parentopia: Overview of social interaction technology to extend family engagement and support. University of Minnesota.
https://drive.google.com/file/d/1D_ee7Jp8_Y70tvqXeHYVKGeUFVnRfeuD/view?usp=sharing

Walker, S. (2018b). I think it's made everyone a little closer: A virtual platform extending parent learning as Community of Practice. Poster presented at the 2018 International Society for Learning Sciences conference, London, June 29.

Walker, S. (2017). Creating Parentopia: Design-based Research to Develop an Interface for Parent Learning Communities and Networks. Computer Supported Collaborative Learning bi-annual conference. Philadelphia, PA

Wenger, E., White, N. and Smith, J.D. (2009). *Digital habitats: Digital habitats: Stewarding technology for communities*. Portland, OR: CPSquare.

Zero to Three (2016). *Tuning In: Parents of Young Children Speak Up About What They Think, Know and Need*. Washington, DC. https://www.zerotothree.org/resources/
# An Alternate Statistical Lens to Look at Collaboration Data:
Extreme Value Theory

Kshitij Sharma, Department of Computer Science, Norwegian University of Science and Technology,
Kshitij.sharma@ntnu.no

Jennifer Olsen, Computer Human Interaction Lab in Learning and Instruction, École Polytechnique Fédérale de
Lausanne, Jennifer.olsen@epfl.ch

**Abstract:** To provide beneficial feedback to students during their collaboration, it is important to identify behaviors that are indicative of good collaboration. However, in a collaborative learning session, students engage in a range of behaviors and it can be difficult to indicate which of those behaviors correlate with higher outcomes. In this paper, we propose using Extreme Value Theory (EVT), a method that considers the data points in the tail (upper or lower) of the distribution, to analyse the relationship between collaborative process variables and outcome measures through insights derived from high impact, low-frequency events. Specifically, in this paper, we analyse the relationship between dual gaze patterns and outcome measures across two different datasets. In both datasets we found that students with lower outcomes had lower focus during the collaborative session. This paper provides a contribution by both introducing EVT as a viable method for analysing CSCL data as well as demonstrating the effectiveness of eye-tracking as a collaborative indicator to use to adapt to in real-time.

**Keywords:** Eye-tracking, dual eye-tracking, extreme value theory, CSCL, collaborative process, intelligent tutoring systems, concept maps.

## Introduction

Collaboration can be an effective way of learning, but it is challenging to ascertain how students' actions lead to learning when working in a group and how to determine what actions should be considered when providing students with feedback on their collaboration. To provide timely and effective feedback, it is important to have an indicator of collaboration that can easily be measured in real-time and is a predictor of learning. However, across collaboration indicators, the connections between these student process measures and learning outcomes have not always been clear (Deiglmayr, Rummel, & Loibl, 2015; Olsen, Aleven, & Rummel, 2017). In this paper, we propose analysing only the high impact, low frequency data points to provide additional insights into the relationship between process data and collaborative learning outcomes. We introduce the use of Extreme Value Theory (EVT), a method that considers the data points in the tail (upper or lower) of the distribution, to analyse the relationship between collaborative process variables and outcome measures. Specifically, we are interested in dual eye-tracking as a process variable as it can easily be measured in real-time and be used to provide students with feedback (Sharma et al., 2016, D'Angelo & Begel, 2017). Using EVT, we aim to increase our understanding of the process variables that impact student collaborations and how they can be used to provide students with real-time feedback.

EVT is a novel method to CSCL and presents a complementary viewpoint to the statistical methods often used to analyse collaborative learning data while having fewer data assumptions. EVT can be used for both explanatory analyses (Ramesh and Davison, 2002) and hypothesis verification (Santinelli et al., 2014). The mathematical foundations of EVT are similar in strength, rigor and maturity as compared to the methods based on the central tendencies in the data (Smith 1990). EVT provides a way to estimate the probability of occurrence of rare events, which might also be unseen in the observed data. This makes EVT a unique method to implement proactive feedback to support collaboration. For example, in the context of collaborative eye-tracking, if we could estimate the probability of peers not focusing on any part of the communication mediating interface, we could suggest some remedial actions in a proactive manner. In contexts where the proactive feedback is required, the tail of the data may contain more information about the process than the main body of the distribution.

In this paper, we apply EVT to eye-tracking as it has become a key source of process data in educational research over the past few years. Research using eye-tracking covers a wide range of educational ecosystems from online (Sharma et. al., 2015) to face to face classes (Raca & Dillenbourg, 2013), from co-located (Schneider et. al, 2016) to remote collaborative learning (Sharma et. al., 2012), and to understand teachers' classroom orchestration processes (Prieto et. al., 2016). Eye-tracking has not only been used to understand the learning processes in various contexts, but it also has been used to provide students appropriate, real-time, and adaptive feedback on their learning processes (Sharma et al., 2016, D'Angelo & Begel, 2017).
One of the most common practical uses of dual eye-tracking (DUET) data in CSCL has been to quantify collaborative outcome and processes. In terms of collaborative outcome, recent results have shown DUET measures of gaze cross-recurrence (looking at the same area of the screen at the same time) to be useful in quantifying collaboration quality (Schneider et. al., 2013; Jermann and Nuessli, 2012) and the differences between the expertise (Papavlasopoulou et. al., 2017). Further looking at the similar areas of the screen in each time window was found to be correlated with learning gains (Sharma et. al., 2015; Schneider et. al., 2016; Sangin et. al., 2011). DUET measures such as average entropy (Sharma et. al., 2012), transitions among areas of interest (Villamor and Rodrigo, 2018) were correlated with task based performance in pair programming tasks. In terms of collaborative processes, eye-tracking research has shown to useful in relating gaze to dialogue patterns such as the eye-voice span (time difference between the speaker's gaze at an object and verbalization, Allopenna et al., 1998), voice-eye span (time difference between the speaker's voice and listener's gaze to the referred object, Griffin and Bock, 2000) and eye-eye span (time difference between the speaker's gaze at an object and the listener's gaze at the same object, Richardson et al., 2007). Gaze cross-recurrence was also useful in explaining the conceptual knowledge gains of the peers in a collaborative learning scenario with intelligent tutoring systems (Belenky et. al., 2014). One common theme across these studies are the statistical methods used. Most of these studies used methods based on the central tendencies of the data. We propose a complementary viewpoint for the DUET data in this contribution that is based on the extreme values present in the data.

Within this paper, we aim to answer the research question of how the visual focus of peers is related to the collaborative learning outcomes. We propose two types of shifts from the traditional analyses. First, we move from the main body of the distribution (central tendency) to the tails of the distribution (extremes). Second, we move from the individuals (or groups) as a unit of analyses to the specific moments in the interaction. Specifically, we are interested in *how the extreme values (i.e., the moments with extremely high or extremely low values) can inform us of the students' learning gains*. To answer this question, we present the EVT method in two different dual eye-tracking (DUET) contexts. The first study is in a collaborative concept map context with university students, and the second study is in a collaborative Intelligent Tutoring Systems (ITS) context with elementary school children. For both contexts, we hypothesized that the gaze of the students with lower test scores will have a higher tendency to wander all over the screen. In the following sections, we will present the EVT methodology and how it can be used to provide insights into effective student collaboration processes. This paper contributes to the CSCL literature by providing an alternate method for analysing relationships between process and outcome data that complements existing methodology while also extending our understanding of the relationship between visual focus and collaborative outcomes.

# Method

## Extreme Value Theory

Extreme events are those with high impact and low frequency (HILF), and EVT is the branch of statistics used for modelling HILF events. The basic idea is to model an extreme event in a way such that the analyses and implications account for the needs of unprecedented situations. For example, in finance and environmental sciences, HILF events are often analysed to allow people and companies to be proactive against negative events (e.g., risk, losses, natural disasters). We can extend this proactive policy to education by considering negative and positive HILF events. EVT deals with asymptotic data where the data used in the analyses is a small subset of the whole data -- usually below 5th percentile or above 95th percentile. Mathematically, EVT is based on the tail of a given distribution. Because EVT is based on the tails, it does not impose any assumption on the distribution and can be applied to any known (e.g., normal, student, uniform, exponential) or unknown distribution. EVT was developed initially for independent and identically distributed variables (for a complete introduction to Extreme Value Theory, see Coles, 2001), but it can easily be extended to stationary variables (variables whose distributions are not affected by the time shifts), which are most variables addressed in the CSCL community.

## Advantages and disadvantages

There are three main advantages of EVT. **First**, since EVT does not have any assumptions for the data distribution, it can be used to analyse the tail (data below the 5th percentile of above the 95th percentile) of any given distribution. This makes EVT applicable with those datasets which cannot be analysed using the common parametric approaches. For example, ANOVA requires the data to follow a normal distribution and if not, normalisation operations can affect the interpretability of results. **Second**, when analysing the dependence structure between two variables, such as correlations, EVT does not assume the dependency to follow a known structure. For example, in case of correlations, this structure is assumed to be linear. This lack of assumption allows EVT to be applied irrespective of the nature of the distribution that generates the data and dictates the
relationships among different variables. The **third** advantage of EVT is over non-parametric models. Non-parametric models, which are used in CSCL as a way of hypothesis testing, provide one value (p-value). These methods summarise the data and can handle only low dimensional problems. Given the advents in big data, non-parametric, which are designed for smaller datasets, are at an inherent disadvantage. Also in time series analyses, dynamic models provide much more information about the process than non-parametric models. EVT accounts for time series analyses by providing methods to consider the covariate dependence on time while analysing CSCL data.

The main disadvantage of using EVT is related to the fact that EVT considers the tail of the data only. By only considering the tail, the parameter estimation becomes more difficult for datasets with few events. In other words, EVT is appropriate to be used contexts where the data is sufficiently large. However, in the “big data era” this problem is often solved by itself and in these circumstances the methods based on EVT provide robust approach for estimating the probabilities of rare events.

## EVT: Univariate

There are two ways to model data using EVT: blockwise maxima and points over threshold. In **blockwise maxima**, we divide the set of observations into M blocks of N data points. This results in a sequence of maximum values for each block. One of the most important results of EVT formulation is that for any given distribution for the main body of data, known or unknown, the sequence of maxima follows a unique distribution called Generalised Extreme Value (GEV). Once we have the sequence of maxima, we simply fit the GEV distribution on them using any likelihood optimisation technique. This distribution is characterized by three parameters: position, scale and shape. These values are then used to estimate the value which has a low probability of being exceeded (see further in this section).

Using **points over threshold method**, we consider all the data points over a threshold. This threshold is usually a high percentile of the observed data (90th or 95th percentile). These data points above the threshold are called the exceedances. Using points over threshold method, the exceedances follow a Poisson distribution and the exceedance size follows a Generalised Pareto Distribution (GPD). The parameters of GPD can be estimated by GEV. The shape parameter for both GPD and GEV remains the same. This parameter determines how long/heavy the tail of the observed data is. Once we have the threshold, the rate parameter of the Poisson distribution and the scale and shape parameters of GPD can be estimated, like GEV, by using a likelihood optimization technique. Figure 1 shows the difference in the two approaches.

The position (or rate of Poisson), scale and shape parameters are difficult to interpret in CSCL situations, since these values do not correspond to any behavioural indicator. Therefore, we calculate a quantile value at a high level (above 90th or 95th percentile), which has an important interpretation. This value is called **Return Level**, which represents a measure of an extreme event, possibly unseen, with a certain probability. For example, if a return level is calculated at the 95th percentile, this indicates that the actual (unseen) extreme event will exceed this value with a 0.05 probability. In other words, a return value at 95th percentile is the value that has a probability of 0.05 of being exceeded. One can estimate this value by position, shape and scale measures from GEV or rate of arrival of exceedances, shape and scale measures of GPD.

One might inquire that if the high percentile can be directly calculated from the data, then why we need this estimation. The answer to this question is that we could surely calculate the return levels from the data, however minor discrepancies in the data would result in large errors for such computation, resulting in an erroneous return value larger or smaller than expected. If the return value is larger than it should be then one cannot proactively take actions; and if the return value is smaller, then one will be acting upon at a wrong time. It can be shown, mathematically that estimating GEV or GPD parameters are the correct way of computing the return level in the data. Finally, once we have the return level for the collaborating partners, we can compute the difference between them: by value and/or when they appear, and this difference can be used as a measure to further correlate against the quality/outcome of collaboration.

## EVT: Bivariate

To analyse the time series data from peers, the bivariate case of EVT can be useful. Bivariate EVT measures the extremal dependence between two time series data, such as that between a collaborating dyad. It models the probability of observing an extreme event in one time series given that there is an observable extreme event in the other one. This probability can be quantified using the tail-dependence between the two time series. In a classical statistical approach, the dependence between two time series is measured by correlation, which is computed using the central tendencies of the data. In the case of EVT, the tail-dependence is calculated at the high percentile of the data, as in case with the return levels.
<figure><img src="image_158.png" /><figcaption>**Figure 1.** (a) A random variable simulation with the blockwise-maxima; (b) density plots for one of the blocks; (c) the red horizontal line shows the threshold for the POT method; (d) the whole distribution, the red vertical line shows the threshold for the POT method and denotes the beginning of the tail for the distribution.</figcaption></figure>

There are two measures of extremal dependence, originating from classical multivariate EVT: asymptotic dependence and asymptotic independence. The coefficient of asymptotic dependence (CAD) is the tendency for one variable to be over a high threshold when the other exceeds this threshold. This value is always between 0 and 1. The only possibility of asymptotic independence is when CAD is 0. When the CAD is greater than 0, the variables are asymptotically dependent. On the other hand, the coefficient of asymptotic independence (CAI) is the measure of strength of this extremal dependence. This is measured by a conditional probability that the smaller values in the time series of one variable are below a infinitesimal threshold, given the smaller values in the other time series are below that threshold. Mathematically, it can be shown that a value of 1 for CAI shows the perfect dependence and a value of 0 for CAI shows perfect independence. In summary, the CAD shows the level of asymptotic dependence between two time series while the CAI shows the strength of this dependence. Figure 2 shows an example of how to determine CAD and CAI.

<figure><img src="image_159.png" /><figcaption>**Figure 2.** Example illustrating the determination of the coefficient of extremal dependence and the strength of dependence for a typical time-series pair. The dashed lines represent the 95% confidence intervals for CAD (top, Chi) and CAI (bottom, Chi Bar). The tail-dependence and its strength is determined by the values at the higher quantiles (typically between 95% and 99%). The red lines correspond to 95%.</figcaption></figure>

## Examples

In this section, we provide two different examples from distinct collaborative learning scenarios. For each context, we show how EVT can be applied on one gaze variable computed from both the studies, and how the different EVT based measurements differentiate the quality/performance levels in collaborative learning outcomes.

## Collaborative concept map

This data set involves 24 dyads from a larger study that tested a hypothesis about the relation between individual and collaborative gaze patterns (Sharma et. al., 2015). Each dyad was engaged in a collaborative concept-map building activity. The students were sitting on two sides of a visual separation and could talk to each other. Prior to the concept-map building activity, they individually watched two videos from Khan Academy about resting membrane potential and they were asked to build the concept-map about the same topic. The main task was to relate the pre-defined concepts and add new concepts if they felt necessary. The students watched the video at their own pace and the total duration for the concept-map activity for each dyad was between 10 and 12 minutes. For this contribution, the dependent measure is calculated as follows. The final concept-map was compared with the concept-map created by the two experts. The pair received a score using the following rules: 1) one mark for
each correct connection between two concepts, 2) one mark for each correct label of the edge between two concepts, 3) half a mark for each partially correct label of the edge between two concepts. The pairs were then divided into two levels based on the concept-map score using a median split.

## Collaborative intelligent tutoring systems

This data set involves 14 4th and 14 5th grade dyads from a larger study that tested the hypothesis about differential benefits of collaborative versus individual learning (Olsen et. al., 2014). The dyads were engaged in a problem-solving activity around fractions using a networked collaborative ITS, which allowed them to synchronously work in a shared problem space where they could see each other's actions while sitting at their own computers across the room from each other. The students could communicate verbally through a Skype connection. Each dyad worked with the tutor for 45 minutes in a pull-out study design at their school. The morning before working with the tutor and the morning after working with the tutor, students were given 25 minutes to complete a pretest or posttest individually on the computer to assess their learning. During the experiment, dual eye tracking data, dialogue data, and tutor log data in addition to the pretest and posttest measures were collected. For this contribution, the dependent measure is the average posttest score of each dyad.

## Variable: Spatial Entropy

To capture the visual focus, we use Spatial Entropy (SE) that is one of the measures used to analyse DUET data in previous research (Olsen et. al., 2018; Sharma et. al., 2018) and show the results base on EVT analyses for both learning contexts. SE measures the spatial distribution of the gaze of each peer. To compute SE, we first define a 50-pixel by-50-pixel grid over the screen and we compute for each peer the proportion of fixation time located in each grid cell (Figure 3). This proportion is computed over a time window of five seconds. This results in a proportionality matrix and the SE is computed as the Shannon entropy of this 2-dimensional vector. The spatial entropy is also task-independent, as it can be computed for any task, but the interpretation of the entropy values might be dependent on the visual stimuli. A low value of SE would mean that the subject is concentrating on a few elements on the screen, while a high SE value would depict a wider focus size.

What does extreme spatial entropy mean? The idea is to capture the visual focus size (not attention, although in the contexts of the two examples they might be related) of the participants. The higher the spatial entropy is, the larger the focus size is. A spatial entropy value of zero indicates that the participant is looking at only one part of the screen and higher values indicate that the participant looks at different parts of the screen, during a given time window. Now, an extreme spatial entropy would indicate that the participant is looking “all over the place”.

## Results

### Collaborative concept map

First, using the univariate EVT, we check the average return levels of the two spatial entropy time series. The average return levels (calculated at 95th percentile) for spatial entropy is lower for the pairs with high collaboration outcome (F[1,15.78] = 6.53, p-value = .01, one-way ANOVA without assuming equal variances) than the average return levels of spatial entropy for the pairs with low collaboration outcome.

Second, considering bivariate EVT results for the spatial entropy of the peers, there are two values to be checked: 1) the level of extremal dependence and 2) the strength of extremal dependence if the level is non-zero. We observe a higher extremal dependence (calculated at the 95% quantile) between the spatial entropy of peers with low collaboration outcome (F[1,22] = 4.28, p-value = 0.01) then the extremal dependence between the spatial entropy of peers with high collaboration quality. We observe an even more significant difference in the strength of extremal dependence (calculated at the 95% quantile) for the pairs with the two different levels of collaboration outcome (F[1,22] = 10.43, p-value = 0.001). Pairs with low level of collaboration outcome have stronger extremal dependence between the spatial entropy of peers then that for the pairs with high level of collaboration outcome.

## Collaborative intelligent tutoring systems

In the univariate EVT case for the ITS data, the average return levels (calculated at 95th percentile) for spatial entropy is negatively correlated with the pair's average posttest score (cor = -0.48, p = .01). Next, we look at the bivariate EVT for the ITS data. There is a negative correlation between the upper tail dependence (calculated at the 95% quantile) for the spatial entropy of peers and the average post test score of the pairs (cor = -0.48, p = .01). In the case of ITS data, we observe a significant and negative correlation between the strength of upper tail
dependence (calculated at the 95% quantile) for the spatial entropy or pairs and their average posttest score (cor = -0.43, p = .03).

<figure><img src="image_160.png" /><figcaption>**Figure 3.** The process of computing entropy. The image on the left shows the exemplar concept-map and gaze patterns (grey circles and arrows). The image on the right shows the placement of the grid.</figcaption></figure>

Discussion

In this paper, we presented a new method in the context of analysing CSCL data, specifically dual eye-tracking data. We propose that EVT based methods are robust enough to estimate the probabilities of the rare events, which can then be used to provide proactive feedback to students. As examples, we presented results from two dual eye-tracking studies: collaborative concept map and collaborative ITS. To explain the findings from both the studies in a unified way, we use the same metric to capture the focus (spatial entropy) of the peers in two studies.

In the univariate EVT case, we propose to use the return value at the 95th percentile. This value of the spatial entropy has 5% chances of being exceeded. Across both the concept map and ITS contexts, the results indicate that the average values of extreme entropy is higher for the pairs with the low collaborative outcome/learning than the pairs with high collaborative outcome/learning. This simply translates to the fact that pairs with high levels of collaborative outcome/learning have lower levels of spread-out gaze patterns.

The bivariate context provides a supporting explanation for the results from the univariate EVT. We observe that both the level and strength for the upper tail distribution of the spatial entropy is negatively correlated with the collaborative performance/learning, i.e., the moments of extreme entropy appear together in time for the pairs with low collaborative outcome/learning. Combining this with the univariate results, we can conclude that the peers with low collaborative outcome/learning not only have higher chances of "looking all over the place" but there are high chances of them looking all over the place at the same time.

This indicates that the pairs with low collaborative outcome have moments where both the participants in the pair have extremely large visual focus. The cause of such behavior could be explained in two different ways. First, both the participants are looking for some information on the screen and thus they have a large visual focus size. Second, the mutual understanding has a missing link that needs to be created between the two peers. This information can be used to intervene proactively by providing greater scaffolding for student interactions to help guide the collaborative learning process when students have low focus at the same time.

In terms of contemporary dual eye-tracking analyses methods, the bivariate EVT is like analysing the gaze cross-recurrence. Gaze cross recurrence (CR) has been found to be correlated with the collaboration quality (Jermann and Nuessli, 2012; Schneider et al., 2013). CR indicates the time peers spent looking at the same area on the screen at the same time. In the bivariate EVT case, visual focus of the peers is compared over time. Having comparable visual focus size does not necessarily mean that the peers are looking at the same part of the screen. However, having a large focus size in each time window will result in a high CR over time, as in an aggregated time frame the peers would be looking at the same area on the screen, which is the whole screen. One interesting finding from the examples presented in this paper is the relation between the extreme visual focus and the average learning gains of the peers. In a recent contribution (Olsen, Aleven, & Rummel, 2017), the researchers did not find a relation between CR and overall learning gains of the students using the same collaborative ITS data. This suggest that by analysing the HilF data, we may be able to gain additional insights that are not apparent when analysing the entire data set.

The relation between the extreme visual focus and collaborative outcome/learning, provides an opportunity for designing proactive feedback tools to support collaboration. Using the methods described in this
contribution, one can identify the key moments to provide the feedback to the collaborators. Most of the recent work done in the direction of using gaze awareness to scaffold collaboration has been focused onto showing the gaze of the peers to each other. For example, Ishii and Kobyayash (1992) and Monk and Gale (2002) designed systems displaying the face of the collaborators. Stan and Brennen (2004) showed that displaying partners' gaze while debugging a program helped finding bugs. In another experiment, Brennan, et. al. (2008) showed that displaying partners' gaze in "Os-in-Q" search helped the collaborators in more effective labor division. Recently, the gaze-awareness has been used to support collaboration in high level tasks such as pair programming (D'Angelo and Begel, 2017). Gaze awareness has also been shown to be useful in co-located collaboration scenarios (Van Rheden et. al., 2017) and remote collaborations using different types of devices (Akkil, et, al., 2018). One common theme across these gaze visualisations is that the gaze is visualized throughout the whole collaborative work, which might hinder the learning experience on a few occasions. Identification of the key moments during the collaboration for providing support might improve the effectiveness of the gaze aware tools.

With the results from the two DUET studies based on the EVT, we can estimate a high value of visual focus for both the participants which has a very low probability of being exceeded. In a collaborative scenario if we observe that the visual focus sizes for both the participants in the dyad is going to exceed a certain threshold, we can proactively support the pair to avoid disruptions in the collaboration. EVT has these added values for gaze-aware feedback systems, which could enable the instructor to be proactive and select the exact moments to provide guidance rather than visualizing the support throughout the collaboration.

## Conclusions

We presented a complementary method to analyse CSCL data based on the tails (lower or upper) of the distribution. This method is based on a mathematical theory, which is novel to CSCL community, called Extreme value theory. The main motivation behind using EVT is to be able to provide proactive scaffolding during the moments, when the collaboration between peers is at a point where the outcome/quality is disruptive. We do not claim the superiority of this method (for a comparison with the traditional methods of analyses, see Sharma et. al., 2016). We propose, that EVT provides a different point of view for the data when the traditional methods do not apply because of the failed assumptions or when the traditional methods do not provide any useful insights about the collaborative processes, outcome or quality.

Within this paper, we provide a contribution both through the application of EVT to educational process data as well as furthering the understanding of how dual eye-tracking relates to collaborative learning outcome measures. By analysing just, the tail of the data, we can distinguish patterns that may not have otherwise been apparent. Across contexts, we have shown a common pattern of with low outcome measures having lower focus at the same time. We propose EVT as an alternative method for the analysis of collaborative learning data that can provide complementary viewpoints to the common CSCL methodological repertoire.

## References

* Akkil, D., & Isokoski, P. (2018). I see what you see: gaze awareness in mobile video collaboration. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications (p. 32). ACM.

* Allopenna, P.D., Magnuson, J.S, and Tanenhaus, M.K. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models* 1, * 2, * 3, * 4, * 5. Journal of memory and language, 38(4),

* Bednarik, R., & Kauppinen, M. (2013). Unravelling the interaction strategies and gaze in collaborative learning with online video lectures. In Proceedings of the 6th workshop on Eye gaze in intelligent human machine interaction: gaze in multimodal interaction (pp. 57-62). ACM.

* Belenky, D., Ringenberg, M., Olsen, J., Aleven, V., & Rummel, N. (2014). Using Dual Eye-Tracking to Evaluate Students' Collaboration with an Intelligent Tutoring System for Elementary-Level Fractions. Grantee Submission.

* Coles, S. (2001). An introduction to statistical modeling of extreme values. London: Springer. http://dx.doi.org/10.1007/978-1-4471-3675-0

* D'Angelo, S., & Begel, A. (2017). Improving communication between pair programmers using shared gaze awareness. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 6245-6290). ACM.

* Deiglmayr, A., Rummel, N., & Loibl, K. (2015). The Mediating Role of Interactive Learning Activities in CSCL: An Input-Process-Outcome Model In Lindwall, O., Häkkinen, P., Koschman, T. Tchounikine, P. Ludvigsen, S. (Eds.) (2015). Exploring the Material Conditions of Learning: The Computer Supported Collaborative Learning (CSCL) Conference 2015, Volume 2. Gothenburg, Sweden: The International Society of the Learning Sciences.
Embrechts, P. (2000). Extreme Value Theory: Potentials and Limitations as an Integrated Risk Management Tool.
Manuscript, Zurich, Switzerland: Department of Mathematics, ETH, Swiss Federal Technical
University.

Fussell, S. R., Setlock, L. D., & Parker, E. M. (2003). Where do helpers look?: gaze targets during collaborative physical tasks. In CHI'03 Extended Abstracts on Human Factors in Computing Systems (pp. 768-769). ACM.

Griffin, Z.M., and Bock, K. (2000). What the eyes say about speaking. Psychological science, 11(4).

Ishii, H. and Kobayashi, M. (1992). Clearboard: a seamless medium for shared drawing and conversation with eye contact. In Proceedings of CHI, ACM.

Jermann, P., & Nüssli, M. A. (2012). Effects of sharing text selections on gaze cross-recurrence and interaction quality in a pair programming task. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (pp. 1125-1134). ACM.

Monk, A.F., and Gale, C. (2002). A look is worth a thousand words: full gaze awareness in video-mediated conversation. Discourse Processes, 33(3), 257-278.

Olsen, J. K., Aleven, V., & Rummel, N. (2017). Exploring Dual Eye Tracking as a Tool to Assess Collaboration. In Innovative Assessment of Collaboration (pp. 157-172). Springer, Cham.

Olsen, J., Sharma, K., Aleven, V., & Rummel, N. (2018). Combining Gaze, Dialogue, and Action from a Collaborative Intelligent Tutoring System to Inform Student Learning Processes. International Society of the Learning Sciences, Inc.[ISLS].

Papavlasopoulou, S., Sharma, K., Giannakos, M., & Jaccheri, L. (2017). Using Eye-Tracking to Unveil Differences Between Kids and Teens in Coding Activities. In Proceedings of the 2017 Conference on Interaction Design and Children(pp. 171-181). ACM.

Prieto, L. P., Sharma, K., & Dillenbourg, P. (2015). Studying teacher orchestration load in technology-enhanced classrooms. In *Design for teaching and learning in a networked world* (pp. 268-281). Springer, Cham.

Ramesh, N. I., & Davison, A. C. (2002). Local models for exploratory analysis of hydrological extremes. Journal of Hydrology, 256(1-2), 106-119.

Richardson, D. C., Dale, R., and Kirkham, N. Z., (2007). The art of conversation is coordination. Psychological Science, 18(5).

Raca, M., & Dillenbourg, P. (2013, April). System for assessing classroom attention. In *Proceedings of the Third International Conference on Learning Analytics and Knowledge* (pp. 265-269). ACM.Santinelli, L., Morio, J., Dufour, G., & Jacquemart, D. (2014). On the sustainability of the extreme value theory for WCET estimation. In OASICS-OpenAccess Series in Informatics (Vol. 39). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.

Schneider, B., Abu-El-Haija, S., Reesman, J., & Pea, R. (2013). Toward collaboration sensing: applying network analysis techniques to collaborative eye-tracking data. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 107-111). ACM.

Schneider, B., Sharma, K., Cuendet, S., Zufferey, G., Dillenbourg, P., & Pea, R. (2016). Using mobile eye-trackers to unpack the perceptual benefits of a tangible user interface for collaborative learning. ACM Transactions on Computer-Human Interaction (TOCHI), 23(6), 39.

Sharma, K., Chavez-Demoulin, V., & Dillenbourg, P. (2016). An Application of Extreme Value Theory to Learning Analytics: Predicting Collaboration Outcome from Eye-tracking Data.

Sharma, K., Alavi, H. S., Jermann, P., & Dillenbourg, P. (2016). A gaze-based learning analytics model: in-video visual feedback to improve learner's attention in MOOCs. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 417-421). ACM.

Sharma, K., Jermann, P., Nüssli, M. A., & Dillenbourg, P. (2013). Understanding collaborative program comprehension: Interlacing gaze and dialogues. In Proceedings of Computer Supported Collaborative Learning (CSCL 2013) (Vol. 1, No. EPFL-CONF-184007, pp. 430-437).

Sharma, K., Caballero, D., Verma, H., Jermann, P., & Dillenbourg, P. (2015). Looking AT versus looking THROUGH: A dual eye-tracking study in MOOC context. International Society of the Learning Sciences, Inc.[ISLS].

Sharma, K., Olsen, J. K., Aleven, V., & Rummel, N. (2018). Exploring Causality Within Collaborative Problem Solving Using Eye-Tracking. In European Conference on Technology Enhanced Learning (pp. 412-426). Springer, Cham.

Smith, R. L. (1990). Extreme value theory. Handbook of applicable mathematics, 7, 437-471.

Van Rheden, V., Maurer, B., Smit, D., Murer, M., & Tscheligi, M. (2017). LaserViz: Shared gaze in the Co-located physical world. In Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction (pp. 191-196). ACM.
# Exploring Disciplinary Boundaries in Early Elementary Students' Developing Practices

Kylie Peppler, University of California Irvine, kpeppler@uci.edu

Joshua Danish, Indiana University, jdanish@indiana.edu

Naomi Thompson, Indiana University, naothomp@indiana.edu

**Abstract:** In CSCL we often refer to knowledge in terms of “practices” to highlight how knowledge is performed in context rather than abstracted from it (e.g., Greeno, 1998). One aspect of context is the notion of disciplines such as science, math, and the language arts (Sawyer, 2006). To date, however, research that explores the relationship between knowledge and disciplinary context has typically focused within a single disciplinary frame, such as science (e.g., Banks et al., 2007). This paper aims to explore the influence of disciplinary boundaries between science and language arts on students’ developing practices. We present data from a mixed-age first and second grade classroom where students collaboratively engaged with a book about honeybees from either a science or a language arts frame. Analyses of students’ individual written critiques reveal how they consistently adapted their criteria for what makes a good representation based upon the disciplinary framing.

## Introduction

Core to CSCL and the Learning Sciences is the belief that context matters (Sawyer, 2006) and that the social, cultural, and physical environment in which students learn new content fundamentally shapes what they have learned (Greeno, 2006). For this reason, sociocultural theorists often refer to knowledge in terms of the “practices” that students know how to engage in, to highlight the situated and interactive nature of knowledge, and how knowledge is performed in context rather than abstracted from context (Greeno, 1998, 2011; Lobato, 2012).

In an effort to understand the relationship between learning and context, researchers have focused on many aspects of learning environments including whether students are working individually or in groups (e.g., Webb et al., 2009) and whether students are learning in or out of school (Kafai, Peppler, & Chapman, 2009) to name a few. However, one aspect of context, which cuts across all of these other variables and remains crucial for supporting learning, is the notion of disciplines such as science, math, and the language arts (Sawyer, 2006). To date, however, research that explores the relationship between knowledge and disciplinary context has typically focused within a single disciplinary frame, such as science, even when it explores students’ developing practices within disparate social contexts such as the school and home (Banks et al., 2007). Furthermore, while there is an abundance of work examining STEM (Science, Technology, Engineering, and Math) disciplines, other disciplinary contexts such as the language arts are largely overlooked. The question then emerges, how do these disciplinary contexts influence students’ practices?

Therefore, this paper aims to explore the influence of disciplinary boundaries between science and language arts on students’ developing practices. We selected students early in their academic career, in first and second grade, to highlight and test whether students’ awareness of disciplinary boundaries develops early in the process of schooling. Students in elementary school classrooms typically remain in the same room and work with the same teacher as they transition between disciplines, contrasted with middle and high school students who frequently move to a new classroom and work with a different teacher for each discipline. One might then assume that disciplinary boundaries are less salient at this stage of schooling, although our experiences suggest that this is not the case. Further, educators typically assume that students can learn discipline-specific practices in a variety of disciplinary contexts, and that this is likely to promote more robust understanding of the links between disciplines. For example, many of our collaborating teachers have asked students to read about the current science topic during their literacy time, assuming that the students will learn the science content and the reading practices simultaneously. However, researchers have rarely studied whether this assumption is true. By exploring disciplinary boundaries at this stage of schooling, and within a context where disciplinary boundaries might be more subtle, we aim to demonstrate how salient these boundaries actually are for students, and how they impact student learning and knowing. Our core research question is: *To what extent does explicit disciplinary framing of an activity as science or language arts affect young children's classroom practices?*

We present data from a mixed-age first and second grade classroom where students were asked to evaluate a short book about honeybees before and after extensive language arts and science instruction. This activity was chosen because it required students to make their disciplinary values explicit, allowing us to contrast the influence of the two disciplinary contexts and the practices that students engage in within those contexts.
Students were randomly assigned to one of two conditions where the honeybee book was framed as either a "science" representation or as a "language arts" representation. Students then completed several activities to give them a range of opportunities to reveal their perspective on what makes a "good" or accurate representation within the specific disciplinary context. The students in each group then completed an identical 10-week science curriculum where they learned about how honeybees collect nectar (Danish, Peppler, Phelps, & Washington, 2011) as well as standard language arts instruction on grammar, spelling, and strategies for good writing/storytelling before again critiquing storybooks about bees. Close analyses of students' written critiques reveal how they consistently adapted their criteria for what makes a good representation based upon the disciplinary framing, often overlooking errors in one disciplinary condition that were treated as crucial in the other.

## Contrasting disciplinary practices using student representations

Our work is grounded in sociocultural theories of learning (Cole, 1996; Greeno, 2006, 2011). These theoretical frameworks share a common ancestry in Vygotsky (1978) and highlight the role of the social environment in learning and development. From this perspective, disciplinary knowledge includes not only specific facts but practices of engaging with content, people, and materials (Greeno, 2011), and even disciplinary ways of perceiving the world around us (Goodwin, 1994; Stevens & Hall, 1998). As such, the notion of "discipline" has emerged as crucial to structuring how we think about learning environments, and how we evaluate students' knowledge (Sawyer, 2006).

Thus, our goal in the current study is to examine students' participation within disciplinary activities in order to begin identifying the relationship between their shared and individual understandings of what the disciplinary context means, while also teasing out the role of the environment in shaping those perspectives. To ground our contrast we identified a common practice of both science and language arts—the critique, or evaluation of quality, of visual representations.

## Representations as a site for disciplinary comparison

Representations, such as drawings and narrative storylines, play a role across disciplines and contexts (diSessa, 2004; Lehrer & Schauble, 2000; Schwartz & Heiser, 2006) and yet serve a different purpose within each discipline. For example, a drawing of a flower might be used as an anatomical reference in a science classroom, and need to be accurate. In contrast, a drawing of a flower in language arts might instead be a starting point for a new story. Representations are also commonplace in early elementary classrooms because they make it easy to capture, share, and relate information in powerful ways (Schwartz & Heiser, 2006; Willats, 2005). Students are frequently asked to create or work with representations as part of their daily activities.

Much of the research into students' engagement with representations, including their critiques, has focused on either the cognitive benefits or abilities of individual students (Schwartz & Heiser, 2006), or a specific disciplinary context such as science or math (Lehrer & Schauble, 2005). However, an increasing body of research has drawn upon sociocultural theories of learning to move beyond individual cognition and explore the relationship between representations and the activities in which they are created, modified, and used (Hall, 1996; Roth, 1997; Roth & McGinn, 1998). These sociocultural studies accomplish this by examining observable patterns in student behavior (practices) as they develop over time, or the immediate influence of a specific activity upon students' actions. We blend these two approaches by comparing students' practices not only across two distinct contexts, but also as they change over time. Furthermore, despite an interest in context, prior sociocultural studies continue to focus largely on representations in math and science. Our present study aims to extend this pattern by contrasting students' creation of representations in a science context with a language arts context. Language arts was chosen because we hypothesized that it would be a distinct enough setting from science to make students' awareness of disciplinary contexts visible.

## Representational critique in science and language arts

Representations play an important and unique role in science as a discipline, allowing researchers to share theories, experiments, and evidence across vast distances and different time periods (Latour, 1987, 1988; Roth & McGinn, 1998). When working with representations, one important practice shared by multiple disciplines is the ability to critique and evaluate representations, both those that others have produced as well as one's own, in an effort to improve and refine it (diSessa, 2002). This practice of critique is also instrumental in providing students with an opportunity to develop and explore their own identity with respect to the representation and the discipline (Greeno, 2011). Effective critique of a representation requires an understanding of the purpose of the representation and the context of its use. Fortunately, research shows that young children (i.e., kindergarten through second grade) are capable of critiquing representations along a number of dimensions, including the
accuracy of their content (Danish & Enyedy, 2007; Danish & Phelps, 2010a). Critique can be particularly powerful as students create their own representations, allowing them to explore both the representational form and the science content referred to by the representation (Danish & Phelps, 2010b; Parnafes, 2010).

While there is significantly less existing research investigating the nature of representations within the context of language arts, we do know that language arts representations, especially storybooks, videos, and other media, play a critical role in the development of language arts skills. We looked to the Common Core standards for our initial criteria. To further identify language arts practices which are likely present when engaging with science content, we also focus on informational and technical writing skills which are relevant to all STEM fields and critical 21st century skills (Duke, 2010; Purcell-Gates, Duke, & Martineau, 2007). Building on the idea of informational writing, we then generated a grounded list of criteria that resulted in a coding scheme that emphasized technical writing, grammar (such as correct capitalization, verb form, etc.) and the construction of a clear story-arc (with a beginning, middle and end to the story), among other criteria further explored in the methods section below.

# Methodology

## Participants

The present study took place in a public Midwestern elementary school with 20 first and 20 second grade students (ages 6-9; 40 total) in a mixed-age classroom, with 37 of the students being present at the two time points in which data was collected and included in the present study. The majority of children were White (90%) with 17% of the students receiving free or reduced lunch. Students were randomly divided into two conditions: science (N = 19) and language arts (N = 18).

## Procedure

The activities in the two conditions (and at both time points) were identical except that the activities in one were framed by the teacher as “science” and in the other as “language arts.” The mixed-age classroom is co-taught by two teachers. One of these teachers led the science condition, and the other led the language arts condition. Steps were taken to ensure that as much as possible, the sequence and the methods of the two conditions were the same by scripting the directions and questions in advance of the conversation. Within each condition, the teacher followed a predetermined sequence for the 50 minute session: 1) generate definitions for what makes a good representation in the assigned disciplinary context (i.e., science or language arts); 2) record the groups’ thoughts on good representations on a whiteboard; 3) introduce a short storybook about a honeybee finding and collecting nectar; 4) ask students to write or draw on a copy of the book about what made it a good or bad representation 5) follow-up whole group discussion of individual observations and thoughts about what made the book either good or bad; 6) discuss a revised version of the book with many intentional errors removed to see if students felt it was “better” or not.

The research team created two storybooks for these activities (one for use at each time-point) representing a honeybee collecting nectar, one for use at each time point. In both cases, we intentionally violated the same number of scientific principles (e.g., the number of anatomically incorrect bees) and language arts principles (e.g., counts of missing or inaccurate punctuation) in an effort to provide approximately equivalent opportunities for students to engage in the practice of discipline-specific critique. We provided hand-drawn images and hand-written text for the students to feel more comfortable in the critique process (to appear more like a peer seeking feedback on an early draft rather than a published text). Data were collected as part of the larger BeeSign study which indicated that students in both conditions learned extensive and high-level science content over the course of the study Danish, Peppler, Phelps, & Washington, 2011; Peppler & Danish, 2012). The knowledge and understanding built during the classroom activities were largely group-based and collaborative. Here, we analyze individual measures to better understand the impact of the group and the environment on individual learning.

## Analysis

To summarize and contrast students’ critical practices in the different conditions, student utterances in whole-group discussion and their writings (including the drawings, text, or other marks on the photocopied storybook) were collected, transcribed, and coded to reflect the content of their critique as relating to science (e.g., accuracy, parsimony), language arts (e.g., grammar, storyline), or being non-domain-specific (e.g., aesthetic preference for color over black and white drawings). This paper reflects findings from students’ written critiques. Additionally, an average percent of time spent engaging science, language arts, and other was calculated per child to account for the wide variability in writing ability at early ages. One researcher coded all of students’ written comments,
with a second researcher coding a randomly selected 30% of all data to establish inter-rater reliability, resulting in high agreement (language arts kappa κ=.691, science κ =.898).

To explore the relationship between time and condition upon the mention of science or language arts content in the written critiques, a split-plot analysis of variance (ANOVA) was performed. Our hypothesis was that regardless of the disciplinary context framing students would pull from their science repertoire to create and talk about their storyboards due to their participation in the extensive science unit that targeted the same content represented in the honeybee books. A chi-square analysis was also completed to determine whether there was a significant difference in the proportion of each topic being mentioned across conditions and time-points.

## Results

Across both conditions, students generally provided more discipline-specific critiques in the post-condition than the pre. This is unsurprising and likely attributable to the time spent learning the respective disciplinary practices in the intervening weeks. Below we describe the shifts in written critique practices across conditions.

### Written critiques

Across both conditions, students' written critiques reflected an impact of both time point and disciplinary framing. Students were less likely than hypothesized to bring their practices of science critique into the language arts condition and vice-versa (see figure 1 and table 1). These results are discussed in further detail below.

<figure><img src="image_161.png" /><figcaption>Figure 1. Percentage of written representational critique elements coded as language arts (left) and science (right). Remaining percentage refers to items coded as non-specialized or other.</figcaption></figure>

<table><caption>Table 1: Mean number of written critique elements coded as language arts and science</caption><thead><tr><th rowspan="3">Condition</th><th rowspan="3">N</th><th colspan="4">Language Arts Codes</th><th colspan="4">Science Codes</th></tr><tr><th colspan="2">Pre</th><th colspan="2">Post</th><th colspan="2">Pre</th><th colspan="2">Post</th></tr><tr><th>M % (count)</th><th>SD</th><th>M % (count)</th><th>SD</th><th>M % (count)</th><th>SD</th><th>M % (count)</th><th>SD</th></tr></thead><tbody><tr><td>Language Arts</td><td>18</td><td>21.89% (3.83)</td><td>16.82%</td><td>47.98% (6.06)</td><td>23.45%</td><td>9.41% (1.05)</td><td>10.55%</td><td>8.08% (0.79)</td><td>10.08%</td></tr><tr><td>Science</td><td>19</td><td>9.49% (1.83)</td><td>8.76%</td><td>17.09% (2.72)</td><td>14.66%</td><td>23.79% (3.05)</td><td>18.75%</td><td>57.36% (6.11)</td><td>19.81%</td></tr></tbody></table>

### Language arts elements

Within the language arts group, there was an overall increase in the use of language arts elements in students' written critiques, from about 22% to about 48% of the students' written critiques. In the science group, there was also a (much smaller) increase in written critique of language arts elements, from about 9% to about 17%. (see both figure 1 and table 1). Although both groups saw their scores increase, the increase in the language arts group was proportionally more than three times the increase in the science group. This was a statistically significant interaction between time of measurement and context framing, F(1, 35) = 21.03, p < .001. The strength of this relationship, as assessed by partial η², was strong, with the interaction factor accounting for 37.5% of the variance in language arts elements.

Results also suggest that there was a main effect for both condition and time point such that the frequency of language arts elements including both pre and post was higher in the language arts condition compared to the science condition (32% vs. 13%) and that more language arts elements were present in the post-test than in the pre-test across both conditions (30% vs. 15%).
Taken together, these results suggest two key trends in students’ practices for written language arts critique. First, the disciplinary framing of the activity had a dramatic impact upon students’ practices, leading students to engage far more consistently on the language arts critiques during the language arts condition. Second, this effect appears to have been magnified over time. Thus the students in the science condition provided only marginally more critiques during the post-activity whereas the students in the language arts condition increased dramatically in the number of written language arts critiques they included. This increase can be attributed to continued engagement in the language arts curriculum during the intervention. Because all of the students experienced an identical classroom language arts curriculum over the 10-week period, we do not believe that these differences were due to chance but rather to the disciplinary framing of the activity. This highlights a key interaction between students’ developing practices and disciplinary framing given that students’ shift in practices related to language arts were much more evident in the language arts context.

As we turn to what this looked like for individual students in each of these conditions, Figure 2 illustrates the increase in language arts elements from the pre to post intervention written critiques. These examples are representative of the kinds of critiques the students generally provided and highlight the marked changes that occurred between the pre and post intervention in the language arts condition.

<figure><img src="image_162.png" /><figcaption>**Figure 2.** Sample language arts pre-intervention critique from Student 1 (left), sample language arts post-intervention critique from Student 1 (right).</figcaption></figure>

In Figure 2 (left), the participant in the Language Art pre-intervention focused on general features of the image. When prompted, “what makes this a good language arts representation?” the student gave a summary response about the bee leaving the hive and being excited to get nectar, stating that “the bee is leaving the hive and he is Excited [sic] to get nectar”. This was coded as “other.” When prompted, “what makes this a bad language arts representation?” the student focused largely on the picture rather than the words, and used both the written and visual edits to identify some of the intentional errors in the structural details of the drawing of the bee. In this example, Student 1 drew on the booklet, adding the missing legs to the bee and adding details to the wings which was coded as “science” as it added greater scientific fidelity to the bee. Notably, the student has ignored other intentional grammar violations on the page, such as the uppercase ‘B’ in bee and ‘L’ in leaves as well as other intentional science and language arts violations that were included in the storyboard, including other inaccurate body parts (such as the missing thorax of the bee).

When comparing Student 1’s critique in the post-test condition after the 10-week curriculum (see Figure 2, right), several changes are apparent. There are still some general or science critiques found in the storyboard, such as reference to the fact that, “almost all of the body parts are [shown],” (coded as structure detail important to science). For example, the student critiques the story arc of the drawing because there is, “nothing before the bee returns to the hive.” The student also makes clear reference to the punctuation on the page, noting that there is, “no punctuation mark where bee is talking.” The student did not, however, pick up on the intentional violation of grammar in the text on the page with an upper case ‘R’ in the word returns.

## Science elements

There was also a significant interaction between time of measurement and context framing in the prevalence of students’ written science critiques, $F(1, 35) = 13.60, p < .001$. The strength of the relationship, as assessed by partial η², was strong, with the interaction factor accounting for 27.9% of the variance in written science elements. There was a slight decrease in the mean percentage of science elements that each student included in their written critiques, from 9% to 8%, in the language arts context, and a significant increase in science elements in the science context, from an initial 23% to 57% of the written student critiques focusing on the critique of science elements (see table 1).
The frequency of science elements was also higher in the science context compared to the language arts condition (40% vs. 8% of the writings) and that more science elements were present in the post-test than in the pre-test (32% vs. 16%). This suggests that the context framing again had an impact in the type of criteria children picked to critique the storyboards, with students increasingly referring to science criteria as they learn them, particularly in the science context. Thus, the findings for students' practices of critiquing the science content of the representations appear to generally parallel those for language arts, reflecting an impact of both context and learning over time.

The case shown in figure 3 illustrates the findings from the science group before and after the 10-week curriculum. As with the language arts critique, this example was representative of the critiques of the classroom as a whole and particularly highlights the marked changes in the critique from the science condition from pre to post intervention.

<figure><img src="image_163.png" /><figcaption>**Figure 3.** Sample science pre-intervention critique for Student 2 (left), sample science post-intervention critique from Student 2 (right).</figcaption></figure>

In figure 3 (left), when the Student 2 was asked, “what makes this a good science representation?” the student response was simply a statement of personal interest (“Winter is my birthday”) and thus was coded for “other.” In response to, “what makes this a bad science representation?” the student makes reference to the fact that, “bees [die] in winter.” While this is not an entirely accurate understanding of how bees survive in the winter (as many survive within the hive), it demonstrated the value that the student placed on’ science fidelity that was an example of a science code.

In the post-intervention critique from the same Student 2, the student noted that this is good because it is a representation of the ‘waggle dance.’ This was coded as a science exemplar as the waggle dance was a central theme of the curriculum of how bees communicate. The student also showed marked improvement on recognizing missing science elements such as, “the bees do not have legs and one [does] not have [antennae]”, which were both coded as structure details important to science (see figure 3, right).

## Discussion

Our analyses suggest that students as young as first grade are well attuned to the particular disciplinary frame in which they work as they critique representations. In fact, they are quite likely to ignore major flaws in sample representations when evaluating it in a content domain in which those flaws are not central (e.g., students ignored grammatical errors when evaluating a representation framed as “scientific”). In short, our results indicate that disciplinary context plays a major role in shaping students’ engagement with representational products, and that students’ engagement in disciplinary content may at times be seen across the curriculum. While this paper focused on the students’ written critiques, separate analyses of the verbal discussions suggested this trend continues when the whole group is in conversation. Given the increasing interest in cross-curricular activities designed to help students make connections throughout their learning activities, this suggests important opportunities for reflecting upon the quality and consistency of students’ representational materials and activities. As the current piece does not dive into the interactional processes of learning, these findings also suggest the necessity of future research to more explicitly explore the impact of disciplinary framing upon student learning, particularly in these cross-curricular contexts. These collaborative micro-processes will continue to be of interest and importance to the CSCL community.

While these findings generally support the situative or sociocultural view that context matters in exploring cognition and learning, the details of our study offer several additional suggestions for researchers and practitioners. First, they point to some very real challenges inherent in cross-curricular activity designs. Increasingly, there is a push in early elementary education to enact interdisciplinary activities designed to help
students see the connections across domains. In our work, we have found, for example, that as soon as the teachers
know we are interested in teaching bees during “science time”, they identify books about bees for reading time,
and have even worked with the art teacher to have students draw bees in art class. And yet, our findings suggest
that these bridges are even harder to build than most educators likely anticipate. Given that the students in our
experiment focused so intently upon language arts criteria as they critiqued science books within the language-
arts-framed condition, it raises a question regarding what students might learn when reading about science topics
in a time that is clearly designated as language arts? An important next-step in this line of research will be to
examine how disciplinary framing of this nature influences student learning and other kinds of practices (e.g.,
creating representations rather than just critiquing them) to help address this question and effective practices to
help bridge these disciplinary boundaries that seem to be so salient at even young ages.

A second and related result of this work is that there is clearly value in attending much more closely to
how teachers frame activities as linked to a specific discipline, and what impact this has upon students. This is
particularly relevant in early elementary classrooms where students typically work with the same teacher across
content areas, as opposed to moving to new rooms and teachers as they frequently do in secondary school, and
thus the shift between disciplines is marked most clearly by the teacher. In our experiment the teacher clearly
framed the storybook activity as science or language arts, and then maintained this frame in follow-up questions.
What is less clear is which features of the teachers’ prompts the students found salient in determining which
practices to perform within the context. Future work which teases this out will be an important first step in
understanding how to help students recognize the disciplinary context in which they are operation in order to
promote discipline-specific practices.

References

Banks, J. A., Au, K. H., Ball, A. F., Bell, P., Gordon, E. W., Gutiérrez, K. D., . . . Mahiri, J. (2007). Learning in and out of school in diverse environments: Life-long, life-wide, life-deep: LIFE Center, University of Washington, Stanford University, and SRI International.

Cole, M. (1996). Cultural psychology: a once and future discipline. Cambridge, Mass.: Belknap Press of Harvard University Press.

Danish, J. A., & Enyedy, N. (2007). Negotiated Representational Mediators: How Young Children Decide What to Include in Their Science Representations. *Science Education, 91*(1), 1-35.

Danish, J. A., Peppler, K., Phelps, D., & Washington, D. (2011). Life in the Hive: Supporting Inquiry into Complexity Within the Zone of Proximal Development. *Journal of Science Education and Technology, 20*(5), 454-467. doi: 10.1007/s10956-011-9313-4.

Danish, J. A., & Phelps, D. (2010a). Kindergarten and First-Grade Students’ Representational Practices While Creating Storyboards of Honeybees Collecting Nectar. In K. Gomez, L. Lyons & J. Radinsky (Eds.), *Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010) - Volume 1, Full Papers* (pp. 420-427). Chicago IL: International Society of the Learning Sciences.

Danish, J. A., & Phelps, D. (2010b). Representational Practices by The Numbers: How Kindergarten and First-Grade Students Create, Evaluate, and Modify Their Science Representations. *International Journal of Science Education*. doi: 10.1080/09500693.2010.525798.

diSessa, A. A. (2002). Students' criteria for representational adequacy. In K. P. Gravemeijer, R. Lehrer, B. V. Oers & L. Verschaffel (Eds.), *Symbolizing, Modeling and Tool Use in Mathematics Education* (pp. 105-130).

diSessa, A. A. (2004). Meta-Representation: Native Competence and Targets for Instruction. Cognition and Instruction, 22(3), 293-331.

Duke, K. (2010, July 19-24). Teaching Reading and Writing of Procedural or How-to Text. Paper presented at the US Department of Education Reading Institute, Anaheim, CA.

Goodwin, C. (1994). Professional Vision. American Anthropologist, 96(3), 606-633.

Greeno, J. G. (1998). The Situativity of Knowing, Learning, and Research. American Psychologist.

Greeno, J. G. (2006). Learning in Activity. In R. K. Sawyer (Ed.), *The Cambridge handbook of the learning sciences* (pp. 79-96). New York: Cambridge Univ Press.

Greeno, J. G. (2011). A situative perspective on cognition and learning in interaction Theories of learning and studies of instructional practice (pp. 41-71): Springer.

Hall, R. (1996). Representation as Shared Activity: Situated Cognition and Dewey's Cartography of Experience. Journal of the Learning Sciences, 5(3), 209-238.
Kafai, Y.B., Peppler, K., & Chapman, R. (Eds.) (2009). *The computer clubhouse: Creativity and constructionism in youth communities*. New York, NY: Teachers College Press.
Latour, B. (1987). *Science in Action: How to Follow Scientists and Engineers through Society*. Cambridge, MA: Harvard University Press.

Latour, B. (1988). *Drawing Things Together*. In M. Lynch & S. Woolgar (Eds.), *Representation in Scientific Practice* (pp. 19-68). Cambridge MA: MIT Press.

Lehrer, R., & Schauble, L. (2000). Developing Model-Based Reasoning in Mathematics and Science. *Journal of Applied Developmental Psychology*, 21(1), 39-48.

Lehrer, R., & Schauble, L. (2005). Developing Modeling and Argument in the Elementary Grades. In T. A. Romberg, Carpenter, T. P., & Dremock, F. (Ed.), *Understanding mathemat:cs and science matters* (pp. 29-53). New Jersey: Lawrence Erlbaum.

Lobato, J. (2012). *The Actor-Oriented Transfer Perspective and Its Contributions to Educational Research and Practice*. Educational Psychologist, 47(3), 232-247. doi: 10.1080/00461520.2012.693353

Parnafes, O. (2010). Representational practices in the activity of student-generated representations (SGR) for promoting conceptual understanding. In K. Gomez, L. Lyons & J. Radinsky (Eds.), *Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010) - Volume 1, Full Papers* (pp. 301-308). Chicago IL: International Society of the Learning Sciences.

Peppler, K., & Danish, J. A. (2012). *E-Textiles for Educators: Participatory Simulations With E-Puppetry*. Paper presented at the The Annual Meeting of the American Educational Research Association, Vancouver, British Columbia.

Purcell-Gates, V., Duke, K., & Martineau, J. A. (2007). Learning to read and write genre-specific text: Roles of authentic experience and explicit teaching. *Reading Research Quarterly* (42), 8-45.

Roth, W.-M. (1997). *Graphing: Cognitive Ability or Practice? Science Education* (81), 91-106.

Roth, W.-M., & McGinn, M. K. (1998). Inscriptions: Toward a Theory of Representing as Social Practice. *Review of Educational Research*, 68(1), 35-59.

Sawyer, R. K. (2006). The Cambridge handbook of the learning sciences. New York: Cambridge Univ Press.

Schwartz, D. L., & Heiser, J. (2006). Spatial representations and imagery in learning. The Cambridge handbook of the learning sciences, 283-298.

Stevens, R., & Hall, R. (1998). Disciplined perception: Learning to see in technoscience. In M. Lampert & M. L. Blunk (Eds.), *Talking mathematics in school: Studies of teaching and learning* (pp. 107-149). Cambridge: Cambridge University Press.

Vygotsky, L. S. (1978). *Mind in society: the development of higher psychological processes*. Cambridge: Harvard University Press.

Webb, N. M., Franke, M. L., De, T., Chan, A. G., Freund, D., Shein, P., & Melkonian, D. K. (2009). ‘Explain to your partner’: teachers’ instructional practices and students’ dialogue in small groups. *Cambridge Journal of Education, 39*(1), 49-70.

Willats, J. (2005). Making sense of children's drawings. Mahwah, N.J.: L. Erlbaum Associates.

## Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant No. 1324047 awarded to Kylie Peppler, Joshua Danish, and Armin Moczek. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Thank you to Janis Watson, David Phelps, Justin Whiting, Alejandro Andrade, the teachers who worked with us on this project, and the students who made this work possible.
# Assessing Collaborative Problem Solving: What and How?

Haley W. C. Tsang, Seung Won Park, Laure L. Chen, and Nancy W. Y. Law
haleywct@connect.hku.hk, swonpark@hku.hk, cllaure@connect.hku.hk, nlaw@hku.hk
The University of Hong Kong

**Abstract:** With the increasing interests in and recognition of the importance of CPS competence as an educational outcome goal, there will be stronger demands for valid instruments and metrics for the measurement of Collaborative Problem Solving Skills (CPS) from policy makers to education practitioners and parents. The availability of valid instruments and norms for CPS achievement would also promote CSCL adoption within the wider education arena. This study explores two methods of assessing CPS: using the ATC21S online tasks and using qualitative discourse analysis of the same groups of students' transactions as they work together on an authentic learning task. By comparing the measurements arising from both methods, we raise issues and contradictions observed and suggest further research directions related to the measurement of CPS.

## Introduction

Collaborative problem solving (CPS) has been identified as an important competence critical for life and work in the 21st century in the education literature (Fiore et al., 2017) and in education policy documents (e.g. Singapore Ministry of Education (2009)). This is similar to what is stated in the P21 Framework for 21st Century Learning which is developed with participation from educators, business leaders and other stakeholders (Partnership for 21st Century Skills, 2009).

Collaborative learning (including CSCL) have gained increasing prominence as a field of study in education, and the general consensus on the importance of CPS skills, methods and tools for assessing CPS are limited. There are two major, popularly known CPS assessment frameworks and systems in education: the one developed by the ATC21S Consortium (Griffin & Care, 2015; Hesse, Care, Buder, Sassenberg, & Griffin, 2015), and the one used in the assessment of CPS in the Program for International Student Assessment (PISA) 2015 (OECD, 2017). Besides these two studies which have been administered to large populations of students in countries around the world, there are also smaller scale studies, such as those conducted by ETS in the US (Liu, Hao, von Davier, Kyllonen, & Zapata-Rivera, 2016) and researchers (Lin, Hsiao, Chu, Chang & Chien, 2015) in Taiwan. While these reported studies differ in terms of the content focus of the problems, the collaboration arrangements, and the technology platforms used, all of these assess CPS as an individual's attribute. On the other hand, studies in CSCL generally focus on the interactions and joint activities among group members and the performance of groups and communities. Koschmann (2001, p. 19) argue that the concern of CSCL is "with the unfolding process of meaning making ..., not so-called "learning outcomes""'. Stahl (2014) further elaborated that studies of collaborative learning focusing on the outcomes of individuals misses the most important aspect and educational potential of CSCL, that the group's outcome is not equal to the aggregate of the individual outcomes, and that the group is the appropriate unit of analysis for understanding collaborative learning. Despite the above differences between CPS assessment and CSCL, this paper will present a less traditional CPS assessment study with findings which may indicate the direction of future interplay between the two domains.

## Literature review

Hesse et al. (2015, p. 38) define CPS as "approaching a problem responsively by working together and exchanging ideas". CPS thus comprises two constructs, collaboration and problem solving, as exemplified in the framework described, which is grounded on research in the CSCL area. Collaborative problem solving is similar to individual problem solving, except that in CPS all the steps are directly observable as the group members need to communicate and negotiate their understanding during each step of the process. Hesse et al's (2015) framework is adopted by the ATC21S Consortium as the assessment framework for their CPS assessment system. The PISA 2015 assessment framework (OECD, 2017) is also grounded on the same theoretical underpinnings as elaborated in Hesse et al's (2015), but expressed in a 4x3 matrix, comprising four stages of problem solving (exploring and understanding, representing and formulating, planning and executing, and monitoring and reflecting) and three communication processes (establishing and maintaining shared understanding, taking appropriate action to solve the problem, and establishing and maintaining team organization). Since in this study, the ATC21S assessment system has been deployed as one of the methods to assess students' CPS skills, we will further elaborate on this below.
# The ATC21S assessment framework

The ATC21S assessment framework has a three-level hierarchy of component skills (Hesse et al., 2015), as shown in Table 1. At the top level, CPS comprises social and cognitive process skills. The former is further differentiated into participation skills, perspective-taking skills and social regulation skills, while the latter is differentiated into task regulation skills, and learning and knowledge building skills.

<table><caption>Table 1: The ATC21S CPS assessment framework (Hesse et al., 2015, p. 41-52)</caption><thead><tr><th>Level 1</th><th>Level 2</th><th>Level 3</th></tr></thead><tbody><tr><td rowspan="6">(1) Social process skills</td><td rowspan="3">(i) Participation skills</td><td>(a) Action</td></tr><tr><td>(b) Interaction</td></tr><tr><td>(c) Task completion/perseverance</td></tr><tr><td rowspan="2">(ii) Perspective taking skills</td><td>(a) Adaptive responsiveness</td></tr><tr><td>(b) Audience awareness (mutual modelling)</td></tr><tr><td rowspan="4">(iii) Social regulation skills</td><td>(a) Negotiation</td></tr><tr><td>(b) Self-evaluation (Metamemory)</td></tr><tr><td>(c) Transactive memory</td></tr><tr><td>(d) Responsibility initiative</td></tr><tr><td rowspan="10">(2) Cognitive process skills</td><td rowspan="5">(i) Task regulation skills</td><td>(a) Problem analysis</td></tr><tr><td>(b) Goal setting</td></tr><tr><td>(c) Resource management</td></tr><tr><td>(d) Flexibility and ambiguity</td></tr><tr><td>(e) Information collection</td></tr><tr><td rowspan="3">(ii) Learning and knowledge building skills</td><td>(f) Systematicity</td></tr><tr><td>(a) Relationships (Representations and formulations)</td></tr><tr><td>(b) Rules: "If...then"</td></tr><tr><td></td><td>(c) Hypothesis "what if..." (Reflection and monitoring)</td></tr></tbody></table>

At the third or lowest level, participation skills include: (a) action – skills in performing the actions needed for the solving process, (b) interaction – skills of carrying out interactions with others in a chain of alternating actions (upon others) and in response to others' actions, and (c) task completion/perseverance – the ability to persist in the task performance until the task completion.

Perspective taking skills include: (a) adaptive responsiveness – skills of responding to others’ views or contributions and incorporating them into one’s own thoughts and actions, and (b) audience awareness (mutual modelling) – skills of adapting one’s own views, efforts or accomplishments from others’ (or audiences’) perspectives when presenting them so that they are more easily understood or appreciated.

Social regulation skills include: (a) negotiation – resolving differences in work procedures, views, targets, priorities, potential solutions, etc. through compromise, consensus or other means; (b) self-evaluation (metamemory skill) – knowing one’s position, knowledge, strengths, weaknesses and other attributes; (c) transactive memory skills – knowing other collaborators in the same terms as in knowing oneself in metamemory skills; and (d) responsibility initiative – taking initiatives and responsibilities at different stages of the problem-solving process to achieve high-quality solutions through the joint efforts of the group.

Task regulation skills include: (a) problem analysis – dividing a problem into a set of more manageable tasks for solving the problem; (b) goal setting – setting clear, specific and achievable goals and sub-goals to direct and motivate the group’s problem solving efforts, (c) resource management – obtaining, allocating, utilizing and assessing resources based on the available pool of human resources (group members’ knowledge, skills and special capabilities) and other tangible resources such as equipment, tools and software applications, (d) flexibility and ambiguity – broadening the range of solution methods through problem representation, re-organization or new strategies, and determining acceptable tolerance of ambiguity and treating them as opportunities to further explore the problem space for alternative steps or strategies; (e) information collection – identifying and collecting information to solve a problem to meet the needs in terms of relevance, scope, detail, and time availability; and (f) systematicity – using planful, methodical approaches such as means-ends analysis and adopting results monitoring and reflection

Learning and knowledge building skills include: (a) representation and formulations – identifying relationships between elements in the problem space, and making representations and formulations of the inner connections among elements of tasks, (b) rules – ability to recognize contingencies and causal relationships as in making “if...then” rules, and (c) hypothesis – setting hypotheses by generalizing observations or parts of the solution and testing them through “what if...” inquiries, followed by monitoring and reflection of actions and
strategies.

## Design of CPS assessment systems

All CPS assessments systems are computer-based, and these can be generalized into two approaches. In the first approach, students work online on a collaborative task, communicating through a chat box, as exemplified by the ATC21S system (Awwal, Griffin, & Scalise, 2015). The CPS system licensed by the University of Melbourne for use by the research team only supports dyads (as randomly- or pre-assigned pairs) collaborating on a bundle of about three to four tasks, which are to be completed within an hour. The system is designed on the basis of a jigsaw model of collaboration such that any one person would not have enough information to complete the task without help from the collaborator. Process and click stream data generated during the task interactions are recorded in time-stamped session logs, and the chat box interactions are recorded in the chat logs. A sophisticated system of coding, mapping, and scoring was developed for each of the indicators based on the CPS assessment framework (Adams et al., 2015). The system can generate reports on individual and class (i.e. everyone who participated in the assessment in the same session) performance.

In the second approach, the PISA 2015 CPS assessment, items were constructed according to the assessment matrix such that each item can be classified as targeting one of the 12 problem-solving-task-focus and collaboration-focus combinations. Students collaborate with different numbers of computer agents and playing different roles in the interactions, according to the specific collaboration skills required (OECD, 2017). The extent to which other team members are able to collaborate can then be precisely controlled in this system. Student performance is assessed through the specific responses they select from the choices offered.

Both the ATC21S and PISA 2015 assessed CPS as a generic skill, and subject matter knowledge of the student in relation to the problem contexts is assumed to not affect the students' assessed CPS performance. A team from the US Educational Testing Service (ETS) studied students' CPS skills in the context of knowledge restructuring and revision in the study of science topics (Liu et al., 2016). They adopted the PISA 2015 CPS assessment framework in the design of their system, and conducted two studies, the first with individual students working with two computer agents, and the second with pairs of students working with two computer agents. They found that students make bigger learning gains as measured by the accuracy of their responses when they were collaborating with a human peer. However, students' CPS outcomes were not reported.

## Nature of problem and roles of team members

In the CSCL literature, knowledge building (Scardamalia, 2002) is often the most valued targeted learning outcome in the process. Thus, open-ended authentic problems closely related or meaningful to students' everyday life and requiring some sustained engagement are often selected as the targeted context for collaborative learning. This is very different from the problems used in the ATC21S, PISA 2015 or the ETS assessment systems.

Another challenge that CPS assessment systems have to confront is the roles taken up by different members in the team, beyond those captured in the collaboration aspects of the assessment framework. One particular area of concern is the role of leadership in collaborative teams, which is an important construct in the management and organizational psychology literature (Mercier, Higgins, & Da Costa, 2014). Leadership refers to the skill of taking responsibilities as well as initiatives in activities of a group to direct the group to achieve its goals (Miller, Sun, Wu, & Anderson, 2013). Leadership roles can be a result of being elected (by peers) or assigned (e.g. by teachers), but can also be emergent, that is, as a dynamic property of the situation, resulting from the social and task interactions (Gressick & Derry, 2010).

Mercier et al. (2014) further differentiated emergent leadership into two types based on the nature of the moves made by the participants, which can be focusing on organizing the group, or on advancing the intellectual aspects of the group effort. They define three types of organizational leadership moves: turn management, planning and organizing, and acknowledgement; and two types of intellectual leadership moves: idea management and development, and topic control. Mercier et al (2014) further defined emergent leadership as instances when a participant's leadership moves were taken up by others in the group. Their study included findings that have important implications for how to design CPS assessment. First, they found that the extent to which students are more willing to work on ideas from other students is affected by the collaborative medium – being greater in electronic medium like a multi-touch tablet than the paper medium, which indicates that the former is more supportive of complex conversations. Second, they also found that more than one person exhibiting emergent leadership can occur in a group. Third, by studying emergent leadership when the same groups of students engaged in tasks associated with two different subject areas, math and history, they found that the emergence of leadership differed over different task types.

## Study purpose and design
The study reported here is part of a bigger study that aims to develop an online role play serious games system that would support the learning and assessment of CPS for children aged 11 to 15. A key challenge in designing such a system is to develop valid measures for CPS assessment. In this study, our goal is to compare the CPS achievement of students as assessed by the ATC21S CPS system with a hand-coded CPS assessment based on video-recorded transactions of students working in groups over the space of two weeks to design an online game to promote cyberwellness. Their collaborative interactions were videotaped twice, with a week's separation in between. The specific research questions addressed in this study are as follows:

1. Is the ATC21S assessment framework appropriate for coding the face-to-face transactions of the students participating in the study?

2. How do students' CPS behavior and performance change over time, if at all?

3. Do the students' CPS achievement as measured by the ATC21S assessment system correlate with the hand-coded CPS scores?

4. What role can ATC21S CPS assessment scores play in the further development of CPS systems?

## Settings and participants

The study was conducted in the context of a three-session summer course for primary and secondary school students on game design to promote cyberwellness. The game design course was offered to support students interested in participating in a Minecraft coding competition. Interested students registered through the online portal of the competition organizer, which did not set any selection mechanism or criteria on the applications for participation. The summer course was held as 2-hour workshop sessions on three consecutive Friday mornings. Students were grouped into teams of three to five students of approximately the same age. The goal of the course was for the students to develop jointly a game (storyboard) that would be both interesting and help others to learn about cyberbullying and how to handle them. The students were encouraged to build their games in Minecraft, but that was not a necessary part of the course activities. A brief rundown for the three workshop sessions is as follows:

Workshop 1: Welcome, introduction to the goals of the course, introduction to examples of and ways to handle cyberbullying, group assignment, and working in groups to discuss the goals of the game they wish to construct.

Workshop 2: Introduction to game design and development, including showing examples of storyboards and Minecraft games, and group work on game design: game goal, rules, and storyboard.

Workshop 3: Group preparation on presentation of preliminary game design to other groups for peer review and feedback, further refinement and submission of final game design storyboard, and group interviews.

A total of 44 students were admitted to the workshops on a first-come-first served basis without any selection criteria. According to the University of Melbourne Assessment Research Centre, the CPS tasks have only been trialed and validated for students aged between 11 and 15, and are deemed to be too difficult for younger students. Of the 44 course participants, only 29 were aged 11 or above. They were invited to arrive at the course venue an hour before the start of Workshop 2 to take the ATC21S CPS assessment. Of these, only 14 took the ATC21S CPS assessment and gave consent to the research team to use all of the artefacts they produced during the workshops as well as all audio and video recordings for research purposes.

The group work sessions in Workshops 2 and 3 were video-recorded. Workshop 1 was not videoed as the time period available for the group work was too short. The total time of the video recording for each group was around 55 minutes in Workshop 2, and 65 minutes in Workshop 3. For the purpose of this paper, we report on one of the groups with children aged 11 or above. All four members in this group attended the three workshops and took the ATC21S CPS assessment, and the video recordings from both workshop sessions were clear.

## Results

The background of this group of students and their CPS scores in the ATC21S assessment are listed in Table 2.

<table><caption>Table 2: The students' demographic information and CPS achievement as assessed by the ATC21S CPS system</caption><thead><tr><td>Student ID</td><td>Demographic information</td><td>Cognitive score</td><td>Cognitive level</td><td>Social score</td><td>Social level</td></tr></thead><tbody><tr><td>GD28</td><td>Female, 14 year old, 8<sup>th</sup> grade</td><td>-0.46</td><td>3</td><td>0.85</td><td>5</td></tr><tr><td>GD29</td><td>Female, 11 year old, 6<sup>th</sup> grade</td><td>-0.33</td><td>3</td><td>0.94</td><td>5</td></tr></tbody></table>
<table><tr><td>GD30</td><td>Male, 12 year old, 6<sup>th</sup> grade</td><td>-0.45</td><td>3</td><td>0.89</td><td>5</td></tr><tr><td>GD31</td><td>Male, 12 year old, 6<sup>th</sup> grade</td><td>-0.19</td><td>3</td><td>1.84</td><td>6</td></tr></table>

Based on the ATC21S CPS assessment framework presented in Table 1 above, two of the research team members worked in parallel and then together to develop a detailed coding scheme for coding the video transcript. It is found that other than the off-task communications, all of the verbal transactions can be coded using the assessment framework. Other than the verbal transactions, the transcript also records instances where a member *starts* engaging in a task. These observation statements are coded under participation skills-action. An inter-rater reliability analysis on a sample of the dataset (25%) is performed to determine if there is agreement between the two independent coders. The Cohen's kappa (κ = .82, p < .001) obtained is considered to be "very good" (Alman, 1999).

As the present study is exploratory in nature, we decide to use a simple metric for our computation of CPS: that of frequency counts of turns made by each of the participants under the social and cognitive process skills categories. However, it is observed that while most of the discourse turns can be classified as instances of positive behavior in the subcategories, some of the turns are negative instances. For example, utterances such as "this is too difficult" or "I didn't say I want to do it, I am just saying ..." are coded as 'avoids undertaking the work' and is given a frequency count of "-1" to indicate that this is a negative instance of the code. Turns that are examples of ignoring other's ideas or arguing without giving reasons are considered negative instances of social regulation.

<table><caption>Table 3: The CPS scores computed based on the number of turns made by each student in the two workshops</caption><thead><tr><th rowspan="2">Student ID</th><th colspan="4">Workshop 2</th><th colspan="4">Workshop 3</th><th colspan="4">Overall</th></tr><tr><th>GD 28</th><th>GD 29</th><th>GD 30</th><th>GD 31</th><th>GD 28</th><th>GD 29</th><th>GD 30</th><th>GD 31</th><th>GD 28</th><th>GD 29</th><th>GD 30</th><th>GD 31</th></tr></thead><tbody><tr><td>Social (+ve)</td><td>39</td><td>34</td><td>37</td><td>24</td><td>35</td><td>26</td><td>38</td><td>41</td><td>74</td><td>60</td><td>75</td><td>65</td></tr><tr><td>Cognitive (+ve)</td><td>22</td><td>9</td><td>10</td><td>9</td><td>8</td><td>9</td><td>16</td><td>11</td><td>30</td><td>18</td><td>26</td><td>20</td></tr><tr><td>Social (-ve)</td><td>7</td><td>12</td><td>5</td><td>7</td><td>3</td><td>6</td><td>5</td><td>9</td><td>10</td><td>18</td><td>10</td><td>16</td></tr><tr><td>Cognitive (-ve)</td><td>2</td><td>3</td><td>5</td><td>0</td><td>3</td><td>0</td><td>7</td><td>0</td><td>5</td><td>3</td><td>12</td><td>0</td></tr><tr><td><strong>Social score</strong></td><td><strong>32</strong></td><td><strong>22</strong></td><td><strong>32</strong></td><td><strong>17</strong></td><td><strong>32</strong></td><td><strong>20</strong></td><td><strong>33</strong></td><td><strong>32</strong></td><td><strong>64</strong></td><td><strong>42</strong></td><td><strong>65</strong></td><td><strong>49</strong></td></tr><tr><td><strong>Cognitive score</strong></td><td><strong>20</strong></td><td><strong>6</strong></td><td><strong>5</strong></td><td><strong>9</strong></td><td><strong>5</strong></td><td><strong>9</strong></td><td><strong>9</strong></td><td><strong>11</strong></td><td><strong>25</strong></td><td><strong>15</strong></td><td><strong>14</strong></td><td><strong>20</strong></td></tr></tbody></table>

The scores in Table 3 show a significant change in the scores associated with some of the students over the two workshops, both in terms of absolute numbers and relative to each other. The scores for GD29 and GD30 were relatively stable across the two workshops, and both increased in their cognitive scores. However, it is notable that GD29 has greatly reduced the frequencies of negative contributions in both social (from 12 to 6) and cognitive (from 3 to 0) dimensions. For GD28, while she maintained a high social score between the two workshops, her cognitive scores appeared to have been greatly reduced. On the other hand, in Workshop 3, GD31 increased significantly in his social score to rank the same with GD28 and GD30, and his cognitive score also rose to be the top score of the team. If we look at the overall scores over the two workshops, then GD28 and GD30 has the highest social score while GD28 has the highest cognitive score.

Comparing the scores in Tables 2 and 3, there is consistency in that the social scores of the students were much higher than their cognitive score. In fact, according to the ATC21S scoring rubric, 6 is the highest level of performance that a student can achieve. However, according to the scores in Table 2, GD31 has the highest score in both the social and cognitive dimensions while GD28 has the lowest scores in both the social and cognitive dimensions. How can we interpret these scores? Do they tell us anything about the students' CPS ability at all?

## Student's perception of their own collaboration process

Towards the end of Workshop 3, a focus group interview was held with each of the participating groups. The following were the interview questions regarding how the team worked together to develop the game:

1. How did the team come up with the game ideas presented in their storyboard?

2. How did the team go about their division of labor for the project?

3. Did the team have to look for information to complete their project, and if so, what and how?

4. Did any contradictory views arise during the process, and if so, how were these resolved?

5. Did the team encounter any problems or challenges during the work process?

6. If you are given a second chance to create a game, will you do things differently?

While these questions were designed to find out the members' perceptions about various aspects of the
collaboration process, a surprising finding was that the role of GD28 as a leader was talked about by all the other students throughout the interview. The following is an excerpt from the part of the interview about division of labor:

GD30: We (GD29, GD31, GD30) argued when deciding whom to be the presenter.

GD28: Exactly. Every time I was forced to present.

...

GD30: Then, we would tell her (GD28) the key points and she will be the one who present them. That's all.

GD31: As an analogy, we (GD29, GD31, GD30) are like the sheep while you (GD28) are the shepherd.

Again, when asked how they would work differently if they were to create a game again (I stands for the Interviewer):

GD28: I will run away.

GD29: The first thing I will do is...

GD31: First, I will get the team leader back. Then...

GD29: First I will get the team leader back and lock her up. Then...

I: That means you all want to listen to her instructions.

GD30: Yes.

It is clear from these exchanges that GD29, GD30 and GD31 unanimously consider GD28 as their leader in the project. According to GD30, GD28 played two roles—resolving conflict and representing the team:

GD30: The three of us often had different opinions. We would nominate our team leader whenever there was a presentation.

## Observation of the students' collaboration process

While rigorous coding of the students' turn-by-turn transactions provide us with a theoretically grounded view of the collaboration process, it is important that we do have a qualitative, holistic sense of how the students actually worked and interacted during the two workshop sessions. It should be noted that of the four students, GD30 was the only one who did not know how to program in Minecraft. Below is a brief summary of the key episodes: *Workshop 2* (Students were asked to work on their game design following the distributed worksheets)

1. Deciding on the group name: GD28 led the discussion. GD28 and GD30 constructed one using GD31's name, amid protests from GD31.

2. Discussing game ideas. GD28 suggested popping up questions for players to answer in Minecraft as the game mechanics. GD31 was more interested in playing around with Minecraft than completing the worksheets. GD30 requested GD31 to put aside the laptop to focus on the worksheet. GD28 then prompted the group for more game interaction ideas. GD30 put forward his ideas after hearing examples from GD28.

3. Working on storyboard—division of labor. GD29 asked what she should draw. After GD28 made her suggestion, GD29 expressed her desire to work in Minecraft instead. With help from GD30, GD29 was finally persuaded to work on the drawing task. GD30 then assigned a task to GD31 without giving reason, thus starting a conflict with GD31.

4. Discussing the detailed design of the game quests. GD28 initiated the discussion. GD29 and GD31 participated actively, but GD30 interrupted to say that this should be left to GD28 as the decision maker. GD28 paid close attention to GD29 and GD31's ideas. GD30 then joined in to give his ideas.

5. Starting to draw the storyboard. GD28 gave suggestions to GD29 and GD31 on how to build the game in Minecraft. GD30 did not take up specific jobs but assigned jobs to GD29 and GD31.

*Workshop 3* (Students were asked to give a short presentation on their game design as completed in Workshop 2 for peer feedback, and were given some time to prepare for it. After this, students were asked to work further and complete everything for a final presentation.)

1. Discussing who should be the presenter. GD29 and GD31 were playing on a smartphone. GD30 urged them to focus on the group discussion.

2. Discussing the group name. GD31 wanted to change, but no one else agreed.

3. Discussing the detailed design of the game quests. GD29 initiated the discussion and put forward her idea. GD30 built on her ideas and made further suggestions. GD31 disagreed with GD30 and both argued
strongly. GD28 stepped in to make other suggestions. A final agreement was reached.

4. *Creating the game in Minecraft.* GD31 expressed doubts about GD29’s competence in Minecraft, ignored GD29’s suggestions, turned the computer towards himself, and said that GD29 was stupid and was doing useless work. GD30 told GD31 that he was very impolite. GD29 and GD31 then entered into an intense argument over different approaches to work in Minecraft.

5. *Preparing for the first presentation.* GD28 took up the presenter role as she was considered the leader by the others. she practiced for the group presentation. GD30 asked how the game would handle the case if the player fail to achieve the game goal. GD28 and GD30 discussed the game refinement.

6. *First presentation.* Only GD28 came out to presented on behalf of the group. Not much feedback.

7. *Continuing the storyboard work.* GD28 assigned tasks to GD29 and GD31, GD30 worked with GD28 to help the other two students to complete the assigned tasks. GD29 and GD31 continued to argue, GD30 helped to clarify what needed to be done, and also helped GD31 with the task completion.

8. *Preparing for the group presentation.* GD31 finished creating part of the game in Minecraft. GD29 expressed delight and appreciation of the work by GD31. GD30 suggested to present not only the required paper-based storyboard, but also the partially completed game in Minecraft. GD31 referred to himself as the technical support for the group in building the game in Minecraft.

9. *Finalizing the game storyboards.* GD28 worked on finishing the writing part of the game storyboards. GD29 and GD30 worked together to stick pieces of the game scene paper drawings on the big cardboard for presentation.

10. Final presentation. GD28 was the main presenter, accompanied by the entire team. The Minecraft game was presented.

It can be seen that in Workshop 3, GD28 talked much less as she was focused on preparing for the two presentations. During this time GD30 was helping as the person to regulate the group and to resolve conflicts.

## Discussion and conclusion

We begin our discussion by addressing the four research questions of this study. First, the ATC21S assessment framework was operationalized in the hand-coding of the CPS interactions, and found to be adequate for analyzing CPS in the face-to-face context. Second, as revealed by both the CPS scores and the observations of the collaboration process, the CPS behavior and performance of individual students and the team as a whole changed over time. Third, no conclusive correlation is observed between the ATC21S assessment scores and the hand-coded scores, whether for individual workshops or aggregated over both workshops. The fourth question about how the ATC21S assessment scores may contribute to future development of CPS assessment systems is more complex, and is discussed below in terms of the issues we have uncovered in this study.

As shown in our study, even when the same assessment framework is used for the assessment, the exhibited CPS behavior (on which the score is computed) is a fluid measure, dependent on the problem context and the stage of the problem-solving task progression. GD28 may seem to have declined in her leadership role in Workshop 3 based on the scores. She had in fact been engaging much more in the completion of the worksheet task. As the group settled into a more task focused mode of operation, she did not have to spend as much time or effort on regulating the group’s activities. Moreover, we can see that GD30 had been assisting her in the coordination of the project. GD30 might have been able to engage more in the task completion part of the teamwork if he had knowledge of programming in Minecraft, and may not be doing so much task and social regulation work. The particular way a student engages in a task is always dependent on the nature of the task, the familiarity of the team members with the content/disciplinary knowledge and skills required in solving the problem, their familiarity with each other, and the stage of the problem solving progress. Is it meaningful to assess the CPS skills of individuals? What might be meaningful measures of a team’s CPS skills? What contextual information might need to be specified for meaningful interpretation and use of CPS measures?

From our observations of the team’s work progress, the conflicts arose mainly from the need to agree on what to enter on the worksheet and how to create the game in Minecraft. Through the interactions and conflict resolution process, the students improved in their teamwork to become more productive in their collaboration. However, the scores of the team members in Workshops 2 and 3 are not able to reveal the fine grain differences and development. These scores, similar to those from the ATC21S and PISA 2015, focus only on the CPS process but not the outcomes. The primary reason for valuing CPS as a core competence is the assumption that those with higher CPS capability will delivered better, higher quality collaboration outcomes. Is it reasonable to measure CPS capability, whether as an individual or group attribute, without giving any regard to the quality of the products of the collaboration? If CPS outcome were to be assessed, how should this be done?
The four students' CPS scores have been computed by the ATC21S system, which has been developed on the basis of an enormous amount of data and shown to be reliable for the set of assessment tasks in the system. However, there is as yet no reported studies on the construct validity of these measures, i.e. how these CPS scores might inform us on the assessee's CPS ability in authentic collaboration and task settings. What might be educationally meaningful ways of using CPS assessment scores?

Our study shows that there is a strong need for more studies on assessing CPS. So far, efforts to assess CPS have been led by researchers in the psychometric and assessment communities, focusing on CPS as an individual attribute. Many of the researchers in the CSCL community see CPS as a group attribute, and outcomes are often reported within the specific study context, but not in terms of a generic individual or group attribute. We hope the issues we have uncovered will stimulate interactions between the CSCL and assessment communities.

## References

Adams, R., Vista, A., Scoular, C., Awwal, N., Griffin, P., & Care, E. (2015). Automatic coding procedures for collaborative problem solving. In P. Griffin & E. Care (Eds.), *Assessment and teaching of 21st century skills* (pp. 115-132). Dordrecht: Springer.

Altman, D. G. (1999). *Practical statistics for medical research*. New York: Chapman & Hall/CRC Press.

Awwal, N., Griffin, P., & Scalise, S. (2015). Platforms for delivery of collaborative tasks. In P. Griffin & E. Care (Eds.), *Assessment and teaching of 21st century skills* (pp. 105-113). Dordrecht: Springer.

Fiore, S. M., Graesser, A., Greiff, S., Griffin, P., Gong, B., Kyllonen, P., ... Rothman, R. (2017). *Collaborative problem solving: considerations for the National Assessment of Educational Progress*. Washington DC: National Center for Education Statistics (NCES).

Gressick, J., & Derry, S. J. (2010). Distributed leadership in online groups. *International Journal of Computer-Supported Collaborative Learning, 5*(2), 211-236.

Griffin, P., & Care, E. (2015). *Assessment and teaching of 21st century skills: Methods and approach*. Dordrecht: Springer.

Hesse, F., Care, E., Buder, J., Sassenberg, K., & Griffin, P. (2015). A framework for teachable collaborative problem solving skills. In P. Griffin & E. Care (Eds.), *Assessment and teaching of 21st century skills* (pp. 37-56). Dordrecht: Springer.

Koschmann, T. (2001, December). Revisiting the paradigms of instructional technology. In *Meeting at the crossroads. Proceedings of the 18th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education* (15-22).

Lin, K. Y., Yu, K. C., Hsiao, H. S., Chu, Y. H., Chang, Y. S., & Chien, Y. H. (2015). Design of an assessment system for collaborative problem solving in STEM education. *Journal of Computers in Education, 2*(3), 301-322.

Liu, L., Hao, J., von Davier, A. A., Kyllonen, P., & Zapata-Rivera, J. D. (2016). A tough nut to crack: Measuring collaborative problem solving. In Y. Rosen, S. Ferrara & M. Mosharraf (Eds.), *Handbook of research on technology tools for real-world skill development* (pp. 344-359). IGI Global.

Mercier, E. M., Higgins, S. E., & Da Costa, L. (2014). Different leaders: Emergent organizational and intellectual leadership in children's collaborative learning groups. *International Journal of Computer-Supported Collaborative Learning, 9*(4), 397-432.

Miller, B., Sun, J., Wu, X., & Anderson, R. C. (2013). Child leaders in collaborative groups. In C. E. Hmelo-Silver, C. A. Chinn, C. Chan & A. M. O'Donnell (Eds.), *The international handbook of collaborative learning* (pp. 268-279). Routledge.

OECD. (2017). *PISA 2015 Results collaborative problem-solving*. Paris: OECD.

Partnership for 21st Century Skills. (2009). P21 framework definitions. Retrieved from http://www.p21.org/storage/documents/P21_Framework_Definitions.pdf

Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith (Eds.), *Liberal Education in a Knowledge Society* (pp. 67-98). Chicago: Open Court.

Singapore Ministry of Education. (2009). Third masterplan for IT in education (2009-2014). Retrieved from https://ictconnection.moe.edu.sg/masterplan-4/our-ict-journey/masterplan-3

Stahl, G. (2014). The group as paradigmatic unit of analysis: The contested relationship of CSCL to the learning sciences. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.644.8856&rep=rep1&type=pdf

## Acknowledgements

The authors wish to acknowledge that this work is funded by the Research Grants Council of the HKSAR Government, #T44-707/16/N, under the Theme Based Research Scheme.
# Metaphorical Reasoning Together: Embodied Conceptualization in a Community of Philosophical Inquiry

Sandra Lagrange-Lanaspre, LiDiLEM, Université Grenoble Alpes, sandra.lagrange-lanaspre@laposte.net

**Abstract:** In this paper, we show that taking an embodied stance at cognitive moves as not limited to speech acts but including concrete sensori-motor actions, either isolated or coverbal gestures, enriches our understanding of collaborative reasoning. More specifically, we follow the bimodal co-construction and circulation of metaphors in a Community of Philosophical Inquiry (Lipman, 2003). We show how the participants used many source concepts in the exploration of the abstract concept of ‘thinking’, and were able to build a *conceptual synthetizing metaphor*. Our results support a bimodal metaphors salience hypothesis: metaphorical gestures play a great role in making reasoning visible to others and therefore fostering collaborative learning.

## Introduction

According to Lipman, the founder of Philosophy for Children, a Community of Philosophical Inquiry (CPI), as a pedagogical practice, is supposed to favor deep collective reasoning and better cognitive understanding of a complex issue addressed through critical, creative, and caring thinking (Lipman, 2003). Its aims at teaching students how to build on opinion and to reason collectively about a philosophical question. In a CPI, students are encouraged to listen to each other with respect, to elaborate on each others’ ideas, to assist each other in justifying opinions and drawing conclusions. In Lipman’s words, it consists in “attempt[ing] to follow the inquiry where it leads rather than be[ing] penned in by the boundary lines of existing disciplines” (2003, p. 20).

But, even if the face-to-face interactions in a CPI involve multimodal communication, studies have so far mostly focused on students’ individual speech developed in such setting or on the role of the moderator described through series of verbal actions, as if learning would only take place in the brain (e. g. Simon & Tozzi, 2017). Clearly stepping away from traditional individualistic cognitivism (Menary, 2010a), adopting an embodied view on cognition offers avenues for renewing research on collective reasoning and collaborative learning. Such theoretical approach has methodological implications that guided our empirical work on a videotaped CPI among eleven 12-to-14-year-old students who are discussing about “thinking”. This study shows that taking cognitive moves as not limited to speech acts but including concrete sensori-motor actions enriches our understanding of conceptualization. More specifically, we follow the bimodal co-construction and circulation of metaphors in the CPI as activating concrete sensori-motor experience to apprehend abstract concepts collaboratively.

## Theoretical background

Our approach to metaphor is based on a strong embodied view on cognition. As it occurs in a CPI, metaphor also takes part to a more global cognitive dynamic of collective reasoning instantiating how learning is socially grounded.

## 4E cognition and the embodied approach to metaphor

Following the cognitive approach to metaphor, we consider the metaphorical process as the fundamental embodied basis of the conceptual system that makes it possible to think-and-act in our environment (Lakoff & Johnson, 1980). In this perspective, metaphor consists in building and apprehending complex abstract concepts (here after named ‘target concepts’) by comparison with familiar concrete concepts bodily experienced and transmitted (here after named ‘source concepts’). Metaphor expression exploits several modes of communication, including gesture (McNeill, 1992). In particular, Müller considers that the bimodal combination of speech and gesture is one of the key contextual resources available to catch attention and focus it on the dynamic process of metaphor activation (Müller, 2008).

More precisely, metaphorical gestures belong to the category of representational gestures, which can be defined as gestures playing a referential function by directly "exploit[ing] imagery" (Goldin-Meadow, 2004, p.314). A metaphorical gesture bodily refers to a concrete source concept either by drawing its shape and dimension, or by locating parts of it or tracing its trajectory or miming its actions (Colletta et al., 2010, McNeill, 1992). Such gestures are representational: they help express ideas by providing concrete images to refer to abstract concepts (e.g. the gripping and trenching gestures to express respectively founding and dividing) (Müller, 2008, McNeill, 1992).
# 4E learning and reasoning together in a CPI

## Collaborative learning as part of the social dimension of cognition

The very idea of following how metaphor are co-constructed in a CPI through a combination of speech and gesture supposes the students to deal with typical issues related to the Semin and Cacioppo's grounding Social Cognition model (2008), such as monitoring and goal-mediated synchronization. According to this model, monitoring synchronization is based on neurophysiological processes by which a perceiver mirrors actions produced by another. When subjects are engaged in a collective task, goal-mediated synchronization plays a complementary role by helping not only to mirror actions but to select adaptive and complementary responses.

Considering metaphor as an instance of collective reasoning is also consistent with studies on group cognition, which showed that taking the group as a cognitive agent helps make sense of learning processes that overcome the sum of students' individual progress (Stahl, 2006). Following an extended cognition perspective, we do not address metaphorical reasoning together as a cognitive process taking place 'in' a specific group agent, separated from its environment. We rather try to figure out how collective thinking occurs through interactions among individuals who are themselves interacting with their environment, exploiting the contextual resources available to apprehend the concepts at stake. If the group as a whole might build ownership of (part of) the metaphorical process, it is always through the environment shared among the participants. For instance, everybody might align in gesturally conferring the same specific metaphorical meaning to the same place in the room.

## Reasoning in a CPI

CPI is based on a pragmatic approach to philosophy: reasoning about abstract concepts is made possible through a confrontation of concrete individual experiences within a community. In that sense, we claim that it can be linked to embedded cognition since actions with environment are viewed as a basis for conceptualization. According to Tozzi (2009), reasoning in Philosophy for Children can be specified into three distinct reflective tasks: *problematization*, *conceptualization* and *argumentation*. *Problematization* consists in questioning our own points of view and those of others, as well as the assumptions they imply, their origin and consequences. *Conceptualization* aims at defining concepts, developing conceptual networks and distinctions. *Argumentation*, here used in a very restrictive way, consists in basing statements on rational arguments.

The more extensive view on argumentation supported by the research community on argumentation studies also offers fruitful insights about how metaphors generally contribute to reasoning. Metaphorical process is actually at the crossroads of two fundamental practices of reasoning: arguing by categorization and arguing by analogy (Plantin, 2018). In Grize's natural logic, reasoning is based essentially on the argumentative exploitation of images working as cognitive models:

* Natural logic can be defined as the study of logico-discursive operations used to construct and reconstruct a schematization. The double adjective is here to highlight the fact that they are thinking operations, but only as long as they are expressed in discourse. (Grize, 1997, p. 65).

## Visibility' as a condition for reasoning together

Research on collaborative learning showed that just having the students work in groups is not necessarily enough to foster true sociocognitive conflict (Roschelle & Teasley, 1995) and even less unlikely to automatically results in a cognitive resolution of such conflict (e. g. Polo et al., 2016). A usual way to analyze how fruitful student-student or teacher-student interactions are in terms of learning outcomes is to specify the type of talk that they should develop (Dyke et al., 2012, Mercer, 1996). Even if most of these approaches only focus in speech, one of them draw interesting perspectives for addressing collective reasoning more globally: *exploratory talk*. *Exploratory talk* is considered as the most cognitively productive form of talk in small groups:

* First it is talk in which partners present ideas as clearly and as explicitly as necessary for them to become shared and jointly evaluated. Second, it is talk in which partners reason together – problems are jointly analyzed, possible explanations are compared, joint decisions are reached. From an observer's point of view, their reasoning is visible in the talk. (Mercer, op. cit., p. 363)

Interestingly, a key property of *exploratory talk* is to make ideas and arguments *visible* to others. It seems that real-time 'visibility' of thinking-in-action is an essential condition for student-student interactions to actually foster collective reasoning. We assume that such 'visibility' can be reached and studied with a multimodal approach, notably through metaphorical gesture.
# Methodological approach to an empirical study of CPI

Our approach is deeply empirical. We provide a qualitative multi-level case study describing how bimodal metaphorical reasoning takes part to collaborative learning in a session of CPI. After specifying the pedagogical situation studied and the data collection, we present our research questions and working hypotheses. This methodological section ends with an explanation of each of our analytical steps.

## Pedagogical situation and data collection

The analyzed situation occurred in the particular context of a CPI demonstration. The session was moderated by M. Sasseville, one of the leading expert in the field, and observed by teachers, researchers, CPI practitioners and parents. It took place in November 2015, in an urban lower-class French secondary school, and involved a group of eleven students familiar with Philosophy for Children, aged 12 to 14 years, who volunteered to come on a Wednesday afternoon, a time when the school is usually closed. Participants sat in an arc facing observers who were placed on bleachers. The philosophical issue discussed was the definition of thinking. M. Sasseville started the CPI with the prompting question: "Where do thoughts come from?". The whole CPI was videotaped using a 360° camera situated in the center of the participants' arc. Speech was fully transcribed using Elan® software. Gestures where only annotated in selected episodes especially relevant for analysis.

## Research questions and working hypotheses

We explored the role of bimodal metaphors in collective reasoning in the context of this CPI. This global purpose consisted of four research questions:

1. How do the final analogies used by the students apprehend the *target concept* of thinking?

2. Does the metaphorical process trajectory evidence collective elaboration of metaphors?

3. Is metaphorical thinking together associated with advancement in reasoning?

4. More specifically, how do metaphorical gestures take part to conceptual complexification?

We based our work on the two following fundamental hypotheses:

1. Metaphors emerge and evolve through social interaction and interaction with the environment, during a CPI, describing conceptual trajectories that we can retrace *a posteriori*.

2. The salience of co-verbal metaphorical gesture make the associated cognitive process more visible and therefore foster the collective appropriation of the metaphor.

## Analytical steps: Following conceptual co-construction through bimodal metaphors

Our analytical strategy was to follow how conceptual co-construction occurred through metaphors using both verbal and gestural thinking and communicative resources. We proceeded in two main analytical steps. First, we coded the whole CPI discussion using Elan software, in two different ways, in order to identify episodes of special interest. We then undertook a multi-level qualitative case study based on some of these episodes.

### Systematic coding with Elan® software: coupling two complementary approaches

Two complementary coding schemes were held separately to identify episodes of special interest in Elan®, the first focusing on *source concepts* and the second on *cognitive models*.

The *source concept* coding scheme, applied to all the discussion preceding the closing sequence, consisted in 5 main annotating lines. The first line was used to identify and chronologically number episodes in which *source concepts* were used. Lines 2 and 3 were used to systematically annotate respectively verbal and gestural *source concepts*. For each of these lines, two dependent lines were added to relate each expressed *source concept* to a *target concept* and to a speaker. This systematic coding allowed us to highlight *source concepts* flowing and to calculate the frequency of each *source concept*.

The *cognitive model* grid was based on an exploratory coding of the closing sequence, which was later on applied to the rest of the discussion. The analysis of the final round table empirically *grounded* the construction of this second coding scheme (Charmaz, 2006). The analytical grain used was larger, corresponding to the level of the analogies made by the students to describe the main target concept: each code related to a metaphorical *cognitive model* helping them define ‘thinking’. Such comparisons are explicit analogies in the closing sequence, but they may be referred to in ways that are more implicit at different stages of the metaphorical trajectory retraced during the preceding discussion. Actually, our aim was to go back in time in order to figure out how the *cognitive model* underlying each final explicit analogy was gradually constructed through the CPI. In the second coding step, we applied the code of a *cognitive model* whenever at least one key defining feature of the *model* was referred to, either gesturally or verbally, during the discussion. Obviously, such previous occurrences of the *models* were
generally less complex than the final analogies, which is consistent with our hypothesis 1 stating that the metaphor gradually shapes through a specific cognitive trajectory.

## From the whole discussion to the conceptualization phases: multi-level deep qualitative case study

Crossing the results from the two independent coding of *source concepts* and *cognitive models*, we were able to identify that one particular *cognitive model*, 'using a file', was emblematic of the process of metaphorical collective thinking that we were interested in. We decided to focus on this case to conduct a multi-level deep qualitative study instantiating bimodal metaphorical reasoning together. We first specified how it helped the students apprehend the *target concept* of thinking in comparison with the other *cognitive models* that they used. Such analysis, coupled with a visualization of the chronological occurrences of the different *cognitive models* along the discussion, then allowed us to characterize how it related to the other models. After addressing the macroscopic level of reasoning progress through alternation and co-enrichment of multiple metaphors, and the mesoscopic level of the cognitive features composing this specific *cognitive model*, we zoom in to the micro-analysis of the bimodal complexification of such model through three conceptualization phases.

## Results: bimodal construction of metaphor as a ground for reasoning together

In this section, we present the results of this deep qualitative case study. First, the macro-level and meso-level analyses show how a specific *cognitive model* plays the role of a *conceptual synthetizing metaphor* (CSM). We then specify the micro-level bimodal collective complexification of such CSM.

## Metaphorical thinking together: macroscopic and mesoscopic analyses

### Final analogies as *cognitive models* of 'thinking'

During the closing sequence, when the students are asked by the moderator to conclude by making an analogy for 'thinking', they explicitly mention 9 cognitive models of this target concept, which were identified through an exploratory coding (here after mentioned in the chronological order of emergence in the conversation, in students' own words): *to dream*, *a milky way*, *to use files*, *cogwheels*, *to reflect*, *to remember*, *a cloud*, *a blurred picture*, *to imagine*. We used all the verbal and gestural information provided by the participants to characterize these analogies as cognitive models, in order to address our first research question about how they allowed the students to apprehend the abstract concept of 'thinking'. At most, this bimodal semantic analysis allowed for characterizing the 5 following aspects: nature of the fundamental units composing the system; whether it corresponds to a dynamic process or a static state; internal structure; localization; size (Polo, in press). In this closing sequence, some *cognitive models* are only mentioned without being described along these 5 characteristics, for other *models*, only 2 or 3 of these aspects are specified. Sometimes, the information is only conveyed in gesture, as, for instance, the localization of dreaming "in the head", or the dynamic processual dimension of cogwheels.

Such characterization of the 9 *cognitive models* reveals that all the students, at the end of the CPI, share a few common features in their apprehension of was 'thinking' is. First, it is a system consisting of fundamental units, the thoughts, which can be compared to pictures. Second, it involves chronological or causal relations that allow switching from one thought to another. Third, thinking is presented as an internal process localized within the cognizer. It is nothing but surprising that the students share some conceptual features about thinking at this moment of the CPI, since they have been discussing about this concept for more than 40 minutes.

<figure><img src="image_164.png" /><figcaption>**Figure 1.** Emergence and reuse of 5 *cognitive models* of thinking along the CPI.</figcaption></figure>

Nevertheless, coding of the *cognitive models* along the preceding discussion revealed that only 5 of them were used in the CPI before the closing sequence: *to remember*, *a blurred picture*, *to imagine*, *cogwheels* and *to*