five-point Likert scale) but would like to revise the assessment item for the security reason. Teacher D did not have an intention to use the ChatGPT-generated assessment item (rated 2 in a five-point Likert scale) because it is quite similar to examples provided in textbooks, so it is more efficient for teachers to write their own assessment items.

In the survey, teachers have polarized responses about their intentions to use the ChatGPT-generated
assessment items. Among 22 teachers, 10 teachers (45.5%) indicated that they had intentions to use
the ChatGPT-generated assessment items, whereas the same number of teachers (45.5%) indicated
that they did not have intentions to use the ChatGPT-generated assessment items. Two teachers (9%)
responded neutrally about their intentions to use the ChatGPT-generated assessment items. Teachers
who did not have intentions to use these assessment items explained that the ChatGPT-generated
assessment items are too simple and easy, are not aligned well with their instruction, are not aligned
with the curriculum, have inaccurate expressions, have incorrect answers, do not produce items with
the intended item difficulties, are very similar to the textbooks, and are not quite different from items
they can make. Two teachers wrote:

I have very little intention of using it; the difficulty level is very low; the solutions it presents
may not be what the curriculum intends; and I don't think ChatGPT is up to the task of
developing items that accurately assess the competencies that the curriculum wants students to
develop.

I don't intend to use it yet, because the problems that ChatGPT creates are very similar to the
textbooks. Even if I kept asking for new problems, they would just repeat the first problem with
different numbers. In other words, I don't think I would use ChatGPT specifically because it only
gives me typical problems from the textbooks or problem sets I have.

RQ3. Affordances and Challenges of using ChatGPT to design assessment items

In analyzing the teachers’ survey responses, we found that the teachers perceived the affordances of
using ChatGPT in terms of creativity (rich ideas, new types of problems, various contexts, extending
teacher’s limited thinking, and reducing the pains of creating items), efficiency (cost, time, and
speed), specific difficult levels (easy or medium-level difficulty), specific type of assessment
(performance assessment or formative assessment), specific type of items (multiple-choice items),
providing solutions to the problems, diagnosis of errors, creating scoring rubrics, convenience, neutral
(excluding teachers’ own biases or preferences), producing anticipated solutions, and using ChatGPT
for students’ learning. Among these affordances, nine teachers (41%) identified creativity and five
teachers (23%) identified efficiency in cost, time, and speed. However, the teachers identified its
challenges as potential mathematical errors (e.g., incorrect answers, inaccurate solutions, and vague
expressions), misalignment with curriculum, ethical issues (e.g., copyright issues, lack of information
about sources), possibility of teacher’s heavy reliance on ChatGPT, and security issues. In addition
to these potential limitations, the teachers also identified that ChatGPT does not reflect their
interactions with their students during their lessons, insufficient database (not quite different from
commercially available workbooks or textbooks), teacher’s intention for the assessment, the validity
of assessment created by ChatGPT, inappropriate item difficulty, and insufficient item discrimination.
Among these limitations, eight teachers (36.4%) identified the potential mathematical errors and
seven teachers (31.8%) concerned about the security of assessment because of its nature of open source.

## Discussions

The purpose of this study was to explore secondary mathematics teachers' experiences of using ChatGPT to design probability and statistics assessment items. For this purpose, we analyzed secondary mathematics teachers' conversations with ChatGPT and their survey responses about overall experiences with ChatGPT, their intentions to use ChatGPT-generated assessment items, and affordances and challenges of using ChatGPT in designing assessment items in Korea. The first research question examined the prompts that the secondary mathematics teachers used to design probability and statistics assessment items. The results show that most teachers specified some of the key details of the assessment items and did not identify mathematical errors or issues in ChatGPT's responses. On the other hands, the quality of follow-up questions is widely ranged (five teachers received score 0, five teachers received score 1, five teachers received score 2, and four teachers received score 3) and seven teachers evaluated ChatGPT's responses. The second research question examined the ChatGPT-generated probability and statistics assessment items after the teachers' conversations with ChatGPT and their intentions to use these assessment items. The survey results showed that the teachers had polarized responses to their intentions to use. The third research question examined what secondary mathematics teachers perceived its affordances and challenges of using ChatGPT in designing probability and statistics assessment items. Teachers perceived that ChatGPT provides affordances such as creativity and efficiency but identified limitations of potential mathematical errors, ethical issues, misalignment with curriculum, misalignment with instruction, and security of assessment because of its nature of open source.

The results of this study provide implications as follows. First, the analysis of teachers' prompts to design assessment items using ChatGPT provides implications that teachers need to be exposed to different types of prompts they can use in ChatGPT and they need to explore that the prompts they employed would determine whether they could get the desired outcomes. Without specific prompts, challenges, or evaluation of ChatGPTs' responses, some teachers simply accepted the ChatGPT's responses or repeated the same prompts to ChatGPT. Especially, many teachers mentioned that ChatGPT generated too easy items but did not produce the items with the intended item difficulty and sufficient item discrimination. As an exception, one teacher, who rated the ChatGPT-generated item as difficult, mentioned that ChatGPT was able to produce a more difficult item once the teacher added more conditions to the initial ChatGPT-generated item. In this study, we did not provide specific prompts that teachers can use in ChatGPT because we aimed to explore the quality of prompts that teachers use. However, we might offer examples of different types of prompts that teachers can use and then analyze the frequency of using specific prompts or explore how teachers employ different types of prompts to the same ChatGPT's responses.

Second, the teachers perceived that ChatGPT was creative, efficient, and convenient as affordances but addressed mathematical errors, inaccuracy, incorrectness, and vagueness as its major challenges. It is interesting to observe that many teachers identified these mathematical issues in the survey but few of them actually addressed these concerns in their conversations with ChatGPT. Facing such
issues in ChatGPT, teachers should be able to use their mathematical knowledge to critically examine
the mathematical accuracy, correctness, and performance of ChatGPT and address them to ChatGPT.
Another challenges of using ChatGPT in designing assessment items are the misalignment between
ChatGPT-generated items and curriculum in Korea and misalignment between ChatGPT-generated
items and their own instruction. Because of its importance to use the specific grade-level
mathematical vocabulary, concepts, or ideas outlined in the curriculum, it needs to be further
examined whether ChatGPT understands or has an access to the specific curriculum materials and
grade-level expectations in each country.

Lastly, writing assessment items is very stressful for teachers in Korea because students and parents
are very sensitive to the assessment items and often complain if there are any vagueness, errors, or
issues in the assessment and if there are any similarities between assessment items and commercially
workbook items. Because the security of assessment items is one of the most important issues for
teachers in Korea, they might not use the ChatGPT-generated assessment items without any major
revisions or modifications. However, as three teachers commented in the survey, it would be a great
learning opportunity for students to design or discuss a mathematical problem using ChatGPT.
Because teachers experienced that ChatGPT often produced inaccurate, incorrect, and vague
mathematical ideas, they would like to use ChatGPT-generated items as formative assessment for
their students to explore whether students can detect any mathematical errors or issues in ChatGPT's
responses.

References

Einarsson, H., Lund, S. H., & Jónsdóttir, A. H. (2023). Application of ChatGPT for automated problem reframing across academic domains. *Computers and Education: Artificial Intelligence*, 100194.

Frieder, S., Pinchetti, L., Griffiths, R. R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., Chevalier, A., & Berner, J. (2023). Mathematical capabilities of ChatGPT. *arXiv preprint*, arXiv:2301.13867.

Koehler, M. J., Shin, T. S., & Mishra, P. (2012). How do we measure TPACK? Let me count the ways. In R. N. Ronau, C. R. Rakes, & M. L. Niess (Eds.), Educational technology, teacher knowledge, and classroom impact: A research handbook on frameworks and approaches (pp. 16-31). IGI Global.

Mishra, P., & Koehler, M.J. (2006). Technological pedagogical content knowledge: A framework for integrating technology in teacher knowledge. *Teachers College Record*, 108(6), 1017-1054.

Patton, M.Q. (1990). *Qualitative Evaluation and Research Methods*, (2nd ed.), Newbury Park, CA: Sage

Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. *Educational Researcher*, 15, 4–14.

Scherer, R., Siddiq, F., & Tondeur, J. (2019). The technology acceptance model (TAM): A meta-analytic structural equation modeling approach to explaining teachers' adoption of digital technology in education. *Computers & Education*, 128, 13-35.
# Assessment by skills of an interdisciplinary project in which mathematics is the common thread.

Maria Antonietta Lepellere¹ and Marzia Toso²

¹ University of Udine, Italy; maria.lepellere@uniud.it

²ISIS “Arturo Malignani”, Udine; marzia.toso@malignani.ud.it

In this contribution we propose the design of an interdisciplinary STEAM project named “Dark ages?”. It is based on Project Based Learning (PBL) and outdoor learning, in which digital and manipulative tools, virtual and real experiences, in the classroom and in the real world, are intertwined. A formative and summative evaluation proposal is presented, and the first results analysed. Some results of a survey on how students experienced the project are also presented.

Keywords: Project based learning, outdoor learning, STEAM, formative and summative assessment, survey, secondary education.

## Introduction and Theoretical framework

In many contexts it is stated that one of the tasks of the school is to “educate complexity”, train and consolidate the skills of reading reality as a complex system, where many variables operate. Interdisciplinarity can be a way to provide the student with overviews of complexity, in which points of observation, languages and interpretations intersect. From this perspective an interdisciplinary STEAM project “Dark ages?” was proposed. The title itself “Dark ages?” is an example of an interpretation called into question by the question mark. Who says that the Middle Ages are a dark, backward, and difficult period in European history? When? For what reason? Is that true? The question mark raises a doubt, asks questions, broadens perspectives, goes beyond the cliché. The aim of the interdisciplinary project is to bring students to the heart of an era, the medieval one, through STEAM and other subjects, as literature, history and English literature. Part of the project took place at school and another directly in the places studied: Siena Cathedral with the Museum and The Piccolomini bookshop and Pisa with the “Scuola Normale Superiore”, the tower of the clock palace where Count Ugolino had been locked up and the cemetery where the statue of Fibonacci is. The following methodologies are used in the activities: Debate, Active Learning, Outdoor Education, Peer Tutoring, PBL. Furthermore, online research activities and evaluation of digital sources and resources, laboratory activities related to the creation/sharing of materials in the Moodle institutional e-learning area and in other shared digital areas as Genial.ly for the digital Escape Room and Padlet.com, were used in an active learning perspective. The technological tools used were: GeoGebra, Desmos, 3D print, Fusion 360° software, Teodolite for on-site measurements.

The term STEM refers to teaching and learning in the fields of science, technology, engineering, and mathematics. STEM education aims to help the next generation of students to solve real-world problems by utilizing knowledge of multiple disciplines and horizontal competences such as critical thinking, collaboration, and creativity. The addition of the artistic skills to the science and technology education gave birth to a new acronym: STEAM (notice the addition of A for arts). Zemelman, Daniels, and Hyde (2005) provide insight into the ten best STEM pedagogical practices for successful integration of STEM disciplines. There are as follows; using manipulatives and hands-on learning;
cooperative learning; discussion and inquiry; questioning and conjectures; (5) using the justification of thinking; writing for reflection and problem solving; using a problem-solving approach; integrating technology; teacher as a facilitator approach; using assessment as a part of instruction.

PBL has a lot of potential to enhance 21st century skills and engage students in real-world tasks (e.g., Kingston, 2018). It promotes interconnected worldview, links among disciplines, and presents an expanded view of subject matter (Blumenfeld et al., 1991; Kingston, 2018). Therefore, PBL is a promising teaching method for integrated science education that can be defined as an effort to organize or integrate science curriculum content into a meaningful whole by a constructive and context-based approach that crosses subject boundaries and links learning to real world (Czerniak & Johnson, 2014).

Learning outside the classroom essentially can be defined as use of resources out of the classroom to achieve the goals and objectives of learning (Knapp, 2010). The constant focus on textbooks and formal mathematical practice might invoke a view among students that mathematics is abstract, distanced and only useful in a in classroom context working only in the textbook. Existing research on outdoor learning in mathematics indicates positive affective outcomes and possible academic benefits from learning mathematics in an out-of-school context (Moffett, 2011). Moreover, outdoor environments, are real-life contexts enabling students to internalise, transfer and apply mathematical ideas and provides direct experience, and the students need to be active in the learning process. It lends itself to the Inquiry-based mathematics education a student-centred form of teaching whose guiding principle is that the students are supposed to work in ways like how professional mathematicians work (Artigue & Blomhøj, 2013). Lee et al. also suggested community partnerships, where students collaborate with professionals, as an important component of project based learning.

During the process of teaching and learning, teachers do assessment for learning (formative assessment) and assessment of learning (summative assessment). Black and William (2009) after considering the main features of teaching and learning defined formative assessment as: "Practice in a classroom is formative to the extent that evidence about student achievement is elicited, interpreted, and used by teachers, learners, or their peers, to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions they would have taken in the absence of the evidence that was elicited" (see also Cusi et al. 2017 and Aldon et al. 2017). Inquiry based learning needs formative assessment due to its process character.

In this contribution we propose the design of an interdisciplinary STEAM project such that students are actors of their own knowledge through collaboration and discovery. The research questions are the following: How can this type of activity be assessed? How do students perceive this type of activity and the way to evaluate it?

## Methodology and participants

The project was addressed to two 11th grade classes, at Applied Sciences High School of in Udine, Italy. In this article only one class of 19 pupils will be examined. The methodology used for the formative assessment was that of continuous monitoring, with related feedback, of the Forums, Padlets, Logbooks, delivery of the results of the proposed exercises and control via the Moodle eLearning platform and evaluation of the oral presentation of the works. For the final skills for summative assessment, we proposed a rubric containing six macro areas summarized by the following
disjoint descriptors: Collaborate, Evaluate, Read and Understand, Communicate, Digital skills, Learn. A survey was also proposed at the end of the project to evaluate the appreciation and the effectiveness of the proposed project. In addition to a descriptive statistical analysis of the closed questions, a thematic analysis of the open questions was carried out. In the future, a comparative analysis of the results obtained will be carried out.

## The design of the Project and the formative assessment

Interdisciplinary STEAM project requires a completely different way of grading. The normal teaching activity is also remodelled and reorganized, alternating laboratory/experiential moments with moments of study/empowerment. A working methodology for “open classes” is adopted: in some lesson hours, the classes are merged and reorganized into groups based on the proposed activity. The underlying theme of the project was to analyse a historical era, the Middle Ages, from various aspects to answer the question: was it really a “dark age?” Among the various parts of the STEAM project, we will describe those where mathematics plays a predominant role.

**Activity 1. "Fibonacci: the man who gave us numbers".** The students discovered unusual or hidden aspects of medieval (and non-medieval) mathematics through the resolution of a digital Escape Room and shared among peers the mathematical properties discovered. Before this activity the students only knew the definition of sequences as particular functions defined on the set of natural numbers. They had already studied the Fibonacci sequence in the previous school year and this year, with the IT teacher, they reviewed it together with the recurrence sequences. Among others, in the digital Escape Room there were also puzzles related to real life contest. As formative assessment the students, divided into groups, had to discover the solutions to the questions asked, using all the resources they deemed useful. They had to note down through logbooks the solutions of the questions, with the strategy used, and the difficulties encountered. The results were shared with the other groups in the classroom during the lesson. The module on "Fibonacci" concluded with the creation of a physical Escape Room, revisiting of Dante's Inferno, which allowed other classes of the Institute to "taste" the medieval world. It was entirely designed by the students including the divisions of the rooms and the interdisciplinary puzzles to move from one room to another, included the costumes and sets just like real directors and actors. They used the Moodle Forum for sharing ideas.

<figure><img src="image_54.png" /><figcaption>Figure 1: Digital and physical escape room. The Fibonacci statue.</figcaption></figure>

**Activity 2. "The white cathedral of Siena": a masterpiece of gothic architecture.** Students, divided into groups, were asked to study the Cathedral of Siena from different point of view.

a) From a mathematical point of view: the students studied and (re)built some typical elements using the dynamic geometry software GeoGebra, such as friezes, rose windows, golden proportions, conics (see Figure 2 and 3). Individual student products were entered for evaluation in a Padlet.
<figure><img src="image_55.png" /><figcaption>Figure 2: Tessellation of a detail of cathedral with GeoGebra and manipulative objects. Find conics.</figcaption></figure>

<figure><img src="image_56.png" /><figcaption>Figure 3: Let's search for the golden ratio!</figcaption></figure>

b) From a historical-philosophical-architectural point of view, they studied the floor “Come stelle in terra” analysing the proposed allegorical meanings. In Figure 4 there are some examples.

<figure><img src="image_57.png" /><figcaption>Figure 4: Allegorical part</figcaption></figure>

c) From Technology and Engineering point of view, with Peer tutoring, they tried to 3D print some elements by developing the Fusion 360° software and learned to use the Teodolite.

<figure><img src="image_58.png" /><figcaption>Figure 5: 3D print and Teodolite</figcaption></figure>

As final activities of the project they also studied the Unity language to create a virtual museum, using 360-images. In figure 6 there are some examples.

<figure><img src="image_59.png" /><figcaption>Figure 6: Multimedia production</figcaption></figure>
**Activity 3. The outdoor activity: The visit to Pisa and Siena.** The visit included laboratories as: Physical measurements with the laser and with Teodolite, as an application of the trigonometry, using the plans provided to us directly by the Opera di Siena; mathematical and architectural studies with related data analysis; take photos, videos with the aim of preparing a virtual museum.

The students had the opportunity to delve deeper into the topics covered thanks to the interventions of experts: The architect De Benedetti of the Opera for an architectural, geophysical, and historical-cultural point of view; Professor Bellissima, with a contribution of music to mathematics, who discussed Guido d'Arezzo and, starting from the miniatures contained in the Piccolomini Chapel which show the first forms of musical writing, he led the students to play with functions on the Cartesian plane which represent the various forms of musical writing from the beginning up to digital music; Professor Chiantini, who discussed “The fields of fortune”. He starts from a marble mosaic in the Cathedral of Siena, called “The wheel of fortune”, and proposed a reflection on medieval cosmology in the light of non-Euclidean geometries.

## From formative to summative assessment

In relation to the final skills achieved by the students, we propose a rubric, inspired to *The Periodic Table of Skill*, divided into 6 macro areas: Collaborate, Evaluate, Read and understand, Communicate, Digital skills (Dig Comp 2.1), Learn. The group of teachers has chosen, under the guidance of the researcher, inspired to the periodic table (Table 1), to use for each macro competence, some items which are best suited to representing all the disciplines involved. Collaborate includes respect, group interaction, teamwork, valorisation of one's own and others' abilities, knowing how to manage conflicts. Evaluate includes respect shared criteria, to be able to reconstruct the operations carried out. Read and Understand include understanding the deliveries, recognizing the topic, recognize resolution and argumentative strategies, identify information that responds to one or more topics, relating implicit and explicit information, recognize the logical-syntactic function of an argument, critically interpret information. Communicate include, recognize the main theme or argument of a text or specific parts of it, capture the intentions and the author's point of view, correct the cultural references used to support the argument. Digital skills include manage data, information and digital content, share information through digital technologies, develop digital content, integrate and rework digital content, use digital technologies creatively. Learn include acquiring a specific working method by making mistakes, apply the main rules and formulas, plan and monitor learning, deal with problems by identifying the appropriate resources, identifying conceptual nodes and connecting them, evaluating information to support reasoned conclusions.

<table><caption>Table 1: The Periodic Table of Skills</caption><thead><tr><th colspan="4">COLLABORATE</th><th colspan="4">EVALUATE</th><th colspan="6">READ AND UNDERSTAND</th></tr><tr><th>Respect</th><th>Interact in a group</th><th>Teamwork</th><th>Enhancement of their own and others' abilities</th><th>Respect criteria shared</th><th>Being able of reconstruction the operations performed</th><th>Comprehend deliveries</th><th>Recognize the theme the topic of an object</th><th>Recognize the strategies resolutions and argumentative</th><th>Individuate information who respond to one or more topics</th><th>Put in information report implicit and explicit</th><th>Recognize the function logical-syntactic of a topic</th><th>To interpret critically the information</th></tr></thead><tbody><tr><th colspan="13">TO COMMUNICATE</th></tr><tr><td>FOCUS</td><td>Recognize the theme or main argument or a specific parts of it</td><td>SCOPE</td><td>Capture the author's intentions, point of view, or purpose</td><td>CONGRUENCE, REFERENCES Correctness and congruence of the cultural references used to support the argument</td><td>Manage data, information and digital content</td><td>Share information through digital technologies</td><td>Develop digital content</td><td>Integrate and rework digital contents</td><td>Use digital technologies creatively</td><td>Apply the main rules and formulas</td><td>Plan e monitor your own learning</td><td>Dealing with problems identifying the adequate resources</td><td>Identify conceptual nodes and connect them</td><td>Evaluate information for hold up reasoned conclusions</td></tr><tr><th colspan="13">DIG COMP 2.1</th></tr><tr><td colspan="13">LEARN</td></tr></tbody></table>
The need to have 2 votes on the register led us to unify some macro-sectors. In the first we merged Collaboration, profitable relationships, respect, civil discussion of ideas, management of other people's places and objects, theatrical interpretation and problem solving. The second included: interdisciplinary path, mathematical study, argumentative and expository ability (Table 2.)

<table><caption>Table 2: Student results</caption><thead><tr><th colspan="15">Collaboration, profitable relationships, respect, civil discussion of ideas, management of other people's places and objects, theatrical interpretation and problem solving.</th></tr><tr><th>10</th><th>8,5</th><th>9,5</th><th>9,5</th><th>7,5</th><th>9</th><th>8,5</th><th>9</th><th>10</th><th>10</th><th>9,5</th><th>9</th><th>10</th><th>9</th><th>9</th><th>9</th><th>10</th><th>8,5</th><th>8,5</th></tr></thead><tbody><tr><th colspan="15">Interdisciplinary path, mathematical study, argumentative and expository ability</th></tr><tr><th>10-</th><th>7</th><th>10</th><th>8</th><th>7</th><th>9</th><th>8</th><th>9-</th><th>8</th><th>8</th><th>10-</th><th>7</th><th>9</th><th>7</th><th>10</th><th>8</th><th>8</th><th>8</th><th>7</th></tr></tbody></table>

As can be seen, the students' grades, from 0 to 10, were very high thanks to the interdisciplinarity of the project with a mean of 8 in the first case and 9 in the second. In 8 cases out of 19 the scores for the two types of judgment were close to each other (with a difference of 1 or 2 point). The scores of the first evaluation framework were far higher than the second in 10 cases over 19 while just in 1 case the second type of evaluation gave far better result. There is a positive correlation 0.41 between the two evaluations, meaning that the two ratings are not independent in the sense that the second is positively influenced by the first. But this correlation is not very high, which means that the two assessments managed to capture specific skills that would not have emerged with a single assessment.

## The survey

A project evaluation survey was also proposed to the students. We report the results only regarding 4 over 36 questions: 1. Which aspect did you appreciate the most? 2. Which aspect bothered you the most? 3. I think that evaluating an interdisciplinary activity is adequate and correct. 4. How likely are you to recommend this activity to another class council at your school?

1. Which aspect did you appreciate the most? The appreciation of the group work and with the other classes and the visit to Siena were the most used. We report just some sentences in this direction: I appreciate the concrete observation of what was studied during the visit to Siena; Moments of discussion, brainstorming, research, and related sharing. The division of the material to be studied between the classes and between the groups created in the classes. This allows us to delve into a specific topic in an exhaustive way without being totally unaware of the other aspects related to the macro topic; I appreciate the exchange of information between classes, especially when one class exposed something to the other class, the class listening could intervene by adding content or asking questions.

2. Which aspect bothered you the most? Most students did not find any negative aspects, Just for few the main problem is the time: required for the project seemed excessive for the fear of having to tackle the program too quickly, but not enough for visiting the cities.

3. I think that evaluating an interdisciplinary activity is adequate and correct. The possibility for the answers were: strongly disagree, disagree, neutral, agree and strongly agree. The 42% were strongly agree, the 26% were agree, the 32% were neutral and nobody disagree or strongly disagree.
4. How likely are you to recommend this activity to another class council at your school? On a scale from 0 to 10, only two gave a score of 4, the other was greater than 7 with a total mean of 8.

The project was appreciated, and this serves as an encouragement to design new ones.

## Conclusions

The experience of an interdisciplinary activity where different student skills is brought into play is a challenge for both teachers and stimulating students. PBL involves a dynamic classroom approach, which emphasizes on long-term learning, interdisciplinary and student-centered activities. Students need both manipulative and technological objects and need to experience on-site activities. The possibility of interacting with teachers other than one's own and with experts is an opportunity to compare various communication and operational methods and offers opportunities for discussion, socialization, and integration between students. The activity also facilitates dialogue and collaboration between teachers, encouraging the sharing of effective strategies.

Students' assessment should be considered an integral part of instruction. Each instructional activity could be seen as an opportunity for the teacher to assess as well as for students to learn. Emphasis should be on formative assessment that aims at supporting students learning. This includes reflection, self and peer evaluation, and teachers' feedback throughout the project process. Assessment should include a specific end-of-project phase that ensures reflection on what was learned as well as the creation of a project artefact. A public presentation of the project supports students' communication skills, can motivate students, and presents an opportunity for feedback. Instead of a presentation, the product itself can be public.

We believe that A (for arts and design) in STEAM is an important addition from the original STEM approach. Arts and design permit a more divergent thinking in students, giving space to more creative solutions to problems. For a real implementation of STEAM, it is needed a radical change in the educational culture and administration. Curriculum and teacher assessment methods should change giving importance not only to the concepts but also the acquisition of transversal competences. Our findings highlight the importance of having an assessment rubric with criteria that addressed both disciplinary specific skills as well as generic STEM skills.

The excellent results obtained encourage us to promote other projects like this. But evaluating this type of project is not easy and it is a challenge that teachers are called upon to accept to have students who are increasingly ready to face the complexity of our times.

## Acknowledgment

This project was possible thanks to the precious collaboration of the teachers of the “Malignani” school in Udine Brocato Maria Concetta, Donnini Letizia and Fabris Letizia. We also thank the professors Zangrando Nicola of Udine University and Bellissima Fabio, Chiantini Luca, Scala Riccardo and Scianna Giuseppe of Pisa University.

## References

* Aldon, G., Cusi, A., Morselli, F., Panero, M., & Sabena, C. (2017). Formative assessment and technology: reflections developed through the collaboration between teachers and researchers. In
G. Aldon, F. Hitt, L. Bazzini & U. Gellert, Mathematics and technology: a CIEAEM source book. Series 'Advances in Mathematics Education'. Springer International Publishing.

Artigue, M., & Blomhøj, M. (2013). Conceptualizing inquiry-based education in mathematics. *ZDM, 45*(6), 797-810.

Black, P., & Williams, D. (2009). Developing the theory of formative assessment. *Educational Assessment, Evaluation and Accountability*, 21, 5–31.

Blumenfeld, P. C., Soloway, E., Marx, R. W., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991). Motivating project-based learning: Sustaining the doing, supporting the learning. *Educational Psychologist, 26*(3-4), 369–398.

Cusi, A., Morselli, F., & Sabena, C. (2017). Promoting formative assessment in a connected classroom environment: design and implementation of digital resources. *ZDM, 49*, 755-767.

Czerniak, C. M., & Johnson, C. C. (2014). Interdisciplinary science teaching. In S. K. Abell, & N. G. Lederman (Eds.), *Handbook of research on science education* (pp. 395-411).

Kingston, S. (2018). Project based learning & student achievement: What does the research tell us? *PBL Evidence Matters*, 1(1), 1-11.

Knapp, C. E. (2010). Teaching for Experiential Learning: Five Approaches That Work. *Journal of Experiential Education*, 33, 288-290.

Lee, J. S., Blackwell, S., Drake, J., & Moran, K. A. (2014). Taking a leap of faith: Redefining teaching and learning in higher education through project-based learning. *Interdisciplinary Journal of Problem-Based Learning, 8*(2), 2. [https://doi.org/10.7771/1541-5015.1426](https://doi.org/10.7771/1541-5015.1426)

Moffett, P. V. (2011). Outdoor mathematics trails: an evaluation of one training partnership. *Education* 3-13, 39(3), 277-287.

Zemelman, S., Daniels, H., & Hyde, A. (2005). Best practice: New standards for teaching and learning in America's school (3rd Edition). Portsmouth, NH: Heinemann
# Advancing assessment in fractions: designing and implementing formative proficiency tasks

Hui-Chuan Li

University of Edinburgh, Moray House School of Education and Sport, Edinburgh, United Kingdom; huichuan.li@ed.ac.uk

While many studies have focused on the challenges students face in the domain of fractions and the corresponding pedagogies of teaching and learning, there has been inadequate and disproportionate attention dedicated to assessment resources, particularly those tailored for formative assessment in the context of fractions. This study takes a step forward in contributing new insights to this field by designing fraction proficiency tasks explicitly intended for formative assessment of students’ comprehension of fractions. These fraction proficiency tasks were administered to a class of 35 fifth-grade students (ages 10–11) with mixed abilities in a primary school in Taiwan to evaluate their understanding of fractions. Findings of the study offer valuable insights into assessing students’ understanding of fractions and provide a comprehensive view of the diversity in students’ understanding and the extent of these differences. Implications for future studies are also presented.

Keywords: Fractions, fraction proficiency, assessment resources, assessment for learning, formative assessment.

## Introduction

The teaching and learning of fractions persistently present challenges for both teachers and students. It has been argued that, for many students, learning fractions often involves merely manipulating symbols to arrive at the correct answer. While they might employ the appropriate fractional terms and solve some fraction-related problems, several critical aspects of fractions still escape them (Soni & Okamoto, 2020). Students’ struggles with fractions often stem from the intricate relationships between various representations and fundamental arithmetic operations (Cramer et al., 2002), wherein the simultaneous symbolic nature of fractions contributes to these challenges. For mathematicians, fractions are rational numbers expressible in the form “a / b” where b ≠ 0, rather than simply representing parts of wholes. They are not just ratios of two natural numbers but also constitute numbers in themselves.

In school, many children often receive only brief exposure to the concepts and procedures of fractions and are taught fraction algorithms with minimal emphasis on their conceptual underpinnings (Lenz et al., 2022). One of the conventional concrete approaches to learning about fractions often involves thinking in terms of partitioning or equal sharing. It is wise to base this idea on discrete countable objects, as well as on objects that may require the dissection of a continuous whole. However, this concrete approach does not cover the entirety of understanding fractions. For instance, the concept of “equal sharing” is just one among many properties of fractions and, on its own, is not adequate to convey a meaningful understanding of fractions to children.

Much research has concentrated on the challenges students encounter specifically within the field of fractions and their associated teaching and learning pedagogies, However, there has been insufficient and disproportionate attention given to assessment resources, especially for formative assessment,
specifically designed for fractions. The significance of formative assessment, also referred to as
assessment for learning, lies in its capacity to offer continuous feedback and insights into students'
understanding and progress. This study has taken a step forward in contributing new insights to this
field by designing fraction proficiency tasks explicitly intended for formative assessment of students'
comprehension of fractions. In doing so, it seeks to provide valuable insights into the assessment of
students' understanding of fractions, offering a comprehensive perspective on the diversity and extent
of challenges encountered by students.

Fraction proficiency tasks

Drawing from Tsai and Li's (2017) fraction proficiency framework and an extensive review of
fraction-related studies, 15 tasks were designed to assess students' comprehension levels and identify
areas where they might encounter difficulties in the field of fractions. The content of these 15 tasks
was specifically organized in four major topics to encompass the five dimensions of fraction
proficiency identified by Tsai and Li (2017), which include: (1) the part-whole, measure, quotient,
operator and ratio constructs of fractions, (2) the concept of equivalent fractions, (3) the procedural
fluency for and conceptual understanding of fraction operations, (4) the relationship between
fractions, decimals and percentages, and (5) the transition between different forms of representations
involving fractions. Lesh's (1981) representation model (Dimension 5) was integral to all tasks that
required students to solve problems by transitioning between representations. In the following
sections, examples of tasks for each topic will be provided.

**Topic 1: Five constructs of fractions**

Topic 1, centered on five constructs of fractions (Dimension 1), involved designing four tasks aimed at assessing students' understanding of part-whole, measure, quotient, operator, and ratio constructs related to fractions. The statements for these four tasks are provided in Table 1.

**Topic 2: Equivalent fractions**

In Topic 2, focusing on equivalent fractions (Dimension 2), three tasks were formulated to explore
students' comprehension of equivalent fractions and their conceptualization of expanding and
reducing fractions to determine an equivalent form. Table 2 outlines these three tasks.

Topic 3: Multiplication of fractions

Topic 3, emphasizing the multiplication of fractions (Dimension 3), comprised five tasks designed to assess students' understanding of the reasoning behind their procedural skills in performing fraction multiplication. The statements for these four tasks are provided in Table 3.

Topic 4: Fractions, decimals and percentages

In Topic 4, three tasks were devised to examine the extent to which students recognize the relationship
between fractions, decimals, and percentages (Dimension 4).
<table><caption>Table 1: Descriptions for tasks related to Topic 1</caption><thead><tr><th></th><th>Statement of Task</th><th>Transitions between representations (Dimension 5)</th></tr></thead><tbody><tr><td rowspan="5">Topic 1<br>Five constructs of fractions<br>(Dimension 1)</td><td>What is a fraction?<br>How would you would explain to someone what a fraction is? Please offer three different explanations, and one or more of your explanations needs to relate to a real-life situation.</td><td>From the fractional symbolic representation to the real-life situation representation.</td></tr><tr><td>Are they reasonable to you? (Students are given a set of cards that visually represent a fraction)<br>Please select the cards you consider reasonable and explain your reasoning behind your choices.</td><td>From the pictorial representation to the fractional symbolic representation.</td></tr><tr><td>Who spends more?<br>Mary and John went to McDonalds. Mary spends 1/4 of her pocket money and John spends 1/2 of his. Do you agree it is possible that Mary spent more than John? Why do you think this?</td><td>From the real-life situation representation to the fractional symbolic representation.</td></tr><tr><td>Which ones are reasonable? (Students are given a set of cards that visually represent a fraction)<br>Please look at the cards provided; which of these cards are reasonable and which are not?</td><td>From the pictorial representation to the fractional symbolic representation</td></tr></tbody></table>

<table><caption>Table 2: Descriptions for tasks related to Topic 2</caption><thead><tr><th></th><th>Statement of Task</th><th>Transitions between representations (Dimension 5)</th></tr></thead><tbody><tr><td rowspan="4">Topic 2<br>Equivalent fractions<br>(Dimension 2)</td><td>What are equivalent fractions? (students are given a set of cards that visually represent an equivalent fraction).<br>Please write down your observations from these cards, and then elaborate on how your findings are connected to equivalent fractions.</td><td>From the pictorial representation to the fractional symbolic representation</td></tr><tr><td>Match the pairs (students are given a set of cards that visually represent an equivalent fraction)<br>Please match equivalent fractions from these cards and then explain how you paired them.</td><td>From the pictorial representation to the fractional symbolic representation</td></tr><tr><td>Who gets more?<br>At two different tables where 2 children were sharing 3 chocolate bars and 6 children were sharing 9 chocolate bars. These chocolate bars are all the same size.<br>Please indicate who will receive more and elaborate on your thought process behind your choice.</td><td>From the real-life situation to the spoken representation</td></tr></tbody></table>
<table><caption>Table 3: Descriptions for tasks related to Topic 3</caption><thead><tr><th></th><th>Statement of Task</th><th>Transitions between representations (Dimension 5)</th></tr></thead><tbody><tr><td rowspan="6">Topic 3<br>Multiplication of fractions<br>(Dimension 3)</td><td>Let's fold a paper fraction (Students are given pieces of colour paper)<br>Please fold the fractions: 1/8, 1/6 and 1/12 using the paper provided.</td><td>From the fraction symbolic representation to the manipulative representation</td></tr><tr><td>Jenny's birthday party<br>Jenny wants to invite her three best friends to come to her party. Each of her three friends can consume 3/4 of a pizza<br>Please illustrate with a diagram to show the quantity of pizza<br>Jenny will require, and then provide a mathematical written representation to represent your drawing.</td><td>From the real-world situation to the pictorial representation</td></tr><tr><td>What do you think 2/3 × 5?<br>Please provide a real-life scenario that represents the mathematical operation 2/3 × 5 and then use a drawing to represent 2/3 × 5.</td><td>From the fractional symbolic representation to the real-life situation and to the pictorial representation</td></tr><tr><td>How much cake had Jenny's brother eaten?<br>Jenny's mum made a square-shaped cake for her birthday. At the party, half of the cake was eaten and then the rest was put in fridge. The next day, Jenny's brother ate 2/3 of the remaining part of the cake.<br>(Students are given pieces of color paper)<br>How would you fold the paper to illustrate the portion of cake Jenny's brother had consumed? Afterwards, provide a written mathematical representation to explain the folding method.</td><td>From the real-life representation to the manipulative representation</td></tr><tr><td>What do you think 1/4 × 3/4?<br>Please provide a real-life scenario that represents the mathematical operation 1/4 × 3/4 and then use a drawing to represent 1/4 × 3/4</td><td>From the fractional symbolic representation to the real-world representation and to the pictorial presentation</td></tr></tbody></table>

## Data collection and analysis

This study recruited a class of 35 fifth-grade students (ages 10–11) with mixed abilities from a primary school in Taiwan to evaluate their understanding of fractions. The four topics covered in the fraction proficiency tasks had been introduced to the participants in their previous school years as part of the current mathematics curriculum in Taiwan. The fraction proficiency tasks were given to all students without time limits for completion. Most students finished within 60 to 70 minutes and submitted their answer sheets and materials (such as cards and colored folding paper) to their teacher upon task completion.
<table><caption>Table 4: Descriptions for tasks related to Topic 4</caption><thead><tr><th></th><th>Statement of Task</th><th>Transitions between representations (Dimension 5)</th></tr></thead><tbody><tr><td rowspan="5">Topic 4<br>Fractions, decimals and percentages<br>(Dimension 4)</td><td>What is a percentage?<br>Please explain what a percentage is and provide some examples from your life where percentages are commonly observed.</td><td>From the fractional symbolic representation to the real-life situation representation</td></tr><tr><td>To what extent is Tom sure?<br>When Tom is going to school, his mum asks him if he is prepared well for his school test today. Tom replies: "Yes". Mum asks: "Are you sure?", Tom says: "One hundred per cent sure". Mum asks: "So you will get a full mark home, will you?" Tom makes a funny face and says: "Um, um, fifty per cent sure".<br>Please explain the meanings of "one hundred per cent sure" and "fifty per cent sure," and then demonstrate how fractions can represent these expressions.</td><td>From the real-life situation representation to the fractional symbolic representation</td></tr><tr><td>How are they related?<br>Here are three numbers: 0.4, 2/5 and 40%.<br>Please explain the relationship between these three numbers and how they can be converted from one form to another.</td><td>From the symbolic representation to the spoken representation</td></tr></tbody></table>

Apart from the data collected from the students' drawings and paper folding exercises, which were neither numerical nor narrative, much of the data collected was in the form of words. An inductive coding approach was employed to identify both general and distinctive features from the texts, following these three steps: identifying and labeling, reducing, and summarizing. This aligns with Thomas's (2006) assertion that inductive approaches are designed to facilitate an understanding of meaning in complex data by developing summary themes or categories derived from the raw data.

# The nature of students' fraction understanding

## What can be learned from Topic 1?

This topic, based on Kieren's (1988) theory, assessed students' comprehension of the five constructs of fractions, revealing that their understanding of fractions was either confused or incomplete. Their grasp of fractions predominantly revolved around the part-whole construct, with minimal consideration for the equality of each part of the whole. This aligns with existing literature suggesting an excessive focus on the part-whole construct, hindering students' ability to position fractions on a number line (Saxe et al., 2013). Moreover, the challenge students faced in positioning 3/5 on a number line in this study further confirms the findings of Soni and Okamoto (2020), emphasizing students' struggles in locating fractions accurately on a number line. Another challenge observed was the students' inflexible recognition of a fraction's unit. Chan et al. (2007, p. 26) also argued that "the units concept is a common conceptual deficiency among students, indicating a significant flaw in current fraction teaching practices in Taiwan".
## What can be learned from Topic 2?

An analysis of students' responses to the tasks in Topic 2 supported earlier research findings (Lamon, 2007) that students' reasoning of equivalent fractions was rather rule-based. For example, the "Who gets more" task in Table 2 showed that 20 out of 35 students answered it correctly. However, their explanations generally referred to "Because 3/2=9/6"; "Because they are equivalent fractions" or "By using the rule of expansion or reduction, you then know they are the same" to explain how they solved the task. It is not wrong to describe the equivalence of fractions based on the rules of expansion or reduction, but there is a danger that students apply "rule-based" explanations without understanding them (Levenson et al., 2004). This rule-based emphasis is also echoed in Yang's (2005) finding that both teachers and students tended to "rely on rule-based methods to explain their reasoning" in the field of fractions. This suggests the importance of allowing students the opportunity to articulate their thoughts and construct explanations that are not solely rule-based.

## What can be learned from Topic 3?

An understanding of fraction multiplication often challenges students because they have to distinguish it from multiplication of whole numbers, that is, from repeated addition to multiplicative reasoning. An operator construct of fractions is fundamental for interpreting the meaning behind the multiplication of fractions (Thompson & Saldanha, 2003). In this topic, the "What do you think 2/3 × 5?" task (see Table 3) shows that multiplication of a whole number and a proper fraction was presented by more than half of the students as repeated addition (e.g., $2/3 × 5 = 2/3 + 2/3 + 2/3 + 2/3 + 2/3$). Such additive reasoning, although it provides a useful connection between multiplication and addition, may not be meaningfully interpreted for multiplying two proper fractions, which would produce a smaller fraction. This may explain why over half of students encountered challenges in providing a real-life scenario to represent the mathematical operation $1/4 × 3/4$ and to depict $1/4 × 3/4$ through a drawing when responding to the "What do you think 1/4 × 3/4?" task in this topic.

## What can be learned from Topic 4?

In this topic, students' responses to the "What is a percentage?" task (see Table 4) highlighted their struggles in articulating their reasoning behind percentages. However, their responses to the "How are they related?" task revealed that 28 out of 35 students were capable of converting procedurally between these three different forms. This suggests that the students in this study recognised the quotient construct of a fraction - i.e. $2/5$ means $2 \div 5$ and $0.4$ means $4 \div 10$ - and, when dividing the numerator by the denominator, they had no problem converting from a fraction to a decimal. This proficiency contrasts with Moss's (2005) findings, where over half of the students (sixth and eighth graders in Canada) claimed that "1/8 would be 0.8" when expressed as a decimal. This emphasizes the critical role of the quotient construct in comprehending the connection between fractions and decimals. Moreover, the outcomes of this topic demonstrated students' proficiency in employing various strategies to convert between fractions, decimals, and percentages, suggesting they understood the relationships between three different forms that have identical values.

## Fraction proficiency tasks for formative assessment of students' comprehension of fractions

Formative assessment takes various forms, aiding both students and teachers in evaluating learning objectives and adjusting instruction. Fraction proficiency tasks, as demonstrated earlier, can serve as
formative assessments, enabling students to demonstrate their skills and identify errors and misconceptions in fractions. Within the classroom, these tasks can seamlessly integrate into ongoing formative assessment practices, allowing teachers to gain insights into student progress, deepen understanding of fractions, and address individual learning needs efficiently. Teachers observe student engagement, provide immediate feedback, and encourage self-assessment, fostering metacognitive skills and ownership of learning. Peer assessment can further enhance learning by providing diverse perspectives and collaborative feedback (Black et al., 2003).

## Limitations, implications and directions for future research

This study focused on a specific mathematical area – fractions – and it only examined students’ understanding of fractions based on the Tsai and Li’s (2017) framework. It is recognized that various other aspects pertaining to fractions might not have been incorporated in these tasks; also, other related factors might not have been taken into account. The findings are confined by the constraints of the employed methodology as well as the limitations inherent in the sample. However, the fraction proficiency tasks presented in this study offer valuable insights into assessing primary students’ understanding of fractions. They provide a comprehensive view of the diversity in students’ understanding and the extent of these differences.

This study shows that the difficulties encountered by students in the present study resonate with those identified in previous studies. This suggests that fractions continue to pose challenges for students, even among Taiwanese students who are consistently recognized as high-performing in large-scale mathematics comparative assessments such as TIMSS and PISA. The findings of this study also offer assessment resources for teachers to gain a clearer understanding of what students should attain and what areas they need to develop. This assists in integrating diverse aspects of fraction knowledge, aiding both students and teachers in comprehending fractions more effectively.

Another implication of this study for fraction assessment involves reconsidering the role of assessment in contributing to a broader understanding of students' grasp of fractions and their mathematical knowledge overall. As argued by Saxe et al. (2013), fractions-related topics are often seen as disconnected. Therefore, further research is needed, particularly in formative assessment, where evaluating a comprehensive understanding of fractions across multiple facets should take precedence over isolating one facet from the others.

## References

- Black, P., Harrison, C., Lee, C. Marshall, B., & William D. (2003) *Assessment for learning*, Maidenhead: Open University Press.

- Chan, W.-H., Leu, Y.-C., & Chen, C.-M. (2007). Exploring group-wise conceptual deficiencies of fractions for fifth and sixth graders in Taiwan. *The Journal of Experimental Education, 76*(1), 26–57. https://doi.org/10.3200/JEXE.76.1.26-58

- Cramer, K. A., Post, T. R., & delMas, R. C. (2002). Initial fraction learning by fourth- and fifth-grade students: a comparison of the effects of using commercial curricula with the effects of using the rational number project curriculum. *Journal for Research in Mathematics Education, 33*(2), 111–144.
Kieren, T. E. (1988). Personal knowledge of rational numbers: its intuitive and formal development. In J. Hiebert & M. Behr (Eds.), *Number concepts and operations in the middle grades* (pp. 162–181). National Council of Teachers of Mathematics.

Lenz, K., Reinhold, F., & Wittmann, G. (2022). Topic specificity of students’ conceptual and procedural fraction knowledge and its impact on errors. *Research in Mathematics Education*, https://doi.org/10.1080/14794802.2022.2135132

Lesh, R. (1981). Applied mathematical problem solving. *Education Studies in Mathematics, 12*, 235–264. https://doi.org/10.1007/BF00305624

Levenson, E., Tirosh, D., & Tsamir, P. (2004). Elementary school students’ use of mathematically-based and practically-based explanations: the case of multiplication. *Proceedings of the 28th conference of the international group for the Psychology of Mathematics Education (PME)*, 3, 241–248, https://eric.ed.gov/?id=ED489578

Moss, J. (2005). Pipes, tubes, and beakers: new approaches to teaching the rational-number system. In M. S. Donovan & J. D. Bransford (Eds.), *How students learn: history, mathematics, and science in the classroom* (pp. 309–350). National Academies Press.

Saxe, G. B., Diakow, R., & Gearhart, M. (2013). Towards curricular coherence in integers and fractions: a study of the efficacy of a lesson sequence that uses the number line as the principal representational context. *ZDM-International Journal on Mathematics Education, 45*, 343–364. https://doi.org/10.1007/s11858-012-0466-2

Soni, M. & Okamoto, Y. (2020). Improving children’s fraction understanding through the use of number lines. *Mathematical Thinking and Learning, 22*(3), 233–43.
https://doi.org/10.1080/10986065.2020.1709254

Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data. *American Journal of Evaluation*, 27(2), 237–46. https://doi.org/10.1177/1098214005283748

Thompson, P. W., & Saldanha, L. A. (2003). Fractions and multiplicative reasoning. In J. Kilpatrick, W. G. Martin & D. Schifter (Eds.), *A research companion to principles and standards in school mathematics* (pp. 95–113). National Council of Teachers of Mathematics.

Tsai, T.-L. & Li, H.-L, (2017). Towards a framework for developing students' fraction proficiency. *International Journal of Mathematical Education in Science and Technology, 48*(2), 244–255, https://doi.org/10.1080/0020739X.2016.1238520

Yang, D. C. (2005). Number sense strategies used by 6th-grade students in Taiwan. *Educational Studies, 31*(3), 317–333. https://doi.org/10.1080/03055690500236845
# Elaborated feedback for interpreting graphs in Secondary Education

María Sanz-Ruiz¹, Luis Miguel Soto-Sánchez¹, Zaira Ortiz-Laso², José Manuel Diego-Mantecón¹

¹Universidad de Cantabria, Spain; sanzm@unican.es, luis-miguel.soto@alumnos.unican.es, diegojm@unican.es

²Universidad de Alicante, Spain; zaira.ortiz@ua.es

Interpreting and extracting information from graphs can be challenging for secondary education students. While feedback often yields positive effects in correcting student errors, there remains a gap in understanding how prior knowledge influences performance in the context of functions and their reactions to feedback. In this study, 68 students solved a task with a graph through an electronic assessment tool (STACK) that allows multiple opportunities to solve it and progressively provides feedback for arriving at the solution. Our results reveal that most medium achievers made a standard error and overcame it after receiving one or two hints, whereas low achievers committed uncommon mistakes and required at least three hints. Our study also shows different reactions to feedback; low achievers were likely to feel overwhelmed when connecting the feedback provided across multiple attempts as the information was presented individually.

Keywords: Electronic assessment, feedback, functions, graphs, STACK.

## Graphs and feedback through e-assessment

Interpreting graphs is essential for secondary education students to tackle real-life situations and advance academically (Planinic et al., 2012). Research has revealed difficulties around this topic (Graham & Sharp, 1999; Ortiz-Laso, 2017), often concerning the identification of which graph features should be used to extract information (Graham & Sharp, 1999). In this line, Ruchniewicz and Barzel (2019) outlined that electronic assessment (e-assessment) tools help students reflect on graph interpretation and attain the knowledge needed to solve a mathematical task.

Task-related knowledge can be delivered through diverse feedback varying from simple to elaborated. Simple feedback relates to how well a task has been performed (Narciss et al., 2022) and comprises three types: knowledge of result (KR), knowledge of performance (KP), and knowledge of correct result (KCR; Narciss, 2008). KR provides information about response correctness (e.g., correct or incorrect), KP gives the number of correct responses, and KCR delivers the correct task solution. Elaborated feedback delivers concise information and can be divided into five types: knowledge on task constraints (KTC), knowledge about concepts (KC), knowledge about mistakes (KM), knowledge on how to proceed (KH), and knowledge on metacognition (KMC) (Narciss, 2008). KTC clarifies task nature, subtasks, processing rules, and requirements, for example “The first step of the correct solution would be…” (Pinkernell et al., 2020, p. 223). KC delivers conceptual information to reach the solution, such as offering a mathematical definition, whereas KM identifies mistakes’ location, type and origins, for example “You probably made this error…” (Pinkernell et al., 2020, p. 223). KH guides the responder to the right solution, correcting specific mistakes and presenting hints and examples, like “Do not ignore the cards that are negative instances of the given concept, as they provide useful information” (Narciss, 2013, p. 19). Finally, KMC offers guiding questions attracting attention to metacognitive strategies (Narciss et al., 2022).
Previous studies on elaborated feedback have investigated its usefulness according to students' previous knowledge. Fyfe and Rittle-Johnson (2016) state that both high- and low-achievers benefit from elaborated feedback, whereas Pinkernell et al. (2020) claim that only low achievers improve. The type of elaborated feedback also influences its effectiveness; Pinkernell et al. (2020) show that among German low-achieving secondary education learners, the benefits derived from KM or KTC surpassed those obtained from KH.

## Research focus and methods

To shed light on the effectiveness of elaborated feedback, this study investigated how secondary education students request and react to feedback provided through an e-assessment tool and how it influences attaining the correct solution. A sample of 68 students from two high schools in Cantabria, Spain, was selected. Students were in the third and fourth grades of compulsory secondary education, aged 14 to 16. They received instruction through textbooks on functions and graphs and represented three achieving groups: low, medium, and high.

## Proposed task and designed feedback

To assess our objectives, we adopted one of the tasks of Ortiz-Laso's (2017) graph, illustrated in Figure 1. In her research, student responses were classified into three sets: correct answers (100 min), expected error (120 min; students did not realize that Juan was not moving between minutes 50-70), and unexpected errors (non-typical errors related to students' lack of knowledge on graphs).

Juan leaves home to exercise in a mountain zone. He starts walking at his usual pace and then alternates between running and walking at different paces. The graph in Figure represents his activity, where the x-axis is time (in minutes) and the y-axis is the distance from home (in kilometers). How much time does Juan spend in motion?

<figure><img src="image_60.png" /><figcaption>Figure 1. Juan's distance from home</figcaption></figure>

To deliver elaborated feedback on the errors classified by Ortiz-Laso, we considered three Narciss' (2008) categories (KR, KH, and KC) to be displayed through STACK (System for Teaching and Assessment using a Computer Algebra Kernel). This tutoring system was chosen because it dispenses specific feedback for each error and tracks every response (Sangwin, 2013). KR feedback was designed for correct answers, whereas KH and KC were devised for the wrong ones (Figure 2). For the expected error (120 min), the solvers received automatic KH to reflect on each graph part (A: *Is Juan not moving at any moment?*). Then, they also could request progressive hints as follows: extra KH (A1: *What does it mean for the graph to have a horizontal part?*), KC related to the constant part of the graph (A2: *The graph indicates the distance between Juan and his home. If the graph is constant during a period, it means that Juan is not moving during that time.*), and further KC incorporating an explanation about the time variable (A3: *Remember that the walk lasted for 120 minutes, and Juan was not moving between minutes 50-70*). For unexpected errors, the solvers got automatic KC that included a graph description (B: *The graph shows the distance between Juan and*
his home at each moment. His route finishes when the distance is 0 again). Finally, they also had the opportunity to request KC and KH to get a task reformulation (B1: *The graph shows Juan was outside for 120 minutes. You are asked how many of those minutes he was moving)*.

<figure><img src="image_61.png" /><figcaption>Figure 2: Offered Feedback</figcaption></figure>

Data collection resulted in a STACK-generated dataset for each student attempt(s), along with cognitive interviews in which students were asked about the reasons behind their responses and reactions to feedback. Although a mixed-method approach was applied, data analysis was qualitative in nature.

## Results and discussion

The analysis revealed that about one-third of students achieved the correct solution in the first attempt. The remaining ones failed on the first attempt, evidencing difficulties related to the interpretation of graphs, as reported by Graham and Sharp (1999) and Planinic et al. (2012). The first answer varied according to students' academic achievement; the medium-achievers normally made the expected error '120 min', while the low-achievers generally gave a set of unexpected responses. The cognitive interviews revealed that the latter responses stemmed from a lack of students' skills to interpret functions; for example, one of the low-achievers stated: "I was unsure about what to reply because I didn't quite understand the graph". Those who replied 120 min in the first attempt understood the task context, but they either interpreted the graph globally or did not comprehend the meaning of having scope 0. In both cases, they provided the correct answer after receiving the first feedback. During the cognitive interviews, one student stressed: "After reading the feedback, I realized that I needed to look at every part of the graph [...]. The solution was not the biggest value reached in the x-axis".

Differences were also observed in how the students reacted to the feedback. Medium-achievers tended to be reluctant to ask for feedback, and instead, they reattempted the task. About half of them got the correct answer after receiving the automatic feedback (A), while the others required an extra hint (A1). In contrast, low-achievers were willing to demand extra information before providing a new answer, and most succeeded after receiving two extra hints. The rest did not request a third one despite not having reached the solution; they reported feeling overwhelmed and frustrated because the hints did not appear simultaneously, having to retain information from previous ones. A student reported: "Some hints were difficult to understand without thinking about the previous ones". In this case, the students were unable to solve the task, but they started to reflect on their own work, being conscious of the need to engage in learning, something already observed by Ruchniewicz and Barzel (2019). The above suggests that when designing feedback, it should be both concise and presented
accumulatively, at least for low-achievers. The present results should be interpreted cautiously due
to the reduced sample and the uniqueness of the task. Further research into learning graphs and e-
assessments is needed to support our findings.

Acknowledgment

This work is supported by the Erasmus+ Programme of the European Union (2021-1-DE01-KA220-HED-000032031; https://www.authomath.org/). It reflects only the authors' view, and the European Commission's support does not imply endorsement of the content and is not responsible for its use.

References

Fyfe, E. R., & Rittle-Johnson, B. (2016). The benefits of computer-generated feedback for mathematics problem solving. *Journal of Experimental Child Psychology, 147*, 140–151.
https://doi.org/10.1016/j.jecp.2016.03.009

Graham, T., & Sharp, J. (1999). An investigation into able students' understanding of motion graphs. *Teaching Mathematics and its Applications, 18*(3), 128–135.
https://doi.org/10.1093/teamat/18.3.128

Narciss, S. (2008). Feedback strategies for interactive learning tasks. In D. Jonassen, M. J. Spector, M. Driscoll, M. D. Merrill, J. van Merrienboer, & M. P. Driscoll (Eds.), *Handbook of research on educational communications and technology* (pp. 125–143). Routledge.

Narciss, S. (2013). Designing and evaluating tutoring feedback strategies for digital learning. *Digital Education Review*, (23), 7–26. https://doi.org/10.5220/0011116400003182

Narciss, S., Prescher, C., Khalifah, L., & Körndle, H. (2022). Providing external feedback and prompting the generation of internal feedback fosters achievement, strategies and motivation in concept learning. *Learning and Instruction*, 82, 101658.
https://doi.org/10.1016/j.learninstruc.2022.101658

Ortiz-Laso, Z. (2017). *Competencia matemática de los alumnos que acceden al Grado en Educación* [Master dissertation, Universidad de Cantabria].

Pinkernell, G., Gulden, L., & Kalz, M. (2020). Automated Feedback at Task Level: Error Analysis or Worked Out Examples–Which Type Is More Effective?. In B. Barzel, R. Bebernik, L. Göbel, M. Pohl, H. Ruchniewicz, F. Schacht, & D. Thurm (Eds.), *Conference on Technology in Mathematics Teaching–ICTMT 14* (pp. 221–228). Duisburg-Essen Publications.

Planinic, M., Milin-Sipus, Z., Katic, H., Susac, A., & Ivanjek, L. (2012). Comparison of student understanding of line graph slope in physics and mathematics. *International journal of Science and Mathematics Education, 10*(6), 1393–1414. https://doi.org/10.1007/s10763-012-9344-1

Ruchniewicz, H. & Barzel, B. (2019). Technology Supporting Student Self-assessment in the Field of Functions – A Design-Based Research Study. In G. Aldon & J. Trgalová (eds.), *Technology in Mathematics Teaching, Selected Papers of the 13th ICTMT Conference* (pp. 49-74). Springer.

Sangwin, C. (2013). *Computer aided assessment of mathematics*. Oxford University Press.
# Development and validation of a three-dimensional framework for the classification of authentic tasks in mathematics

Marta Mikite¹, Ilze France² and Girts Burgmanis³

University of Latvia, Interdisciplinary Centre for Educational Innovation, Riga, Latvia;

¹marta.mikite@lu.lv; ²ilze.france@lu.lv; ³girts.burgmanis@lu.lv;

Global trends in mathematics education place increasing importance on problem-solving skills in authentic contexts. In this paper, the authors propose a three-dimensional framework for the classification of authentic tasks in mathematics. These dimensions, which combine the complexity of the task but can at the same time be analyzed separately, are (1) complexity of the mathematical model, (2) context of the given problem, and (3) strategic complexity connecting the problem and the mathematical model. To validate the framework, the levels of complexity for 12 tasks are determined and students' performance on these tasks is compared. This framework is a valuable tool for designing both learning and assessment tasks.

Keywords: Authentic task, Mathematical problem, Assessment of complexity.

## Introduction

Global trends in learning goals shape the Programme for International Student Assessment (PISA). PISA's latest mathematics framework highlights the importance of mathematics in today's changing world driven by new technologies. After graduation, citizens are expected to be creative and engaged, making non-routine judgments (OECD, 2023a). This means moving beyond the way mathematics is traditionally taught and prioritizing mathematics learning based on real-life examples (Kaiser & Schwarz, 2010). The latest PISA results show that Latvian students perform above average in mathematics at the lower proficiency level, but below average at the higher proficiency level (OECD, 2023b). This indicates that Latvian students need to improve their performance on problem-solving tasks in which they must think without a pre-known algorithm and in which several solutions are possible, so that they need to be more creative and evaluate their ideas. In this situation, national tests are not a driving force either. Previous analyses of Latvian national assessments show a lack of tasks with authentic contexts and show poor indicators of higher-order thinking skills. This study aims to develop and validate a multifunctional framework for designing mathematical problems with authentic contexts at different levels of complexity. A framework would help to build a common understanding of what characterizes higher-level problems in order to promote the development of higher-order thinking skills among students.

## Literature review

There is no common interpretation of what is considered an authentic task or an authentic context. Some authors define it not as a property of the problem, but as a property of the connection between the problem and its solver (Kramarski, Mevarech & Arami, 2002). From this perspective, the same problem will be authentic for some students and not for others. Others stress that authenticity is determined by the fact that problems are purposeful and meaningful (Jurdak, 2006). In this study we assume that tasks in authentic contexts
require a 'real-world' element whether in terms of meaningfulness, relevance and/or application to the personal lifeworlds of learners, as well as an element of connectedness to other subject domains and contexts beyond the textbook and school. (Tan & Nie, 2015, p. 22)

When designing mathematical tasks in authentic contexts, it is important to take into account that their content consists of multiple dimensions. Dimensions are connected in the task but can also be isolated to be analyzed separately. Pugalee and colleagues (2002) identify four dimensions: thinking and reasoning, discourse, mathematical tools, and attitudes and dispositions. Paredes and colleagues (2020) point out three main aspects that should be considered when classifying mathematics tasks: (1) the context in which the task is placed, (2) the variety of responses to the task, and (3) the level of cognitive demand activated when solving the task. Maaß (2010) has studied previously created classification versions and introduced a new, highly detailed scheme for the classification of mathematical modeling tasks. It categorizes tasks based on their characteristics and specific elements. Not all these elements affect the complexity of a task. To create assessment tasks, a framework is needed that outlines how complexity increases. It is crucial for developing an accurate assessment tool to mark the direction of intervention and improve both teaching and learning.

## Methods

In this study three dimensions are distinguished which determine how complex a task is: (1) context of the given problem, (2) complexity of the mathematical model, (3) strategic complexity connecting the problem and the mathematical model (Table 1). According to the PISA 2003 Mathematics framework, each situation is more or less related to the student's world (OECD, 2003). This transfer distance forms the first dimension. The second dimension is the complexity of the mathematical model. The Structure of the Observed Learning Outcome (SOLO) taxonomy's (Biggs & Collis, 1982) unistructural, multistructural and relational levels are the basis for defining this dimension. The third dimension is about the relationship between a given situation and a mathematical model or the ability to formulate, interpret and evaluate (OECD, 2023a).

<table><caption>Table 1: Three-dimensional framework for a classification of authentic tasks in mathematics</caption><thead><tr><th></th><th>Level 1</th><th>Level 2</th><th>Level 3</th></tr></thead><tbody><tr><th>Context of the given problem</th><td>A simple, straightforward, familiar situation, often in a personal context.</td><td>The situation is described using several sources of information. Although the situation is relatively familiar, it requires a deeper understanding of the context.</td><td>Complex, relatively new situation. Situation analysis or generalization is needed</td></tr><tr><th>Complexity of mathematical model</th><td>A simple mathematical model consisting of a single content element.</td><td>Multiple unrelated elements, an algorithm, a learned procedure.</td><td>Multiple related elements, requiring a deep understanding of mathematical concepts.</td></tr><tr><th>Strategic complexity connecting the problem and the mathematical model</th><td>There is a clear solution path, which may be explicitly or implicitly given in the instructions for the task. The problem allows for one correct answer.</td><td>A solution path may be chosen. There is a need to justify/explain the answer as the context allows interpretations. Assumptions need to be made.</td><td>The limitations of the context must be considered, assumptions must be made and the relevance of the mathematical model to the problem must be evaluated. The solution to the situation may differ significantly depending on the mathematical model chosen.</td></tr></tbody></table>
This study is a first validation step to test whether the complexity of the tasks created by the framework increases. The created tasks are part of the pilot study for the national numeracy monitoring in grades 6 and 7. A total of 856 participants took part in the study. The pilot study was conducted using three different item sets. The total number of items is 27, of which 6 are anchor items, identical in all tests. Items were coded based on their mathematics topic – A stands for “ratios and relationships”, G-“geometry”, L-“time and speed”, E-anchor items. The following numbers represent the task number in the students' worksheets. To ensure reliability, coefficient Cronbach's alpha was calculated for each set of results. Tasks with an authentic content were selected by experts according to the following criteria: (1) match at least the first level of the framework in each dimension, (2) fit the Rasch model. 12 tasks were selected from three item sets for the study. The Wright maps were analyzed comparing the position of different level items against the anchor items.

## Results

The calculated Cronbach's alpha coefficients are 0.76; 0.67; 0.76. Considering that this is the initial pilot study, we consider these Cronbach's alpha coefficients to be acceptable to ensure reliability. In Figure 1 all the items selected for the study are framed and the determined complexity level is shown. For example, in 1/2/3 1 indicates level of the context dimension, 2 the level of complexity of the mathematical model, and 3 the level of strategic complexity.

<figure><img src="image_62.png" /><figcaption>Figure 1. Item positioning in Wright maps of three item sets.</figcaption></figure>

If looking at each set and each dimension separately, tasks with a higher level of complexity are positioned higher in the Wright maps, indicating that students' performance decreases with increasing levels of complexity. Item A_6_1 within the first item set does not fit the expected hierarchy in the second dimension – complexity of the mathematical model. This task requires calculating the unknown term of a proportion, which is in the curriculum at exactly the time the test is taken. Some students may have learned this skill, so it could used as a learned algorithm, but some students made up the solution in the given context.
**Conclusions**

The three-dimensional framework for a classification of authentic tasks in mathematics allows to purposefully increase the level of complexity. It is important to have a step-by-step approach in learning, but it is also essential in assessment to design tasks so that their complexity increases gradually to enable as many students as possible to demonstrate their best performance. It is crucial to further develop and implement the framework.

**Acknowledgment**

This research was supported by the project "Development of Data-Driven Solutions for School Improvement" No: lzp-2021/1-0445.

**References**

Biggs, J., Collis, K. (1982). *Evaluating the quality of learning: The SOLO taxonomy*. New York: Academic Press.

Jurdak, M. E. (2006). Contrasting perspectives and performance of high school students on problem solving in real world, situated, and school contexts. *Educational Studies in Mathematics, 63*.

Kaiser, G., & Schwarz, B. (2010). Authentic modelling problems in mathematics education— examples and experiences. *Journal für Mathematik-Didaktik, 31*(1), 51–76.

Kramarski, B., Mevarech, Z. R., & Arami, M. (2002). The effects of metacognitive instruction on solving mathematical authenti up the solution in the given context.c tasks. *Educational studies in mathematics, 49*, 225–250.

Maaß, K. (2010). Classification scheme for modelling tasks. *Journal für Mathematik-Didaktik, 2*(31), 285–311.

OECD (2003). The PISA 2003 assessment framework – mathematics, reading, science, and problem solving. Knowledge and skills. Paris: OECD.

OECD (2023a), "PISA 2022 Mathematics Framework" in PISA 2022 Assessment and Analytical Framework, OECD Publishing, Paris, https://doi.org/10.1787/7ea9ee19-en

OECD (2023b), PISA 2022 Results (Volume I): The State of Learning and Equity in Education, PISA, OECD Publishing, Paris, https://doi.org/10.1787/53f23881-en

Paredes, S., Cáceres, M. J., Diego-Mantecón, J. M., Blanco, T. F., & Chamoso, J. M. (2020). Creating realistic mathematics tasks involving authenticity, cognitive domains, and openness characteristics: A study with pre-service teachers. *Sustainability, 12*(22), 9656.

Pugalee, D. K., Douville, P., Lock, C. R., & Wallace, J. (2002). Authentic Tasks and Mathematical Problem Solving. *Proceedings of the International Conference-The Humanistic Renaissance in Mathematics Education*.

Tan, J. P. L., & Nie, Y. (2015). The role of authentic tasks in promoting twenty-first century learning dispositions. *Authentic problem solving and learning in the 21st century: Perspectives from Singapore and beyond, 19–39*.
# Fostering technology-enhanced formative assessment in Euclidean geometry proving through graded peer tutoring roles

Annamaria Miranda¹ and Loredana Saliceto²

¹,² University of Salerno, Italy;¹ amiranda@unisa.it and ² lsaliceto@unisa.it

In recent years, educators and researchers have paid increasing attention to formative assessment strategies. We explore an experimental design in which formative assessment strategies intertwine with digital technology and graded peer tutoring to overcome secondary school students' difficulties in proving Euclidean geometry statements. According to the design, each group is decomposed into three helping students, acting as guides at various levels, and one student needing to be guided to learn. Each helping student intervenes to activate a specific assessment process in a specific phase of the activity, supported or not by a digital tool, depending on the peer tutoring role to be performed. We investigate the use of digital technology and tutoring roles in supporting agents (teacher, students, peers) to develop formative assessment strategies in teaching and learning Euclidean geometry.

Keywords: Formative assessment, digital tools, roles, graded peer tutoring, proof.

## Introduction and conceptual background

Formative assessment (FA) is widely regarded as one of the more effective instructional strategies employed by teachers, with a growing body of literature and academic research on the topic (Sadler, 1998; Roschelle & Pea, 2002; Irving, 2006; William & Thompson, 2007; Black & William, 2009; Swan & Burkhardt, 2014). FA refers to a wide range of methods used by teachers to conduct in-process evaluations of student understanding, learning needs, and learning progress during a lesson or course. What distinguishes an evaluation as formative is how it is used, i.e., as a method in which evidence about student achievement is elicited, interpreted, and used by teachers, learners, or their peers, to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions they would have taken in the absence of the evidence that was elicited. (Black & William 2009, p. 7)

Research into FA practices has particularly highlighted the role played by so-called *connected classroom technologies* (CCT), networked systems of personal computers or handheld devices specifically designed to be used in classrooms for interactive teaching and learning (Irving, 2006). CCT supports FA due to their specific features that make them effective tools for FA in accomplishing the following: (1) *monitoring students' progress, collecting the content of students' interaction over longer timespans and over multiple sets of classroom participants*; (2) *providing students with immediate private feedback, keeping them oriented on the path to deep conceptual understanding* (Irving, 2006); (3) *encouraging students to reflect and monitor their own progress* (Roschelle & Pea, 2002). Cusi et al. (2017) designed and implemented CCT-supported digital resources, the *worksheets*, to activate FA processes during classroom mathematics activities. The overall goal of FA is to collect detailed information to improve instruction and student learning while it is taking place.

We design an activity in which digital technology and metacognition support agents in activating FA processes to overcome students' difficulties in proving a statement in Euclidean geometry. A primary goal in secondary school is to have students become proficient at writing proofs in Euclidean
geometry. We refer to formal proofs within the Euclidean axiomatic system as deductive arguments showing that the assumptions of a statement logically guarantee the conclusion. However, this goal is rarely met. Many causes for students' difficulties in proving seem to depend both on how to start a proof and how to bridge the gap between informal and formal reasoning (Moore, 1994; Weber, 2001). Research has also shown that using methods of informal reasoning, including visual representations, can have a positive effect on the outcome of students' proof-writing processes. Visualisation of diagrams sketching the statement of a theorem and the production of arguments, even if they are not mathematically rigorous, can lead to identifying the key idea of the proof (Raman, 2003), and the key idea begins to construct a bridge between argumentation and proof. The effectiveness of starting to prove a statement in Euclidean geometry from a visual representation is amplified if the diagram is drawn in dynamical geometry environments (DGEs). The literature regarding the use of DGEs in proof-related activity has paid attention to conjecture generation and the transition from conjecture generation to proof production (Baccaglini-Frank & Mariotti, 2010). Many studies have been conducted to investigate the affordances of DGEs, which include dragging and measuring modalities that result in the generation and testing of hypotheses by generating various diagrams (Olivero & Robutti, 2007). A statement to be proved in secondary school Euclidean geometry is frequently described with reference to a specific diagram representing a certain general class. A diagram, on the other hand, may represent only one case and thus not capture all the configurations to which the statement may refer. As a result, a diagram-based deductive proof may be valid only in that case, and different proofs may be required for different configurations. DGEs can play a significant role in this type of generalisation because their dragging function allows for easy access to multiple diagrams while maintaining the geometrical relationships imposed on the diagrams.

Our approach to achieving an efficient FA when facing a Euclidean proof intends to exploit the benefits DGEs can give students at the beginning phase of a proving process and to do this by actively engaging them in a peer tutoring setting. Moreno & Duran (2002) describe peer tutoring (PT) as a method of cooperative learning based on the creation of pairs of students with an unbalanced relationship; that is, the tutor and the student needing help do not have equal competencies, but they share a common goal. The method can be the most intellectually rewarding experience of a student's career and serves as an effective way to improve self-esteem (Annis, 2013). Our design considers that socialisation experiences that occur during peer tutoring can benefit both the tutor and the needing student by encouraging students to learn at various levels: the interaction with more expert peers plays a crucial role in students' learning, while the tutoring role develops metacognitive competencies. This involves the development of planning, monitoring, and critiquing behaviours—all metacognitive aspects (Schoenfeld, 1992) on which FA must be focused. We report the design and implementation of a cooperative learning activity that engages secondary school students in interacting according to a peer tutoring relationship framework based on the graduation of peer tutoring by roles, with the aim to foster FA processes and help students overcome difficulties in proving in Euclidean geometry.

**Theoretical framework and research questions**

The theoretical framework for our design, implementation, and analysis of the activity finds its roots in the combination of the use of technology to enhance FA practices, through the three-dimensional model of FA (Cusi et al., 2017), and the role of peer tutoring in activating FA processes. William & Thompson (2007) introduce five key strategies for FA practices in school settings (WT strategies):
(a) clarifying and sharing learning intentions and criteria for success; (b) engineering effective classroom discussions and other learning tasks that elicit evidence of student understanding; (c) providing feedback that moves learners forward; (d) activating students as instructional resources for one another; (e) activating students as the owners of their own learning. The three-dimensional model (Cusi et. al, 2017) considers: the five FA key-strategies described by William & Thompson; the three main agents that intervene (teachers, students, peers), and the functionalities through which technology can support the three agents in developing the FA strategies. The teacher, the student's peers, and the student himself or herself are the agents that activate these FA strategies. Through its three functionalities, the technology can assist the three agents in developing FA strategies: (1) sending and displaying, which fosters communication among the agents of FA processes (e.g. sending and receiving messages and files, displaying and sharing screens or documents with the whole class; (2) processing and analysing, which supports the processing and the analysis of the data collected during the lessons (e.g., through the sharing of the statistics of students' answers to polls or questionnaires, the feedback given directly by the technology to the students while taking tests); (3) providing an interactive environment, which creates environments where students can interact to work individually or in groups on tasks or explore mathematical/scientific contents (e.g. through the creation of interactive boards to be shared by the teacher and students, or through the use of specific software that provides an environment in which it is possible to explore). We design a framework in which peer tutoring (PT) is a set of pair relationships in a group graded at different tutoring levels through assigned roles. In PT, one student guides the other in conducting an assignment or learning a concept. In practice, an older student, or someone more experienced, helps a younger or inexperienced student by activating a helping process that accompanies the student in difficulty through various phases of the learning activity: understanding the assignment, exploration of the given problem, bridging to formalization, reflection, and finally evaluation. In this strand, looking at how a tutor behaves when helping a peer, we individuated the helping functions that a helper should activate to support a peer's difficulties, identifying them as specific roles. Students are engaged in group peer tutoring, where each student, except the supported student, is required to play a tutoring role at a specific level. We define a graded peer tutoring (GPT) within the group, that is, 'a decomposed form of peer tutoring that takes shape within the group through assigned roles at various levels, which depend on the various stages to prove the statement'. Personifying a peer tutoring role stimulates critical reflection not only at the cognitive level, as it allows students' engagement in the mathematical problem, but also at a metacognitive level, as it fosters students' monitoring skills related to a role to play in the activity. Specifically, students playing a peer-tutoring role are forced to reflect on how one learns, learn strategies on how to learn, and, by receiving continuous feedback from the student who needs to be helped, not only monitor how she or he learns but also improve the awareness of their own learning.

We face the issue of promoting FA strategies for the development of students' proving competencies in Euclidean geometry by offering them structured tutoring opportunities that allow them to become aware of their own and others' cognitive processes, enabling them to monitor and coordinate them.

RQ: *How can digital technology and metacognitive peer tutoring roles activate formative assessment processes, helping secondary school students overcome difficulties in Euclidean geometry proofs or enhance proving competencies? Specifically, how are these factors perceived by students?*
## Experimental design and definition of roles

The design of the learning activity foresees helping students face a task requiring proving a statement. Students work in groups of four, structured so that in each group there is a student needing help and three helpers interacting supported by digital technology. According to the GPT model we identify three levels of PT within the group, each corresponding to a specific helping phase that stimulates specific actions T1, T2, T3, and gives in-progress FA feedback (c), (d), (e): T1: *exploring and verifying*; T2: *bridging the gap between informal and formal*; and T3: *monitoring and managing the entire helping process*. Specific, prevalent, but not exclusive, corresponding FA tutoring WT strategies to be activated are: (c) -T1 *providing feedback that moves learners forward*; (d) -T2: *activating students as instructional resources for one another*; and (e) -T3 *activating students as the owners of their own learning*. This tutoring structure is well supported throughout the activity by the FA functionalities (1), (2), and (3) (Cusi et al., 2017) put in motion using digital tools. We associate actions, expected FA strategies and functionalities with specific roles. The *Jumper (J)* is the protagonist of the formative assessment activity, the receiver of help, to make a cognitive jump that reduces the knowledge gap between him and the other members of the group; the *Digital Explorer (DE)*, an intermediate-level tutor, masterfully uses digital tools, helps to understand the statement and to explore the dynamical configurations of the related diagram through GeoGebra (action T1- strategy (c) - functionality (1); the *Bridging Mind (BM)*, an intermediate-level tutor with a good knowledge and proof techniques, helps to bridge the gap between informal visual reasoning and formal reasoning (action T2 - strategies (c) and (d) - functionality (1) and (2)); the *Group Leader (GL)* a dual supervisor of the process, intervening to help other colleagues to elaborate the solution, and of the product, and has the skills to do this, high digital skills, excellent logical-deductive skills, and an aptitude for managing (action T3- strategies (c), (d), (e) - functionalities (1), (2), (3)). The assignment of the roles gives students the opportunity to find the way to stay on track (J), deepen their knowledge based on their awareness of what they already know and how to move on, improve their proving skills (DE, BM), learn how to learn (GL). The assignment is based on the students' contextual learning state ascertained by the teacher, and due to its dynamic nature, it changes as the learning states change.

## Methodology

The experiment took place in a 10th-grade class of sixteen students in southwestern Italy during the school year 2017–2018, in Classroom 3.0, an environment with a flexible setting. Students were required to work in small groups, specifically four groups named HG1, HG2, HG3, and HG4, focused on collectively helping one of the components. In each group, HGi, three helpers, DEi, BMi, GLi, and the Jumper Ji were identified by the teacher according to their learning state. Specifically, the Jumper role was mainly assigned to students completely unfamiliar with the subject, having come from other classes in the previous study path. The goal was to produce proof of the following statement (Figure 1), present it and post it on a Shelf Padlet, a work to be collectively reviewed and evaluated. At the end of the activity, all students were asked to express their overall feedback on the experience, the analysis of which would have given rise to the didactic actions to be undertaken later by the teacher.

Draw a circle with diameter AB and centre C and draw the tangent lines in A and B; a third tangent at a point D of the circle intersects the other two at P and Q, respectively. Prove that PQ ≅ PA+QB.

<caption>Figure 1: The statement to prove</caption>
Digital tools supported all phases of the activity. Each HG had at their disposal a digital environment consisting of various tools and resources: an island station equipped with tablets connected to Internet and adjacent to traditional boards. All groups were equipped with tablets, one for each student, to explore the problem through GeoGebra; boards were used to formalise the proof; and Padlet was used to send, display, and share groups' solutions (GeoGebra and board images) and evaluations of displayed solutions presented by the jumpers; and finally, all students' individual feedback.

## Data collection and Data analysis

All the data concerning the helping-learning activity has been digitally stored on shared Padlet boards. The digital environment contains the shelves where students sent and displayed on the LIM: GeoGebra files of dynamic constructions, images of the boards with the proof of the statement, evaluations of the jumpers' performances, personal feedback of the experience, and a photo gallery. Among the collected data we qualitatively analysed evaluations and feedbacks through a systematic and objective identification of some characteristics of FA processes and strategies (identified in the literature) and of the factors triggering them. Specifically, we were looking for signs of the key-FA strategies and functionalities supporting them at the evaluation phase, carried out by the groups, and at the feedback interview on students' individual immediate perceptions of the entire activity, by labelling and classifying sentences according to roles experienced. More in detail, we collected for each Jumper the groups' evaluations and, for each individual role, the impressions of the experience.

## Findings and discussion

To analyse the impact that the designed components have on the activation of students' FA strategies and functionalities, we focus on the evaluations made by groups at the end and on the final interviews.

### Evaluation of Jumpers by Helping Groups

In this phase, the HGs evaluate the Js's performance. Each group evaluates the other Js' performances according to some shared criteria: correctness and completeness of the proof, clarity of the presentation. Ji is not evaluated by the HGi, having already lived FA moments during the tutoring phases. GL coordinates, manages the internal discussion, processes and analyses (functionality (2)) the answers, and displays on the Padlet the evaluation expressed through a brief judgement (action T3, functionality (1)). We highlight the peer formative evaluation by looking at the FA signs in the judgments. The argument about J1 nuances from *not very convincing, quite understandable* to *good*:

* HG2: J1 *argued well* and showed that he understood the problem.

* HG3: [...] explained the proof in a way quite *understandable*.

* HG4: [...] J1's exposition was not very *convincing*, because of the uncertainty about the Theorem to recall, but then an intuitive hint made it clear.

Looking at J3's evaluation, a lack of *self-confidence* appears, but HG4 argues the contrary:

* HG1: [...] has a bit of *hesitancy*, exposed the proof of the problem well.

* HG2: [...] the proof exposed by J3 was not clear, but *thanks to the help of the group*, she was finally able to prove it correctly using logical deductions.

* HG4: [...] the exposure was the most *exhaustive* and *convincing* of all.

According to HGs, J4's presentation was the least successful; despite this, the evaluation does not highlight the failure but tends to justify it. HGs believe that J4 needs to understand that he can make
the jump with another small effort, that the gap is bridgeable, and that, thanks to this experience, he has already managed to make progress and prove to himself that success is attainable. It emerges the educational value that FA strategies can have on fragile students. Students can assume more responsibility for their own learning and progresses when they are aware of their strengths and areas for improvement. Globally, evaluations *provide feedback that moves learners forward* (c); and activates *students as the owners of their own learning* (e) through encouraging judgments and emphasising positive performances, with the aim of trying to remove their initial sense of inadequacy and reduce the gap between them and the tutors.

**Individual feedback organized by role**

All the feedbacks share an appreciation for the use of GeoGebra (functionality (3)) and the organisation in structured PT working groups, and some of them have highlighted other interesting aspects. We begin with significant excerpts from the answers of the Jumpers, the protagonists of the tutoring activity. Signs of *self-formative assessment* take shape when *J*₁ says he has *become more familiar with the subject*. He recognises his learning improvement, and this is feedback that makes him feel like the *owner of his own learning* (e) and *moves him forward* (c) with a new learning jump:

J1: The experience in 3.0 Classroom allowed us cooperative working. It is particularly useful to self-assess and to *become more familiar with the subject*. I believe that these works, aided by digital tools, bring us into contact with the modern world.

J2 exhibits on a subject that he has always refused. He recognises the positive role that his comrades have played in the process as well as the validity of the experience in *moving them forward* (c), not only for those who are in difficulty but also for *already capable* students (WT strategies (c), (d), (e)):

J2: Effective and interesting experience. It has helped them expose themselves to the subject easily, thanks also to the help of friends. It has helped people with some deficiencies and strengthened some already capable knowledge!

J3 seems to suggest structuring the lessons in future like the one just held; both J3 e J4 emphasise the role of technologies *in providing an interactive environment* (functionality (3)) and from a collaborative perspective and the cooperation to help each other (WT strategy (d)):

J3: The experience we had in the 3.0 Classroom gave us a taste of how the lessons should be carried out. You can take advantage of software like GeoGebra that shows the benefits of dynamic geometry, unlike *static geometry*, in solving a geometric problem. In addition, using online platforms increases collaboration, and joining multiple minds to create a single work will surely lead to an optimal result.

J4: It was a genuinely nice and interesting experience. It allowed me to work better because we worked into groups so that we could compare and help each other, also thanks to the use of tablets and the GeoGebra software.

Let us now look at some feedback from helping roles *DE* and *BM*. *DE4* emphasises the importance of the learning environment, intended not only as a physical place but also as a digital interactive place. A *DE* who does not have his own tablet appreciates moving around the classroom equipped with a device. In terms of functionality, it refers to *providing an interactive environment* (3) where students can interact to work individually or in groups on tasks or explore mathematical contents (Cusi et al., 2017), and the software *provides feedback moving students forward* (WT strategy (c)):

DE4: Unlike other activities that usually take place in the classroom, it allows us to learn encouraging comparison with others and the development of more opinions.
It emerges that, although each had their own means, they all worked with digital technologies:

*BM1:* [...] we put ourselves to prove, but we also made use of new technological tools made available by the school that will certainly help us in the future [...]

With the transition from exploration with GeoGebra to axiomatic *BM3* activates an *instructional resource for the Jumper* (WT strategy (d)). *BM3*, attentive to Jumper success, says:

*BM3:* The experience was incredibly positive and productive. Working in a group, each making their own contribution, made me understand that together we can quickly reach a solution. A strong point for me was the drawing conducted with GeoGebra, thanks to which the understanding of the proof was easier and more immediate. I did not find any weaknesses because every group member contributed to the work.

It is interesting to note that cooperation and technology contribute to group and individual growth:

*BM4:* [...] The possibility of cooperating and comparing each other in a constructive way contributes to collective and individual growth.

The Leader, although *engaged* in a tutoring activity (WT strategy (d)), was not bored. He no longer suggests the ordinary lessons because these are more interesting:

*GL2:* [...] interesting and engaging because it allows us to work in an unusual way from ordinary lessons and to collaborate easily thanks to the availability of the material, pushing us to work better and to attend the lessons with more interest.

*GL3* fits the role of leader perfectly, as he gives feedback and moves towards learning to teach, making his colleagues move towards learning. The appreciation of novelty of the method compared to the *usual*, of the *interaction*, of the *comparison with each other* activates FA strategies (d), (e):

*GL3:* [...] A new way of teaching compared to the usual. We were able to interact more with each other and *better express our opinions* and for me it was also much easier because we worked as a group so we could help each other solve problems and explain them to each other, each with their own ideas.

A cross-cutting objective was to increase the responsibility of leaders towards the community since a knowledgeable student is often used to working alone because he or she believes that others can slow down his pace. *GL4* captures this feedback (FA Strategy (d), (e)):

*GL4:* [...] if you are in a group with people, *you are not comfortable with* the activity is counterproductive. However, these activities can be particularly useful to help those who are having difficulty or to *reinforce* concepts that are already known.

The potential power of formative assessment for enhancing teaching and learning in mathematics education is undiscussed and strengthened using digital technology and metacognitive strategies. An attempt to solve the problem of how to help students overcome difficulties in proving Euclidean geometry can be made by designing an ad hoc activity. We design and implement an activity in which digital technology and roles are the driving forces to activate the FA process and functionalities by structuring groups according to *graded peer tutoring* in such a way to produce, at each step, instructions to move forward for the student needing help, supported by digital tools. Regarding the factors triggering FA processes, the analysis highlights that students appreciate working in structured groups, *interacting with each other by working together*, and *learning about new platforms* in a digital technology environment. Regarding the evaluation phase, *HGs* activated the following processes and strategies: *processing and analysing* the solution, *sent and displayed* on Padlet, and the presentation; *using an interactive environment* to express the evaluation; *providing* through the shared evaluation
*feedback that moves learners forward*; and stimulating awareness to be *the owners of their own learning*. The first results are promising. Further research, both theoretical and practical, is needed.

## References

Annis, L. F. (2013). The processes and effects of peer tutoring. *Human Learning, 10*(1), 39–47.

Baccaglini-Frank, A., & Mariotti, M. (2010). Generating conjectures in dynamic geometry: The maintaining dragging model. *International Journal of Computers for Mathematical Learning, 15*(3), 225–253. https://doi.org/10.1007/s10758-010-9169-3

Black, P., & Willam, D. (2009). Developing the theory of formative assessment. *Educational Assessment, Evaluation and Accountability, 21*(1), 5–31. https://doi.org/10.1007/s11092-008-9068-5

Cusi, A., Morselli, F., & Sabena, C. (2017). Promoting formative assessment in a connected classroom environment: design and implementation of digital resources. *ZDM Mathematics Education, Vol. 49*(5), 755–767. https://doi.org/10.1007/s11858-017-0878-0

Irving, K. I. (2006). The impact of educational technology on student achievement: Assessment of and for learning. *Science Educator, 15*(1), pp. 13–20.

Moore, R. C. (1994). Making the transition to formal proof. *Educational Studies in Mathematics, 27*, 249–266. https://doi.org/10.1007/BF01273731

Moreneo, C., & Duran, D. (2002). * framworks: Cooperative and collaborative methods*. Edebè, Barcellona, Spagna.

Olivero, F., & Robutti, O. (2007). Measuring in dynamic geometry environments as a tool for conjecturing and proving. *International Journal of Computers for Mathematical Learning, 12*(2), 135–156. https://doi.org/10.1007/s10758-007-9115-1

Raman, M. (2003). Key ideas: What are they and how can they help us understand how people view proof? *Educational Studies in Mathematics*, 52, 319–325.
https://doi.org/10.1023/A:1024360204239

Roschelle, J., & Pea, R. (2002). A walk on the WILD side: How wireless handhelds may change computer-supported collaborative learning. *International Journal of Cognition and Technology, 1*(1), 145–168. Sadler, D. R. (1998). Formative assessment: Revisiting the territory. *Assessment in Education: principles, policy & practice, 5*(1), 77–84. https://doi.org/10.1080/0969595980050104

Schoenfeld, A. H. (1992). Learning to think mathematically: Problem solving, metacognition, and sense-making in mathematics. In D. Grouws (Ed.), *Handbook for research on mathematics teaching and learning* (pp. 334–370). New York, NY: Macmillan.
https://doi.org/10.1177/002205741619600202

Swan, M. & Burkhardt, H. (2014). Lesson design for formative assessment. *Educational Designer, 2*(7), 1–24. https://nottingham-repository.worktribe.com/output/994366

Weber, K. (2001). Student difficulty in constructing proofs: The need for strategic knowledge. *Educational Studies in Mathematics, 48*(1), 101–119. https://doi.org/10.1023/A:1015535614355

Wiliam, D., & Thompson, M. (2007). Integrating assessment with instruction: What will it take to make it work? In C. A. Dwyer (Ed.), *The future of assessment: Shaping teaching and learning* (pp. 53–82). Mahwah, NJ: Erlbaum.
# CLASSIFICATION tasks as basis for assessment – Family resemblance as principle for choosing mathematical objects

Norbert Noster¹, Arnon Hershkovitz², Michal Tabach² and Hans-Stefan Siller¹

¹University of Würzburg, Germany; norbert.noster@uni-wuerzburg.de, hans-stefan.siller@uni-wuerzburg.de

²Tel Aviv University, Israel; arnonhe@tauex.tau.ac.il, tabachm@tauex.tau.ac.il

Formative assessment is aimed at supporting learning processes. Classification tasks can be used as a basis for formative assessment, particularly for evaluating students' understanding of definitions and object properties. Here, we present six design principles for choosing mathematical objects in such tasks—across grade-levels and across the curriculum—using literature-based ideas about categories of objects, family resemblance, hierarchical structures of categories, and intuitiveness. We demonstrate how we used these principles for designing a digital classification task, and discuss further areas in which these design principles should be considered.

Keywords: Classification, Assessment, Design principles, Mathematics Education

## Introduction

Learning processes in school are evaluated on a regular basis. Such evaluations that take place throughout a learning process are generally referred to formative assessment. Amongst other possible purposes it can be used to evaluate how well a learning process is going and to identify room for improvement. It can be served as aid for the teacher to plan further instructions to address issues that have been identified (Cizek, 2009). Before actions like these can take place, it is necessary to create events that serve as basis for assessment. Developing tasks that can be used for the purpose of formative assessment can be challenging, especially when trying to identify weaknesses in learners understanding of a given topic. Classification tasks have been found useful for assessing understanding (Vollrath, 1977). Utilizing these tasks, this contribution takes a conceptual and theoretical perspective aiming at providing design principles for choosing mathematical objects to be used in classification tasks for formative assessment. In this sense we use it for the purpose of testing for students' conceptions, that can serve as a tool and aid for teachers to plan following instruction. The principles focus on choosing objects for the classification task derived from ideas of classifications and relations between categories. These principles are then used to analyze an example of a digital classification task as well as findings related to that task.

## Classification tasks

Classifying (also referred to as categorizing or sorting) means grouping objects that can be treated equivalent regarding a certain criterion (e.g. any object that can be used to sit on can be classified as a chair) (Rosch, 1978). It is a process that helps us structure our surroundings, for example by differentiating between edible and non-edible or living and non-living things (Richler & Palmeri, 2014; Rosch, 1978). It can be considered part or result of a learning process. Therefore, it is not surprising that it can be found in mathematics curricula all around the world (Mullis et al., 2016). Using classification tasks, that ask learners to classify given objects provide us with insight into their knowledge, which is why they are suitable for the purpose of assessment. Here we concur with
Vollrath (1977), who asserts the importance of classification tasks by stating that they “can make the students conscious of the characterizing properties of the concept, guide them to a definition and control their understanding of the definition” (p. 212).

Depending on the choice of objects to be classified and the choice of a classification criterion, certain features can be highlighted, and student identification of these features may be assessed. Once objects are chosen and a classification criterion is set, there are still three main options for the design of a classification tasks (Vollrath, 1978): 1) Tasks in which the objects are not classified, and the classification criterion is known, e.g., “Given is a set of quadrilaterals; classify each of them based on the existence of a line of symmetry (Has / Does Not Have)”; 2) Tasks in which the objects are classified, and the classification criterion is unknown, e.g., “Given are two groups of quadrilaterals; find the property that all the objects in Group A has and all the objects in Group B do not have”; 3) Tasks in which the objects are not classified, and the classification criterion is unknown, e.g., “Given is a set of quadrilaterals; find a property and divide the objects into two groups so that all the objects in Group A have this property and all the objects in Group B do not have it”.

Different aims can be associated with different types of classification tasks, such as assessing understanding of a property, initializing development of conceptual understanding, and providing an overview of the relationship between objects and properties (Vollrath, 1977). While all three variants can be used for assessing understanding up to an extent, the first one can be used to find out to what extent a property is being identified in a set of given objects. The second one focusses more on deriving a definition from the classified objects, whereas the third variant can lead to different classifications that are not associated with the learning goal. Our focus is on the first variant as it focusses on applying a definition (of the classification criterion) rather than identifying it. Furthermore, this kind of task could be designed digitally with correct and incorrect classifications being automatically identified, so individual assessment could be automated, and immediate feedback to students could be provided (Feldt-Caesar, 2017).

For designing classification tasks, the suiting set of objects and classification criteria need to be well defined, so that they will serve the assessment purposes. This yields our first two design principles.

*Design Principle 1: Define the classification criterion along with its values.* This criterion should be in line with the curriculum and is strongly related to the assessment requirements. Note that classification can be done to two groups or more; for example, we can ask students to classify angles by acuteness (Yes / No) or by their type (Acute, Right, Obtuse, Straight, Reflex). Another example: For a given set of quadrilaterals, we can define a classification criterion “Has (Property of) Reflective Symmetry” (Yes / No), “Has (Property of) Rotational Symmetry” (Yes / No), or “Has This Property of Symmetry” (Reflective / Rotational / None).

*Design Principle 2: Define the set of objects to be classified.* This should be in line with the curriculum and with the assessment requirements. In many cases, it may be easy to rely on pre-defined mathematical sets of objects, e.g., polygons, two-dimensional geometrical shapes, simple fractions, integers, etc. A choice should be taken at this point whether each object would belong to a single classification group. For example, if we chose the classification criterion “Has This Property of Symmetry” (Reflective / Rotational / None), and we focus on quadrilaterals, we should decide whether we want to include rectangles, which can be classified to both groups, or not.
Once classification criterion and the types of objects are chosen, we move on to choosing the specific objects that will be used in the task. For this, we present the notion of family resemblance.

**Relations between categories of objects as the basis for family resemblance**

Classifying objects can be complex as it can be done based on various characteristics (Pothos et al., 2011). In this context, one key term is *category*, which is defined as a group of objects that are considered to be equivalent with respect to a criterion (Rosch, 1978). Importantly, as different classification criteria could be set for the very same set of objects, categories are not given a priori. For example, in the case of classifying quadrilaterals based on the existence of a line of symmetry (Has / Does Not Have), all the rectangles would be under a single category and all the parallelograms would be under single, different category; however, if the classification criterion would be color (Red / Blue), a red rectangle and a red parallelogram would be under a single category while a blue rectangle and a blue parallelogram would be under a single, different category. Even if we limit ourselves to considering only mathematical properties, different classification criteria may yield different categories. For example, think of the set of polynomials and the following objects: x; x + 1; x², x² + 1; classifying them by the criterion “Is a Quadric Polynomial” (Yes / No) will yield the following categories: {x², x² + 1} (Yes), {x, x + 1} (No), while setting the classification criterion “Is a Monomial” (Yes / No) will yield different categories: {x, x²} (Yes), {x + 1, x² + 1} (No). Objects of the same category share family resemblance (Rosch & Mervis, 1975) vis-à-vis the classification criterion. It is important to state that family resemblance refers to the extent to which an objects shares features (including irrelevant ones) with other objects of a category. From here, we derive a third design principle:

*Design Principle 3: Identify categories of objects for each classification group.* These categories should be in line with the curriculum and with the assessment requirements. For example, if we chose to classify based on parity of functions (Even Function / Not Even Function) and we are focused on the polynomials, possible categories for the “Even Function” group could be: monomials with even exponent, or quadric polynomials of the form $ax^2 + c$ (with $c \neq 0$); categories for the “Not Even Function” could be: monomials with odd exponent, or linear polynomials.

In each of these groups, categories can be organized into a hierarchical taxonomy in which the classification criterion is inherited from a broader category to its sub-categories (Bernabeu et al., 2022; Rosch & Mervis, 1975). Constructing this hierarchy will help in identifying family resemblance, hence our fourth design principle:

*Design Principle 4: Construct a hierarchical taxonomy of categories for each classification group.* Think of a task for classifying functions based on parity that was presented in the previous paragraph. The two categories for the “Even Function” group could be seen as stemming from a higher-level category of polynomials with only even exponents to which another category could belong: constant polynomials.

Once the taxonomy for each classification group is set up, we can identify objects with different levels of family resemblance. The longer the path on the hierarchical structure from one category to another, the lower the family resemblance between objects of those categories. To assess understanding of a concept, it is important to include in the task objects of different characteristics, from which we derive another design principle:
*Design Principle 5: For each classification group, choose objects of different levels of family resemblance.*

So far, the choice of objects has been defined by their mathematical characteristics. The final step of choosing objects has to do with the students' point of view, specifically regarding their level of knowledge. For this, we regard the notion of intuition.

**Intuition in mathematics education**

Identifying an object as having a criterion could be done intuitively – that is, immediately, with confidence, without the need to justify this choice (Fischbein, 2002). For example, kindergarten children will identify a 3-sides polygon which has two equal sides and a third side parallel to a horizontal line as a triangle, while at the same time reject it from being a triangle if “it stands on its head” (Sinclair & Moss, 2012). Likewise, a circle would be intuitively rejected by kindergarten children from being a triangle, as it is an intuitive example for a different type of objects (Tsamir et al., 2008). Thus, an object, or a property of an object could be intuitively identified by students based on their prior experiences with these objects. In mathematics education textbooks, for example, geometrical figures are usually presented in an orientation that is parallel to horizontal and vertical lines. Hence such figures would be identified intuitively, while figures oriented differently would be less intuitive to identify (ibid). In other words, the intuitiveness of an object reflects on the level of difficulty it presents to students. For this, we yield the final design principle:

*Design Principle 6: For each classification group, choose both intuitive and non-intuitive objects.*
Note that intuitiveness and non-intuitiveness of mathematical objects is highly sensitive to the types of objects and to the classification criterion in matter, so there cannot be general guidelines as how to design them.

**Example of applying the design principles**

In this section we have a look at a digital classification task that we have developed and studied (Hershkovitz et al., 2023; Noster, Hershkovitz, Siller, et al., 2022; Noster, Hershkovitz, Tabach, et al., 2022). We present it through the lens of the design principles stated above. Considering the curriculum for these grade levels in both Israel and Germany, where we studied using this task, we decided to design the task around the concept of symmetry, specifically reflective symmetry. Following Design Principle 1, we defined the classification criterion and values: Has at Least one Symmetry Line (Has / Has Not); hence this is a two-way classification task. Following Design Principle 2, we chose to focus on quadrilaterals, which are known mathematical objects for children at these ages.

In the context of quadrilaterals, there is a well-established categorization into, e.g., parallelograms, rectangles, squares, trapezoids, kites; these categories appeared in the textbooks of the populations we sampled, hence we based our design on them. Following Design Principle 3, we identified the following categories for the “Has” group: rectangles, squares, and kites; and the following groups for the “Has Not” group: Parallelograms, and Non-Isosceles Trapezoid.

Another well-established framework that was relevant to our choice of objects was the House of Quadrilaterals, which describes hierarchical relationships between different categories of quadrilaterals based on their definitions. This helped us in building taxonomies for the two
classification groups. Following Design Principle 4, we constructed the following hierarchical
taxonomy for the “Has” group: Squares are sub-category of Recatngles and Kites; Parallelograms
and Non-Isosceles Trapezoids are sub-categories of general non-symmetrical quadrilaterals.

Now, following Design Principle 5, we chose objects to represent different levels of family
resemblance. For the “Has” groups, we first chose a square and rectangle, which share high family
resemblance to each other vis-à-vis reflective symmetry, and a kite, which share low family
resemblance with both. For the “Has Not” group, we first chose three objects with low family
resemblance between them: parallelogram, non-isosceles trapezoid, and a general non-symmetrical
quadrilateral.

Finally, following Design Principle 6, we relied on a framework of intuitive and non-intuitive two-
dimensional geometric objects (Tsamir et al., 2008), and added the notion that shapes with lines of
symmetry that are either horizontal or vertical are more likely to be identified as symmetric than
shapes with diagonal lines of symmetry (Götz & Gasteiger, 2022). Squares, rectangles, and kites are
generally intuitively perceived as having a line of symmetry, while the general non-symmetry
quadrilateral was built in a way that it would be intuitive to assume it had no lines of symmetry;
parallelograms are non-intuitive as not having a line of symmetry, as children often mix this notion
with rotational symmetry (which parallelogram do have). We added a tilted square, which is non-
intuitive due to the diagonal lines of symmetry, and a rotated parallelogram, yet another non-intuitive
objects, however, note that these two objects share high family resemblance with the square and
parallelogram, respectively. See Figure 1.

<figure><img src="image_63.png" /><figcaption>Figure 1: Examples for a digital classification task to differentiate between quadrilaterals with and without reflective symmetry</figcaption></figure>

Discussion

This contribution aims at providing design principles for choosing mathematical objects to be used
in classification tasks for formative assessment; these principles may be implemented across grade-
levels and across topics. We focused particularly on classification tasks in which a set of objects is
given along with a classification criterion, as this type of tasks provide a productive arena for
assessing individual students’ knowledge and can be easily assessed automatically. Based on the
literature, we have identified the following design principles, that can guide the choice of the
mathematical objects to be classified:
1. Define the classification criterion along with its values;

2. Define the set of objects to be classified;

3. Identify categories of objects for each classification group;

4. Construct a hierarchical taxonomy of categories for each classification group;

5. For each classification group, choose objects of different levels of family resemblance;

6. For each classification group, choose both intuitive and non-intuitive objects.

While explaining and demonstrating each principle as a “standalone”, we also demonstrate the application of these principles together, to compose a classification task in a digital environment. Of course, these design principles should also be tested empirically. For example, the notion of intuitiveness of objects needs to be taken into field test, and cannot be based solely on the designers’ assumptions (Noster, Hershkovitz, Tabach, et al., 2022). An iterative task design should follow, to verify the applicability of the design which resulted from applying the principles. Large data collection and its analysis could further inform the design choices.

As we are aiming at digital assessment tasks, there are also other aspects to be considered, to which we have not referred. These include for example the issue of feedback use. A digital environment could provide feedback of different types: simple vs. elaborated; immediate vs. delayed; or feedback on correctness vs. on strategies being implemented (Attali & van der Kleij, 2017; Shute, 2008; Tärning, 2018). To the best of our knowledge, there are no conclusive guidelines in the literature as for the most effective combination of these options for the purpose of serving assessment. This is an avenue for further research we are planning to pursue.

Another issue to be considered and tested relates to the layout of the objects on the screen. This may involve aspects like arrangement of the classification areas, e.g., unclassified objects are placed in the middle, in between the classification areas vs. to the left or right of the classification areas; arrangement of the objects, e.g., on a grid vs. randomly located; or issues related to size and color. We consider it an empirical question that needs to be tested based on large data collection, preferably in a set of randomized controlled studies, which again could lead to an iterative process of refining the design. To this we shall add more traditional design issues related to user interface, which are crucial in digital learning environments (Park & Song, 2015; Sagrario & Simbulan, 2007).

While the considerations presented here provide principles for choosing objects for classification tasks, it should be stated that this is only the first step in task design, as depending on the concrete task the effects may vary. Task design should be iterative (Liljedahl et al., 2007), but our principles serve as a foundation for a first predictive analysis, which has proven to be useful in the reflection of the data.

## Acknowledgment

We thank the Bavarian Research Alliance and their Funding Programme for the Initiation of International Projects (BayIntAn) as well as their Scholarship Program for Bavarian and Israeli Early Career Researchers for supporting this project.
## References

Attali, Y., & van der Kleij, F. (2017). Effects of feedback elaboration and feedback timing during computer-based practice in mathematics problem solving. *Computers & Education, 110*, 154–169.
https://doi.org/10.1016/J.COMPEDU.2017.03.012

Bernabeu, M., Moreno, M., & Llinares, S. (2022). Preservice primary teachers’ curricular reasoning when anticipating primary students’ answers to geometrical figure classification tasks. *Proceedings of the Twelfth Congress of the European Society for Research in Mathematics Education*.
https://hal.science/hal-03751484

Cizek, G. J. (2009). An introduction to formative assessment: History, characteristics, and challenges. In H. Andrade & G. J. Cizek (Eds.), *Handbook of formative assessment* (pp. 3–17). Routledge.
https://doi.org/10.4324/9780203874851-6

Feldt-Caesar, N. (2017). Konzeptualisierung und Diagnose von mathematischem Grundwissen und Grundkönnen. *Konzeptualisierung Und Diagnose von Mathematischem Grundwissen Und Grundkönnen*. https://doi.org/10.1007/978-3-658-17373-9

Fischbein, E. (2002). *Intuition in Science and Mathematics: An educational approach*. Kluwer Academic Publishers. https://doi.org/10.1007/0-306-47237-6

Götz, D., & Gasteiger, H. (2022). Reflecting geometrical shapes: Approaches of primary students to reflection tasks and relations to typical error patterns. *Educational Studies in Mathematics, 111*(1), 47–71. https://doi.org/10.1007/s10649-022-10145-5

Hershkovitz, A., Tabach, M., Noster, N., & Siller, H.-S. (2023). Student behavior while engaged with feedback-enhanced digital sorting tasks. In M. Ayalon, B. Koichu, R. Leikin, L. Rubel, & M. Tabach (Eds.), *Proceedings of the 46th Conference of the International Group for the Psychology of Mathematics Education (Volume 3)* (pp. 51–28). PME.

Liljedahl, P., Chernoff, E., & Zazkis, R. (2007). Interweaving mathematics and pedagogy in task design: A tale of one task. *Journal of Mathematics Teacher Education, 10*(4–6), 239–249.
https://doi.org/10.1007/s10857-007-9047-7

Mullis, I. V. S., Martin, M. O., Goh, S., & Cotter, K. (Eds.) (2016). TIMSS 2015 Encyclopedia: Education Policy and Curriculum in Mathematics and Science.

Noster, N., Hershkovitz, A., Siller, Hans.-S., & Tabach, M. (2022). Students’ strategies for identifying reflective symmetry of extra-mathematical shapes in a digital environment. *ERME Topic Conference on Mathematics Education in the Digital Age*.

Noster, N., Hershkovitz, A., Tabach, M., & Siller, H.-S. (2022). Learners’ strategies in interactive sorting tasks. In I. Hilliger, P. J. Muñoz-Merino, T. De Laet, & A. F. T. Ortega-Arranz (Eds.), *Educating for a New Future: Making Sense of Technology-Enhanced Learning Adoption*. EC-TEL 2022. Lecture Notes in Computer Science, vol 13450 (pp. 285–298). Springer.

Park, H., & Song, H. D. (2015). Make e-learning effortless! Impact of a redesigned user interface on usability through the application of an affordance design approach. *Educational Technology & Society, 18*(3), 185–196.
Pothos, E. M., Perlman, A., Bailey, T. M., Kurtz, K., Edwards, D. J., Hines, P., & McDonnell, J. V. (2011). Measuring category intuitiveness in unconstrained categorization tasks. *Cognition, 121*(1), 83–100. https://doi.org/10.1016/j.cognition.2011.06.002

Richler, J. J., & Palmeri, T. J. (2014). Visual category learning. *WIREs Cognitive Science, 5*(1), 75–94. https://doi.org/10.1002/wcs.1268

Rosch, E. (1978). Principles of Categorization. In E. Rosch & B. B. Lloyd (Eds.), *Cognition and Categorization* (pp. 27–48). Lawrence Erlbaum.

Rosch, E., & Mervis, C. B. (1975). Family resemblances: Studies in the internal structure of categories. *Cognitive Psychology, 7*(4), 573–605. https://doi.org/10.1016/0010-0285(75)90024-9

Sagrario, M., & Simbulan, R. (2007). Learning objects’ user interface. In A. Koohang & K. Harman (Eds.), *Learning objects: Theory, praxis, issues, and trends* (pp. 259–336). Informing Science Press.

Shute, V. J. (2008). Focus on Formative Feedback. *Review of Educational Research, 78*(1), 153–189.

https://doi.org/10.3102/0034654307313795

Sinclair, N., & Moss, J. (2012). The more it changes, the more it becomes the same: The development of the routine of shape identification in dynamic geometry environment. *International Journal of Educational Research, 51–52*, 28–44. https://doi.org/10.1016/j.ijer.2011.12.009

Tärning, B. (2018). Review of feedback in digital applications - Does the feedback they provide support learning? *Journal of Information Technology Education: Research, 17*, 247–283.
https://doi.org/10.28945/4104

Tsamir, P., Tirosh, D., & Levenson, E. (2008). Intuitive nonexamples: The case of triangles. *Educational Studies in Mathematics, 69*(2), 81–95. https://doi.org/10.1007/s10649-008-9133-5

Vollrath, H. J. (1977). The understanding of similarity and shape in classifying tasks. *Educational Studies in Mathematics* 1977 8:2, 8(2), 211–224. https://doi.org/10.1007/BF00241026

Vollrath, H. J. (1978). Klassifikation nach Ähnlichkeit. *Der Mathematikunterricht*, 24(2), 105–115.
# Effects of a teacher professional development program in formative assessment on teachers' conceptions of feedback and assessment and their self-reported feedback practices

Torulf Palm<fnref n="1" />, Gavin Brown<fnref n="2" />,<fnref n="3" />, and Björn Palmberg<fnref n="4" />

Formative assessment (FA) has been shown to have the power to improve student achievement. Therefore, many professional development (PD) initiatives have been carried out to support teachers to develop their FA practice. However, accomplishing such practices have been proven difficult. Among factors that are important for outcomes of PDs are teachers' beliefs and conceptions. This study examines the effects of a PD in FA on teachers' conceptions of assessment and feedback, and self-reported feedback practices. These variables were measured through a survey in the beginning and at the end of the PD. Differences between the intervention and a control group were examined at both time points using factor analytic methods and with t-tests on change scores. For the intervention group, significant positive differences were found in both the means of important conceptions and in the strength of relationships, while this was not the case for the control group.

Keywords: Beliefs, conceptions, feedback, formative assessment, professional development.

## Introduction

Formative assessment (FA) is a classroom practice in which teachers and/or students elicit evidence of students' learning needs through assessment and then adapt teaching and/or learning accordingly. It has been shown to have the potential to improve student achievement (e.g., Baird et al., 2014). However, although some professional development programs (PDs) have succeeded in helping teachers accomplish FA practices that improve student learning (e.g., Andersson & Palm, 2017), they have often been unsuccessful in accomplishing substantial improvements in teachers' FA (e.g., Randel et al., 2016). Teachers' conceptions of, or beliefs about, assessment and feedback are among factors that affect implementation of FA components (Brown et al., submitted). However, studies examining effects of PDs on teacher conceptions of assessment and feedback are rare, but e.g., Deneen and Brown (2016) did not find effects from a course on assessment literacy on teachers' conceptions of assessment. This paper focuses on the effects of a PD program in FA on mathematics teachers' conceptions of assessment and feedback, and self-reported feedback practices.

<footnote n="1">¹Umeå University, Umeå Mathematics Education Research Centre (UMERC), Department of Science and Mathematics Education, Sweden; torulf.palm@umu.se</footnote>

<footnote n="2">²Umeå University, Department of Applied Educational Science, Sweden; gavin.brown@umu.se</footnote>

<footnote n="3">³The University of Auckland, Faculty of Education and Social Work, New Zealand; gt.brown@auckland.ac.nz</footnote>

<footnote n="4">⁴Umeå University, Umeå Mathematics Education Research Centre (UMERC), Department of Science and Mathematics Education, Sweden; bjorn.palmberg@umu.se</footnote>
# Methods

## Design

The PD ran as an experimental intervention with control group. A pre- and post-experiment survey was conducted in a northern Swedish city with a large control group and a small experimental group. Differences between groups were examined at both time points using factor analytic methods.

## Participants

A total of 461 teachers working between school years 1 and 9 responded to the survey. Among them, 257 teachers responded to the survey at both times. They were matched between time 1 (2021) and time 2 (2023) so the variance over time could be properly evaluated. Expectation maximization was used to impute the small amount of missing data.

## Professional development intervention

The PD was designed based on reviews on characteristics important for PD outcomes (e.g., Heitink et al., 2016) and on our own previously arranged PDs (e.g., Andersson & Palm, 2017). It was organized by a research team led by the first author. The researchers and the teachers met once a month during 3-6 hours for three years. The teachers also met by themselves once a month. The meetings included lectures about FA and concrete activities for its implementation, as well as group discussions and analysis of the content and suggested activities. General FA strategies were concretized for mathematics (e.g., what effective questioning would look like in mathematics). Time was also put aside for the teachers to plan for implementing FA activities in their classrooms. In the following meeting the teachers evaluated the try-outs, shared experiences of success and discussed how they could overcome obstacles and develop the use of a particular activity. The researchers supported these discussions and intervened with suggestions when deemed useful. The teachers were supported in their self-regulated learning of FA by being provided an evaluation tool, and time to use it, for evaluating and setting goals for their practices. Generally, the programme possessed a formative, process-oriented character and provided support for the teachers to influence the program.

## Instruments

The Swedish Teachers Conceptions of Assessment inventory (TCoA) measures three major constructs. For the purposes of this study, only the conception that assessment improves teaching and learning (Assessment Improves) was selected as it was most sensitive to the impact of the formative assessment PD program and is likely to facilitate implementation of FA. This Assessment Improves factor has 4 1st-order factors (i.e., assessment helps teachers improve teaching, assessment helps students improve learning, assessment is reliable, and assessment is diagnostic).

The Swedish Teachers Conceptions of Feedback inventory (TcoF) (Brown et al., 2023) consists of six feedback conceptions and a self-reported Formative Feedback Practices factor. The Formative Feedback Practices factor is predicted by the two feedback conceptions Feedback Improves Performance and Students Ignore Feedback. Of the seven factors in the TcoF, only four were retained in this study (i.e., Students Ignore Feedback, Feedback Improves Performance, Feedback Involves Students in Peer and Self-feedback, and Formative Feedback Practices) as they were most likely to be sensitive to the professional development and are likely to facilitate implementation of FA.
## Data analysis

The model we used had assessment conceptions predicting feedback conceptions and practices on the assumption that feedback generally occurs after assessment events (Hattie & Timperley, 2007). To account for the repeated measures design, a cross-lagged, bivariate path model with autoregressive paths (Curran & Bollen, 2001) was tested. Within each time point, the assessment conceptions factor (Assessment improves) with four dependent scores was regressed onto a general feedback factor. This general feedback factor had three dependent scores from the retained feedback conceptions factors, and was regressed onto the Formative Feedback Practices factor. A path from Student Involvement in Feedback factor to Formative Feedback Practices was added. Autoregressive paths from each variable at Time 1 were added to the matching variable in Time 2. No cross-lag paths from Assessment or Feedback factors at Time 1 to Time 2 could be identified. Hence, the model could be described as a structural path model within time with auto-regressive paths across time. To compare the model between the two groups, nested invariance testing was conducted (Brown et al., 2017). Structural equation modeling (SEM) and invariance testing were conducted with AMOS v29.0.0 (IBM, 2022).

## Results

Prior to multi-group analysis, the assessment to feedback model with autoregression was found to have acceptable to good fit for the whole group. Also, the input model for the two-group analysis had acceptable fit. Invariance testing showed that measurement weights were not equivalent between groups. Hence, the two groups differed at the unconstrained level, indicating that they were drawn from two separate populations.

The intervention group differed from the control group in significant and substantial ways that were most notable after the intervention itself. Based on a t-test of differences on the change score from pre- to post-intervention time point, the intervention group gained substantially (Cohen's *d* ≥ .50) for Assessment Helps Students Improve; Assessment is Reliable, Feedback Improvement, and Formative Feedback Practices variables. These differences in changes were due to that the intervention group's conceptions of assessment and feedback as well as their feedback practices moved substantially in favour of formative assessment, while the means for the control group fundamentally remained constant. Equally notable, there was a positive shift in the strength of the relationships from the assessment and feedback latent factors to their respective items only in the intervention group. Throughout, the control group, as would be expected without any focused professional development, did not change in means or model path values.

## Discussion

In contrast to the lack of effects found from the assessment course studied by Deneen and Brown (2016), the PD in this study improved teachers' conceptions of assessment and feedback, and their formative feedback practices. Which PD features that were decisive for the effects cannot be determined from the study. However, giving the teachers both time and support for planning implementation in their classes together with support to overcome difficulties may have played a role. Also, the rather substantive length of the PD may have been a factor since belief change often takes time and occurs gradually. These are features found to be important for positive outcomes of PD in FA (Heitink et al., 2016). Finally, the formative character of, and the support for teachers to influence,
the PD together with support for taking individual and collective responsibility for their own learning through self-regulated learning processes may have contributed to the positive outcomes. Beliefs are commonly measured through questionnaires, but a limitation of this study is that the teachers' practices are self-reported and not observed. Future studies that include observations of teacher practices and a focus on which PD features that are decisive for outcomes would be valuable.

## References

Andersson, C., & Palm, T. (2017). The impact of formative assessment on student achievement: A study of the effects of changes to classroom practice after a comprehensive professional development programme. *Learning and Instruction, 49*, 92–102.
https://doi.org/10.1016/j. learninstruc.2016.12.006

Baird, J., Hopfenbeck, T., Newton, P., Stobart, G., & Steen-Utheim, A. (2014). *State of the field review: Assessment and learning*. Oslo: Report for the Norwegian Knowledge Centre for Education, case number 13/4697. Retrieved from http://forskningsradet.no

Brown, G. T. L., Andersson, C., Winberg, M., & Palm, T. (2023). Predicting formative feedback practices: Improving learning and minimising a tendency to ignore feedback. *Frontiers in Education, 8*. https://doi.org/10.3389/feduc.2023.1241998

Brown, G. T. L., Harris, L. R., O'Quin, C., & Lane, K. E. (2017). Using multi-group confirmatory factor analysis to evaluate cross-cultural research: identifying and understanding non-invariance. *International Journal of Research & Method in Education, 40*(1), 66-90.
https://doi.org/10.1080/1743727X.2015.1070823

Curran, P. J., & Bollen, K. A. (2001). The best of both worlds: Combining autoregressive and latent curve models. In L. M. Collins & A. G. Sayer (Eds.), *New methods for the analysis of change* (pp. 107–135). American Psychological Association. https://doi.org/10.1037/10409-004

Deneen, C. C., & Brown, G. T. L. (2016). The impact of conceptions of assessment on assessment literacy in a teacher education program. *Cogent Education, 3*, 1225380.
https://doi.org/10.1080/2331186X.2016.1225380

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.
http://dx.doi.org/10.3102/003465430298487

Heitink, M. C., Van der Kleij, F. M., Veldkamp, B. P., Schildkamp, K., & Kippers, W. B. (2016). A systematic review of prerequisites for implementing assessment for learning in classroom practice. *Educational research review, 17*, 50-62.

Randel, B., Apthorp, H., Beesley, A., Clark, T., & Wang, X. (2016). Impacts of professional development in classroom assessment on teacher and student outcomes, The Journal of Educational Research, 109:5, 491-502, https://doi.org/10.1080/00220671.2014.992581
# Analytical readings' method as a tool to assess the word problem-solving process involving rational numbers

Sofia Peregrina Fuentes¹, Carlos Valenzuela García², María Teresa Sanz³ and Emilia López-Iñesta⁴

¹University of Guadalajara, Mexico; sofia.peregrina5508@alumnos.udg.mx

²University of Guadalajara, Mexico; carlos.valenzuela@academicos.udg.mx

³University of Valencia, Spain; m.teresa.sanz@uv.es

⁴University of Valencia, Spain; emilia.lopez@uv.es

Word problem solving in mathematics is essential, and the processes that students follow to solve them may vary. Consequently, teachers face the challenge of interpreting these strategies, while students sometimes have difficulties explaining their processes. The analytical readings' method is presented as a resource for the assessment of the word problem-solving processes, in addition to helping students develop skills in this area. In this context, the purpose of this study is to document reflection, especially among undergraduate mathematics students with an interest in teaching, on analytical readings as a tool to assess problem solving. Reflections highlight that the analytical readings' method allows students to comprehend the problem, their performed operations, the reasoning behind them, providing a comprehensive view of the thinking process.

Keywords: Word problems, analytical readings, word problem-solving process, assessment, rational numbers.

## Introduction

Word problem-solving is considered a basic competence that promotes the development of arithmetic and algebraic thinking (Siegler et al., 2013). Rodríguez (2012) indicated that when solving a word problem, the student is allowed to “explore, experiment, analyse their progress, change course, reflect on what they have done, notice how they are thinking and approaching the task, etc.” (p. 154).

Currently, word problems are an essential part of the mathematics education of every student around the world, and consequently, teachers must have resources to assess the solving process of these word problems. Therefore, our main objective is to promote the use of the analytical readings' method as a tool to assess the process of solving word problems with rational numbers, for this purpose the reflections of undergraduate mathematics students interested in teaching were documented.

It is clarified that it is not intended to promote the method as the best option or to say that it is the most efficient way of evaluation, but rather it is presented as another tool to evaluate, with advantages and disadvantages, which are documented in this study.

## Theoretical framework

A word problem is a verbal description of a situation in which a question is posed and the answer to which can be found by applying mathematical procedures to the numerical data provided in it (Verschaffel et al., 2020). The word problem-solving process is understood as the student's mental activity from the moment he encounters a problem that must be solved until the task is finished;
considering ideas from Polya (1957), Puig and Cerdán (1988) proposed six phases in the resolution process: reading, comprehension, translation, calculation, solution, and verification.

The reading and comprehension phases were defined separately to emphasize the attention that should be placed on reading the problem at the beginning of word problem-solving instruction. However, reading and comprehension are not independent of each other, since they are aspects of the same operation that has the purpose of understanding the word problem.

The translation phase consists of the identification of the variables involved in the word problem, both known and unknown, and the relationship between them. In this way, three important aspects must be considered: what data is going to be handled, what operations or procedures will be carried out, and in what order.

The calculation phase refers to the execution of operations and algorithms. In this phase, the student's translation skills no longer intervene, but rather their algorithmic skills. It is important to note that Puig and Cerdán (1988) consider that “translation and algorithmic skills are usually independent of each other” (p. 14).

Finally, the solution phase consists of interpreting the numerical result obtained in terms of what is asked in the word problem, and in the verification phase we proceed to verify that said solution is adequate. The verification can range from something informal, such as seeing if the result makes sense (i.e. that there are no negative distances), to something more formal, such as substituting a value in an equation or solving the problem using a different procedure and compare the results.

One of variables associated with the word problem-solving process is the content variable. This is related to the mathematical meaning of the problem. The content variable in this study is related to rational number as operator. These numbers are associated with diverse uses. Based on the phenomenological analysis of fractions proposed by Freudenthal (1983), and the interpretation of Valenzuela (2018), five uses of fractions are distinguished at an abstract level: as fracturer, comparer, measurer, operator, and number. As said before, this work focuses on the operator aspect, that is, when the fraction acts on a quantity by expanding or reducing it. The operator aspect of the fraction is related to the phenomena of reproducing, reducing, enlarging, shrinking, expanding, contracting, etc. This operator aspect of the fractions can be extended to the diverse ways to express a rational number, such as decimal notation, percentage, and ratios.

## Analytical readings' method

Taking as reference the analysis-synthesis method and the Cartesian method (Puig & Cerdán, 2014), the analytical readings' method to solve word problems is proposed, which is used indifferently for algebraic or arithmetic resolution processes. The method consists of the following five steps: reading the problem, forming a dictionary of quantities, building a tree graph, calculation, and solution-verification (see Figure 1).

In the first step, a first reading of the complete problem is done. Then it is read sentence by sentence, identifying the data presented and the verbs associated with them. Also, the main unknown quantity must be identified, that is, what needs to be calculated.
<figure><img src="image_64.png" /><figcaption>Figure 1: Example of the Analytical readings' method</figcaption></figure>

To form the dictionary of quantities, the known and unknown variables that appear in the word problem are identified. This is organized in a table with three columns. The first column indicates whether the information is a given numerical data or an unknown value that must be calculated. In the second column, the numerical value of the data or what the unknown variable will be called is indicated. In the third column, a brief description of the variable is given.

The third step is to build the tree-type graph (Figure 2) following the rules of analysis-synthesis so that at the end all the numerical data in the problem and the main unknown quantity are obtained.

<figure><img src="image_65.png" /><figcaption>Figure 2: Tree-type graph</figcaption></figure>

A tree-type graph has certain unique decomposition properties that allow obtaining a visual reproduction of the strategy followed in the resolution process (Roy & Roth, 2015). A tree is a data structure consisting of nodes connected by edges, with the unique property of not containing cycles.
For the analytical readings' method, we specifically use a rooted binary tree, in which there is a topmost node, commonly known as the root of the tree, and each node is linked to zero or two successor nodes. The nodes without successor nodes are called leaves.

In the fourth step of the method, the calculations that appear in the graph will be conducted, in the order that it shows. Finally, the fifth step is to write the result in terms of what is asked and verify it. In the Figure 1, an example of the steps followed in the analytical readings' method to solve a problem in shown. It should be noted that during the implementation of the analytical readings' method, the transit between its steps should not follow a strict linear order, it is always possible to go back and forth between the different stages.

Methodology

This qualitative research documents how mathematics students conceive the analytical readings' method as a tool to assess the word problem-solving process. In the study 13 students of the bachelor's degree in mathematics at the University of Guadalajara were considered. This is a group with affinity for teaching. The intervention consisted of three sessions that were part of an optional subject offered to the students. The worksheets, surveys, and essays were considered for the analysis. These were complemented with field notes, audio, and video recordings.

In the first session, the analytical readings' method was studied and explored through a document that simulated the solution of a word problem solved by a student -Paula-, reflecting in the process the dictionary of quantities, the tree-type graph, and its solution. The problem in question is the one shown in Figure 1. For this exploration, teams of 3 or 4 members were formed. Teams are instructed to discuss what the student meant by what she illustrated, what her procedure was, and whether her answer is correct. Afterwards, their reflections were presented to the class. Next, the phases in the word problem-solving process proposed by Puig and Cerdán (1988) were explained. The steps that make up the analytical readings' method were delineated, while inviting the students to share their observations, contributions, and doubts. In this session, two word problems were given to the students to solve and practice. For each problem, a student goes to the board to solve it using the method learned, with comments and suggestions from the other students and the teacher's guidance.

In the second session, the students began by solving a problem in teams through the analytical readings' method as a review, comparing and discussing the results as a group. Subsequently, the different aspects of rational numbers and diverse forms to express them were exposed and discussed. Four problems were answered in teams using the diverse forms to express the rational numbers studied using the analytical readings' method (word problems involving fractions, ratios, percentages, and decimal notation). The dictionary of quantities, tree-type graph and solution of each team were compared and discussed.

In the third session, the relationship between the analytical readings' method and the word problem-solving process was discussed in a debate moderated by the professor. And finally, an anonym survey was carried out to identify students' opinions on the use of the method.

As a final assignment, the students were asked to write an essay answering the following questions: 1) How is the solver's thought process reflected when solving a word problem in each of the steps of the analytical readings' method? 2) How is each phase of the problem-solving process related to the
steps of the analytical readings' method? 3) What is the usefulness of the analytical readings' method as an assessment tool for teachers?

## Results

The results are divided into three sections: the relationship between the steps of the analytical readings' method and the phases of the word problem-solving process identified by the students, their opinions on the advantages and disadvantages of the method as an assessment tool, and their views on how this method adapts for word problems that involve each form to express the rationale numbers seen during the intervention.

## Relationship between the steps of the analytical readings' method and the phases of the word problem-solving process

As described in the methodology, during the third session, the relationship between the analytical readings' method and the problem-solving process was discussed by the students. The relationships found by the students are summarized in Table 1.

<table><caption>Table 1: Relationship between the steps of the method and the phases of the problem-solving process</caption><thead><tr><td>Steps of the analytical readings' method</td><td>Phases in the word problem-solving process that are related</td></tr></thead><tbody><tr><td>Reading the problem</td><td>The reading and comprehension phases are related with this step of the analytical readings' method since the student is instructed to identify the question posed and the main unknown quantity, that is, what you should be calculated.</td></tr><tr><td>Forming a dictionary of quantities</td><td>The comprehension phase is also related with the step of forming a dictionary of quantities, because it helps to define the starting point (given numerical data) and know how to classify the information that you consider should be used to reach the solution. Also, a fundamental part of the translation phase is the identification of the variables involved in the word problem, which are reflected in the dictionary of quantities in an orderly manner.</td></tr><tr><td>Building a tree graph</td><td>Other aspects of the translation phase are what operations or procedures will be carried out and in what order? these are reflected in the tree graph.</td></tr><tr><td>Calculation</td><td>The step of calculation in the analytical readings' method, where the calculations that appear in the tree graph are conducted in the order that it shows, is equivalent to the phase of calculation in the problem-solving process, when operations and algorithms are executed.</td></tr><tr><td>Solution-verification</td><td>The solution-verification step of the method is analogous to the solution and verification phases of the word problem-solving process.</td></tr></tbody></table>

## Advantages and disadvantages of the method as an assessment tool

During the second session, when the teams presented and compared their dictionary of quantities, tree-type graph and solution of four different problems, each one using the diverse forms to express the rational numbers, a group dynamic of assessment was carried out. Each team tried to describe the
thought processes of their partners by reading the method. Subsequently, each student gave their opinion on the advantages and disadvantages of the analytical readings' method as an assessment tool in their essays and anonymously in the survey. These opinions are summarized in the Table 2.

<table><caption>Table 2: Advantages and disadvantages of the analytical readings' method as an assessment tool</caption><thead><tr><td>Advantages</td><td>Disadvantages</td></tr></thead><tbody><tr><td>This method works as a powerful assessment tool, through which it is not only possible to assess mathematical knowledge but also to assess reasoning, logic and even reading comprehension skills. This tool can provide extra information about the students' thinking process.<br/><br/>As teachers, the steps of the analytical readings' method help us identify the student's points of deficiency. The incorrect construction of the dictionary of quantities suggests problems with the reading and/or understanding of the text. The erroneous layout of the graph indicates problems with the translation of the text into mathematical processes and the incorrect calculation of the data reveals deficiencies in the student's arithmetic knowledge.<br/><br/>In addition, the method can be very useful for teachers when grading, as it encourages students to express their procedure in an orderly manner.</td><td>The analytical readings' method can be laborious due to the details to be specified in each step; this mainly has two disadvantages:<br/><br/>By virtue of its slow nature, the method is restricted to limited uses in a classroom environment where deadlines must be met.<br/><br/>Before it can be used as an assessment tool, students must be taught to work with this method. When the method is introduced for the first time, the solution process can be confusing at first, especially the tree-type graphs, since it is a new way of representing operations. Therefore, extra sessions would be needed for the group to become familiar with the method.</td></tr></tbody></table>

An example, when discussing the procedure made by a team to solve the next problem:

Problem 1. Melisa and Gerardo are going to paint the house where they both live. Melisa can paint the house in 3/4 of a day, while Gerardo is able to paint the house in 5/6 of a day. If they work together, what fraction of a day will it take them to paint the house?

The students highlighted the importance of seeing the variables in the dictionary of quantities. They argued that if the variable written in the tree graph (Figure 3) had been written in the dictionary, it would have allowed them to know how their classmates were interpreting the sum of the fractions of days that Gerardo and Melisa took to paint the house individually, why they carried out this operation, and therefore better understand what the error was in understanding the problem that led to the wrong answer. During the discussion, it was also mentioned that the tree graph allowed them to see how the fractions 3/4 and 5/6 were related, which led their classmates to an error.
<figure><img src="image_66.png" /><figcaption>Figure 3: Procedure made by a team while solving problem 1</figcaption></figure>

## The use of the analytical readings' method for solving word problems that involve diverse forms to express the rational numbers.

As indicated previously, during the intervention the students worked with four different word problems, which involved fractions, ratios, percentages, and decimal notation. In the survey, participants were asked to indicate how well they think the method is adapted to solve problems involving each of these four ways of expressing rational numbers, the options offered were: 1) *It does not fit well* - the explanation of my procedure feels forced when I use this method. 2) *It fits well* - I feel that the method helped me explain my procedure in a clear and structured way. 3) *Indifferent* - I don't feel like the method helped or hindered me in my procedure. Each answer had to be justified.

<figure><img src="image_67.png" /><figcaption>Figure 4: Students' opinion on the use of the method taking into account the content variable</figcaption></figure>

It can be seen in Figure 4 that the mathematical content variable affects the students' perception of how well the method is adapted to capture their thinking process and word problem-solving process. In particular, students think that the analytical readings' method helped them explain their procedures in a clearer way when solving word problems that involve fractions, but this is not the case for those word problems that involve ratios.

## Conclusions

The objective of this study was to test the analytical readings' method, regarding its use to assess the processes of word problems solving to document the reflections of a group of students. In this regard, there was evidence that the students considered the analytical readings' method as an assessment tool allows the student's mental activity and thought process to be clearly reflected in their procedure when solving word problems. As each step of the method is related to the phases in the word problem-solving process, it makes easier to follow the student's progress through each of the phases and, if there is an error, determine at what point in the resolution process it occurred, and identify its possible
causes. However, according to the students it must be considered that there may be some
disadvantages, such as the difficult, and time it takes to use this method when solving a problem.

The use of the analytical readings' method discussed in this paper is as an assessment tool, but during
the intervention a topic of discussion among the study participants was the potential of the method as
a teaching tool. The advantages of learning to solve certain types of word problems, for example
those involving fractions, were discussed. The possibility that this method offers was highlighted so
that students can explain their procedures, as well as the development of skills to establish
relationships between the quantities that appear in the problem. Although it was also mentioned that
using this method for the first time could be a challenge for students.

References

Freudenthal, H. (1983). *Didactical Phenomenology of Mathematical Structures*. Dordrech: D. Reidel.

Polya, G. (1957). How to solve it (2nd ed.). Princeton University Press.

Puig, L. & Cerdán, F. (1988). Problemas aritméticos escolares. Madrid: Síntesis.

Puig, L., & Cerdán, F. (2014). Acerca de carácter aritmético o algebraico de los problemas verbales.
En B. Gómez & L. Puig (Eds.), *Resolver problemas: Estudios en memoria de Fernando
Cerdán* (pp. 21-34). Valencia: Universitat de València.

Rodríguez, M. A. (2012). Resolución de Problemas. En M. D. Pochulu & M. A. Rodríguez (Eds.),
*Educación Matemática: aportes a la formación docente desde distintos enfoques teóricos* (1ra
ed., Vol. 1, pp. 153-174). Editorial Universitaria de Villa María, Universidad Nacional de
Villa María; Universidad Nacional de General Sarmiento.

Roy, S., & Roth, D. (2015). Solving General Arithmetic Word Problems. En L. Màrquez, C. Callison-Burch, & J. Su (Eds.), *Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing* (pp. 1743–1752). Association for Computational Linguistics. https://doi.org/10.18653/v1/D15-1202.

Siegler, R. S., Fazio, L. K., Bailey, D. H., & Zhou, X. (2013). Fractions: The new frontier for theories of numerical development. *Trends in Cognitive Sciences*, *17*(1), 13-19.
https://doi.org/10.1016/j.tics.2012.11.004

Valenzuela, C. (2018). *Modelo de enseñanza para fracciones basado en la recta numérica y el uso de applets: estudio en comunidades marginadas* [Tesis doctoral]. CINVESTAV.

Verschaffel, L., Schukajlow, S., Star, J., & Van Dooren, W. (2020). Word problems in mathematics education: a survey. *ZDM - Mathematics Education*, *52*, 1-16.
https://doi.org/10.1007/s11858-020-01130-4
# Exploring formative assessment and peer feedback in technology-enhanced mathematics learning environments using bar model virtual manipulatives

Poh Hwee Sim, Claire

Charles University, Faculty of Education, Prague, Czech Republic; clairepohhs@gmail.com

The uptick in the adoption of digital assessment, driven by increased technology integration in classrooms, not only transforms the assessment approach but also holds crucial implications for how teachers assign tasks and shape the way students engage in mathematical reasoning. This paper explores how educators leverage technology to enhance mathematics assessment and feedback. Using screencast (or screen recording) as a primary method of data collection, interactions of students solving word problems utilising the bar model, a web-based virtual manipulative, are recorded. Analysis of data collected may offer insights into students' specific competencies and deficiencies and inform teaching practices to meet their' learning needs. Digital assessment is broadened to include peer feedback and self-evaluation, facilitated by real-time interaction and idea-sharing through screen mirroring, another innovation supported by classroom connectivity.

Keywords: Digital assessment, peer feedback, bar model.

## Introduction

Technology has significantly impacted the assessment and feedback processes in teaching mathematics for both summative and formative purposes. The shift from offline to digital assessment introduces fresh possibilities for evaluating mathematics learning and modifying task structures as well as the scope of assessed abilities and skills (Drijvers et al., 2016). The concept of a connected classroom where teachers and students can exchange digital information has persisted for many years (Stacey & William, 2013). In recent years, technological advancements have rendered this vision more attainable. In the study by Clark-Wilson (2010), using classroom aggregation technology for mathematics was found to have promoted peer assessment as well as self-evaluation. Moreover, it was observed that teachers used feedback from students to inform the planning of future activities. These findings show that classroom connectivity offered fresh possibilities for formative assessment, providing teachers with insights into students' mathematical thinking. Leveraging classroom connectivity, multiple devices may be screen mirrored on the class display for side-by-side comparison and discussion among students. Studies suggest that students adjust their responses when comparing their work with peers, fostering increased opportunities for peer assessment and self-evaluation (Stacey & William, 2013). Drijvers et al. (2016) outline two crucial steps in formative assessment: collecting data on student achievements and devising strategies to enhance performance. Screen recordings serve as primary data. Teachers review these recordings and formulate appropriate measures to improve performance.

## Theoretical Background

### Framework for technology-mediated feedback

Mayer's (2002) Cognitive Theory of Multimedia Learning posits that learning is enhanced when information is presented through multiple modalities, such as visual and auditory channels. Figure 1
illustrates a cognitive theory of multimedia learning. Mayer asserts that multimedia messages that engage these cognitive processes are more likely to promote meaningful learning. His findings support a social agency extension of the cognitive theory of multimedia learning, suggesting that social cues within multimedia messages activate a conversational schema in learners, prompting deeper cognitive engagement. This holds significant implications for how the theory will shape peer feedback and assessment practices. This approach resonates with established conversational theories like Grice's (1975) conversational norms, which underscore the dedication to comprehending the other speaker's communication. Two approaches exist for assessing learning: retention tests and transfer tests (Mayer & Wittrock, 1996). Retention tests assess the capacity for memory recall. Transfer tests evaluate how effectively learned knowledge is applied to novel situations. He asserts that transfer tests offer the best assessment of learner understanding. Mayer's hypothesis proposes that better transfer is facilitated through interaction and conversation.

<figure><img src="image_68.png" /><figcaption>Figure 1: A cognitive theory of multimedia learning (Mayer, 2002, p. 103)</figcaption></figure>

Findings suggest that side-by-side comparison techniques, such as Visual Analysis for Image Comparison (VAICo), offer advantages in terms of speed, clarity, and accuracy in identifying differences in image data (Schmidt et al., 2013). Figure 2 shows various image sets from diverse domains, showcasing the method's adaptability across datasets. A crucial aspect of effective data analysis involves selecting suitable similarity metrics. We suggest teachers adopt this selection method when choosing screenshots of students' solution for side-by-side screen comparison to teach bar model strategies. Rittle-Johnson et al. (2017) underscore the significance of comparison in conceptual learning, with their classroom-based research supporting its efficacy in algebra instruction. Mayer (2002) suggests expanding the cognitive theory of multimedia learning to include social factors affecting learners' engagement in deep cognitive processing, such as combining visual (e.g., selective screenshots) and verbal (e.g., feedback interaction) models. Building on the body of research, we argue that when teachers present students' solutions using side-by-side screen for comparison, they foster real-time sharing and collaboration among students, contributing to multimedia learning through peer feedback.

## Framework for the model method

A distinctive pedagogy of Singapore mathematics, the model method is inspired by Greeno's part-whole and comparison schemas (Nesher, Greeno & Riley, 1982; Kintsch & Greeno, 1985). Students use rectangular bars to visualize mathematical relationships, facilitating comprehension of abstract quantities. For discussion in this study, consider the following illustrations of part-whole and comparison models.
<figure><img src="image_69.png" /><figcaption>Figure 2: Image datasets (Schmidt et al., 2013, p. 6) (b) shapes disappear, re-appear or change their colour (c) retina images from different patients (d) satellite images of a coastline affected by tsunami in Indonesia (e) images with colour coded gene expression information</figcaption></figure>

<figure><img src="image_70.png" /><figcaption>Table 1: Part-whole model for multiplication and division & multiplicative comparison models (Kho et al., 2014, p. 227)</figcaption></figure>

## Research Questions

The study seeks to examine digital assessment and feedback through an extension of mathematical experience using integrated technology. The research questions guiding the study are as follows:

RQ1. How can teachers leverage classroom connectivity to effectively analyse students' conceptual deficiencies in word problem solving utilising bar model virtual manipulatives?

RQ2. How can side-by-side screens be utilised for peer feedback in a technology-enhanced classroom?

## Methods

This study analysed students' digital experiences within technology-enhanced mathematics learning environments, focusing on formative assessment and feedback. For this research, a programme was piloted with an elementary school in the Czech Republic involving a cohort of nine Grade 8 (age 14) participants, during which they engaged in word-problem solving, utilising bar model virtual manipulatives apps provided within their tablet devices. The study leveraged classroom connectivity through digitally accessing and analysing students' mathematical thinking via screencast, i.e., recordings of their on-screen interactions while using bar model manipulatives. The students were given a series of tasks during which on-screen activities were recorded. The pilot programme, led by the researcher who also served as the teacher, consisted of two 3-hour sessions, one-week apart.

## Data collection and analysis

Screencast was utilised as the primary means of collecting data, capturing visual information of students' digital interactions. Following data collection, the teacher-researcher commenced the analysis process by describing and interpreting visual cues within the datasets in the form of analytic memos. The following sections outline the data collection process, which generated samples for analysis. This is followed by sorting and selecting samples for class discussion utilising side-by-side screen. Figure 3 is a flowchart illustrating this process.
<figure><img src="image_71.png" /><figcaption>FIGURE 3: INSTRUMENTATION, DATA COLLECTION AND ANALYSIS</figcaption></figure>

**Stage 1: Pen and paper task** – Baseline establishment

**Stage 2: Demo and Task 1** – Standardised demo and Task with virtual manipulatives

**Stage 3: Group activities (VM Task Group) and Task 2 (VM Task 2)** - Participants collaboratively solved word problems in groups of four or five. The aim was to assess the impact of group activities on students and determine if peer feedback is evident thereafter. Task 1, Group Activities and Task 2 were recorded for analysis, concluding Day 1 of the study.

**Stage 4: Analysis and sorting** - This involved systematically reviewing screen recordings of Task 1, Task 2 and Group Activities and writing analytic memos to capture emerging themes and pattern:

**Initial Observations** – The recordings were viewed multiple times for initial understanding.

**Identifying Patterns** – Patterns, themes and significant moments in the interactions were identified (see Coding). The identification process also included selecting among models with correct solution and incorrect solutions and sorted according to similarity metrics (Schmidt et al., 2013).

**Coding** – Codes were applied to segments of the screen output to represent moments of interactions. Interactions contributing to conceptual understanding: 1) use of the appropriate concept (Table 1), 2) accuracy in partitioning 3) alignments of parts 4) correct labelling, 5) application of operations

**Memo Writing** – Specific moments in the videos were annotated, documenting observations, interpretations and insights corresponding to coded segments.

**Interpretation I** – Coded segments and memos were analysed to identify broader themes related to research objectives. For RQ 1, data on interactions highlighting both strengths and weaknesses in the topic was collected and interpreted. Next steps were formulated to address weaknesses.

**Stage 5: Display and discuss**

**Interpretation II** – The selected screenshots served as instructional material and were displayed in class for side-by-side comparison and discussion. Observations were documented and redirected to Stage 4 for analysis (Figure 2). For RQ 2, reflective prompts (cognitive transfer and image comparison) guided reflections on students' engagement and comments. Cognitive transfer focused on identifying their ability to apply learned concepts in new situations. Image comparison related to their perception of accuracies/inconsistencies in screenshots and observed cross-referencing and editing of their own model construction.
# Results

## Interpretation I

This section addresses RQ1, examining interactions to identify strengths and weaknesses in the topic. We analyse Jakub's on-screen data to understand his grasp of the word problem in detail.

Question: 3/5 of the students in Grade 8 and 2/3 of the students in Grade 7 are girls. Both classes have the same number of girls. Grade 8 has 4 more boys than Grade 7. How many students are there in Grade 8?

For clarity, sequential screenshots (Transformations) of significant interactions are provided (Table 2b) with analytic notes for each moment. Table 2a outlines interpretations and reflections for these interactions. The documentation includes: 1) Appropriate Concept: Yes 2) Partitioning Accuracy: Yes 3) Alignment of Parts: No 4) Correct Labelling: No 5) Application of Operations: No.

The applied codes effectively pinpoint locations in the video, facilitating analysis. A thorough screencast examination reveals precise areas where Jakub struggled, with analytical notes explaining underlying reasons. Despite using the appropriate model, Jakub's solution was incorrect. The on-screen data not only clarifies specific points of struggle but also underscores his potential to engage with a concept he correctly selected but has not yet mastered in its application. Tailored remediation strategies can address Jakub's conceptual gaps, guiding future instructional strategies.

## Interpretation II

We use screenshots from Tereza's and Jakub's screen output (Figure 5 and Figure 6) to address the research objective that side-by-side screen effectively support peer feedback in a technology-enhanced classroom. The screenshots were selected using similarity metrics, aligning with the notion that selecting suitable similarity metrics is crucial for effective data analysis (Schmidt et al., 2013). This adds validity to their utilization in the study. Exchanges between teachers and students primarily reflected prompts and interactions related to the word problem. No individual attribution was recorded regarding which participant commented or posed each question during the study.

<table><caption>Table 2a: Reflections and interpretations of significant interactions</caption><tr><td>(a) The visual appears to be an effort to understand whether the number of girls aligns with the specified conditions outlined in the given word problem.</td></tr><tr><td>(b) The transformation shows that the appropriate bar model concept has been applied, i.e. the comparison model. This grasp of selecting the suitable application of the concept may potentially be attributed to the peer collaboration or peer learning experienced during the preceding group activity.</td></tr><tr><td>(c) The consistent use of the 'x' notation across all units led to confusion, resulted in him reaching an impasse.</td></tr><tr><td>(d) This visual representation highlights a discrepancy in logic, where the number of units drawn is unequal despite being depicted in the same size.</td></tr></table>

In addressing how side-by-side screens are utilised for peer feedback, we integrate Mayer's (2002) method of employing transfer tests to assess learner understanding. Questions and comments are famed and analysed from the perspective of cognitive transfer. Furthermore, we utilise similarity metrics to improve the effectiveness of data analysis (Schmidt et al., 2013) and to foreground inconsistencies between bar model constructions. When students identify these inconsistencies, peer feedback is engaged. Details of the analysis are provided in Table 3a and Table 3b.
<figure><img src="image_72.png" /><figcaption>Figure 4: Jakub's data and analytic memo</figcaption></figure>

This illustrates on-screen video data uploaded for analysis using qualitative research tool. The qualitative tool utilised enables thorough capturing of significant interactions and facilitates the creation of corresponding analytic memos

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /><figcaption>Figure 5: Tereza's screenshot</figcaption></figure>

<figure><img src="image_75.png" /><figcaption>Figure 6: Jakub's screenshot</figcaption></figure>

Note. From MathsBot.com (Hall, 2013)

<table><caption>Table 3a: Analysis of students' contributions</caption><thead><tr><td>Student's contributions:</td><td>Cognitive Transfer</td><td>Image Comparison</td></tr></thead><tbody><tr><td>That (pointing to Jakub's model) is not correct.</td><td>By identifying errors or inconsistencies in peer's work, the student demonstrates understanding of the subject matter and capacity to critically analyse problem-solving approaches.</td><td>Further discussion about why he/ she thought it was incorrect triggers a number of debates among the students. Some were not agreeable there were any errors, others asserted that there were errors which prompted the comments that follow.</td></tr><tr><td>5x is not equal to 3x</td><td>This observation indicated the student's ability to apply his/ her understanding of mathematical concepts, specifically the relationship between quantities represented by variables (5x and 3x)</td><td>This comment shifted some students' attention to Tereza's model, with one pointing out the shaded regions indicated equal values. Someone brought up this might mean Tereza's model was wrong, since 5x was not equal to 3x. Several others disagreed.</td></tr><tr><td>One bar should be longer and one bar should be shorter</td><td>This statement reflected the student's observation of a discrepancy in the lengths of bars depicted in a diagram, showcasing his/ her ability to apply mathematical concepts to evaluate visual information presented in the diagram.</td><td>There were some initial confusion which model or bar was being referred to. This led to a deliberation that concluded that one class has 4 more boys than the other. A number of students started turning to their devices to rework their constructions, others were seen cross-referencing each other's model</td></tr></tbody></table>
<table><caption>Table 3b: Analysis of teacher's contributions</caption><thead><tr><th>Teacher's contributions:</th><th></th><th></th></tr></thead><tbody><tr><td>There are the same number of girls in both classes (restating the important information).</td><td>Highlighting key words to restate the problem prompting students to re-address the problem in a structured way. It brought focus to the problem-solving transfer, promoting meaningful learning.</td><td>Several students started to point out the same number of girls were represented by two perfectly aligned shaded parts on Tereza's models.</td></tr><tr><td>How do we show that 3/5 of the students in Grade 8 and 2/3 of the students in Grade 7 have the same value? Show this on your bar model.</td><td>This is a critical juncture in the learning process. The students' attention was drawn to a seemingly different but equivalent values. This is an opportunity for meaningful learning as they grapple with complex ideas and develop strategies</td><td>Some students noticed that Tereza's model fitted the description. A number of them cross-referenced their own models with Teresa's model and were making changes.</td></tr><tr><td>Everyone, draw (the bars) on your screen. How do you make the rectangles equal?</td><td>This was an opportunity to be seized upon as the bar model approach lends itself to effectively convey abstract concepts through its visual representation.</td><td>There were significantly more discussions, some revising their constructions, shading, re-labelling to emphasise this aspect of the information discussed</td></tr></tbody></table>

They were then asked to write down the algebraic equation. Eventually, they arrived at the equation: $5x - 4 = 4.5x$

They then solved for $x$ by transposition and arrived at the answer: $x = 8$; Answer 40

## Discussion and initial findings

Drijvers et al. (2016) resonate with our study's findings, indicating a shift in mathematics pedagogical practices from teacher-led demonstrations to student-led modelling and discussions facilitated by technology integration. The analysis outcome satisfies the research goals of using technology to help teachers effectively analyse students' conceptual gaps. Screencasts enabled the teacher to monitor students' math activity unobtrusively, facilitating authentic feedback to offer tailored support more effectively. The gradual capture of Jakub's screen output showcased his potential capacity in applying a concept, an aspect that would not be evident if viewed solely as a finished product, e.g., on pen and paper or static display. Further, the analysis suggests that side-by-side screen shows promise in supporting peer feedback. The images featuring similarity metrics sparked discussions among the students, fostering cognitive transfer as they exchange perspectives while examining the images they are comparing.

## Conclusion

While the arguments presented strongly advocate for utilising digital means for assessing mathematics, in practice, implementing the features discussed remains challenging. In our research, we recognised a constraint regarding the efficiency of using screencasts for assessment, particularly in larger classroom settings. While our study involves a modest sample size of 9 students, the scalability of screencasts becomes challenging when applied to classrooms with more students. Managing, reviewing, and evaluating a large volume of screencasts poses logistical and practical hurdles for educators. This limitation highlights the need to explore alternative assessment strategies or technological solutions to ensure effective assessment practices in larger classroom environments. Another significant constraint is the lack of robust tools to cater for authentic mathematical practices such as sketching and scribbling within the app. Certainly, resorting to paper and pen can circumvent these constraints. However, this approach proves impractical when the objective is assessment through technology, as only a portion of the student's work would be visible within the assessment system. As a result, students may struggle to demonstrate their full problem-solving abilities, leading