# Proceedings of FAME 1 – Feedback & Assessment in Mathematics Education (ETC 14), 5-7 June 2024, Utrecht (The Netherlands).

Paola Iannone, Filip Moons, Christina Drüke-Noe, Eirini Geraniou, Francesca Morselli, Katrin Klingbeil, Michiel Veldhuis, Shai Olsher, Hankeln Corinna, Gonscherowski Peter, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Paola Iannone, Filip Moons, Christina Drüke-Noe, Eirini Geraniou, Francesca Morselli, et al. Proceedings of FAME 1 – Feedback & Assessment in Mathematics Education (ETC 14), 5-7 June 2024, Utrecht (The Netherlands). FAME - Feedback & Assessment in Mathematics Education, Utrecht University & ERME, Jun 2024, Utrecht, Netherlands. 10.5281/zenodo.14231455. hal-04807902

HAL Id: hal-04807902

https://hal.science/hal-04807902v1

Submitted on 6 Dec 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
# FAME 1
Proceedings

Feedback & Assessment in Mathematics Education
5-7 June, 2024 - Utrecht, The Netherlands

<figure><img src="image_3.png" /></figure>

**Edited by:**

Paola Iannone, Filip Moons, Christina Drüke-Noe, Eirini Geraniou, Francesca Morselli, Katrin Klingbeil, Michiel Veldhuis and Shai Olsher

e r m e
european society for research in mathematics education

14th ERME Topic Conference (ETC14)
**The International Programme Committee (IPC)**

Chair of the IPC:

Paola Iannone (United Kingdom)

Members:

Christina Drüke-Noe (Germany)

Eirini Geraniou (United Kingdom)

Filip Moons (Belgium)

Francesca Morselli (Italy)

Katrin Klingbeil (Germany)

Michiel Veldhuis (The Netherlands)

Shai Olsher (Israel)

**The Local Organising Committee (LOC)**

Filip Moons (Belgium)

Michiel Veldhuis (The Netherlands)

**Conference webpage:**

https://www.uu.nl/fame

Editors: Paola Iannone, Filip Moons, Christina Drüke-Noe, Eirini Geraniou, Francesca Morselli, Katrin Klingbeil, Michiel Veldhuis and Shai Olsher

Publisher: Utrecht University

Place: Utrecht, The Netherlands

Year: 2024

ISBN: 978-90-393-7788-8

DOI: 10.5281/zenodo.14231455

© Copyright left to the authors

**Recommended citation for the proceedings**

Iaonnone, P., Moons, F., Drüke-Noe, C., Geraniou, E., Morselli, F., Klingbeil, K., Veldhuis, M., Olsher, S. (Eds.). (2024). *Proceedings of FAME 1 – Feedback & Assessment in Mathematics Education (ETC 14), 5-7 June 2024, Utrecht (The Netherlands)*. Utrecht University and ERME. https://doi.org/10.5281/zenodo.14231455

**Recommended citation for single entries in the proceedings**

Authors (2024). Title of paper/poster. In P. Iaonnone, F. Moons, et al. (Eds.). *Proceedings of FAME 1 – Feedback & Assessment in Mathematics Education (ETC 14), 5-7 June 2024, Utrecht (The Netherlands)* (pp. xx-yy). Utrecht University and ERME. https://doi.org/10.5281/zenodo.14231455
# Introduction

## FAME1 – A first ERME topic conference on Feedback and Assessment in Mathematics Education

Assessment is pervasive in the teaching and learning of mathematics at any educational level though often its impact on the students' and teachers' experiences is under-estimated. During the work of the TWG21 - Assessment in Mathematics Education which will be led by Francesca Morselli at CERME14 in February 2025 – we realised that research on the assessment of mathematics is present in many TWGs at CERME. Therefore, we decided to organise the ERME topic conference FAME - Feedback and Assessment in Mathematics Education - to be a forum where all those interested in research related to the assessment of mathematics could meet. The conference took place in Utrecht between the 5th and 7th of June 2024. The call for papers highlighted three themes:

**Theme 1:** Formative feedback on mathematics tasks and its impact on students' experiences of learning mathematics: students' and teachers' perspectives. Within this theme we received papers related to the effect and impact of formative feedback on students' mathematics learning as well as on teachers' perspectives, at any educational level.

**Theme 2:** Resources for the (formative or summative) assessment of mathematics and design of tasks for assessing specific mathematics topics (as for example in the work of TWG22, TWGs 1 to 6 for specific mathematics topics). The issue of task-design for the assessment of mathematics cannot be overlooked and this theme comprised contributions that focused on task design for the assessment of specific topics (e.g., geometry or proof).

**Theme 3:** Teachers' and students' experiences with technology in/for the (summative and/or formative) assessment of mathematics (related to the work of TWG15/TWG16). Technology has become present in the teaching of mathematics but specifically in assessment and feedback. This theme collected contributions which addressed the use of technology for both summative and formative assessment.

At the conference 43 papers and 7 posters were presented. Topics were spread along the three themes, but we also had papers that spanned more than one theme. The papers and posters presented are included in these proceedings.

We had the pleasure to have two invited plenary talks, the first by Michal Ayalon (University of Haifa, IL) who spoke about the benefits of encouraging feedback literacy with trainee teachers and the second by Juuso Nieminen (Hong Kong University, HK) who spoke about student agency in assessment practices. We thank the plenary speakers for their interventions and their papers are also included in these proceedings.

The plenary panel was coordinated by Francesca Morselli (University of Genova, Italy) and included presentations on the theme of the impact of summative assessment at the secondary level in mathematics on students' perceptions and attitudes. We thank out panel speakers, Paul Drijvers (Utrecht University, NL), Alice Lemmo (Universita' dell'Aquila, IT) and Hans-Georg Weigand (University of Würzburg, DE) for their contributions and for the very stimulating debate that followed.
We were of course delighted to see many young researchers at the conference, and for them we organised an event on the morning of the 4th June, before the official start of the conference, for networking and for discussing the process of becoming a researches in mathematics education. Thank you to Katrin Klingbeil, Filip Moons, Eirini Geraniou and Christina Drüke-Noe for having organised the morning activities.

I want to close this brief introduction of the Proceedings of FAME by thanking the IPC and the local organizing committee – without them nothing would have been possible! They are

**International Programme Committee**

<table><tr><td>Christina Drüke-Noe</td><td>Pädagogische Hochschule Weingarten (DE)</td></tr><tr><td>Eirini Geraniou</td><td>University College London (UK)</td></tr><tr><td>Filip Moons</td><td>Utrecht University (NL)</td></tr><tr><td>Francesca Morselli</td><td>Università' di Genova (IT)</td></tr><tr><td>Katrin Klingbeil</td><td>University of Duisburg-Essen (DE)</td></tr><tr><td>Michiel Veldhuis</td><td>Utrecht University (NL)</td></tr><tr><td>Paola Iannone</td><td>University of Edinburgh (UK)</td></tr><tr><td>Shai Olsher</td><td>University of Haifa (IL)</td></tr></table>

**Local Organizing Committee**

<table><tr><td>Filip Moons</td><td>Utrecht University (NL)</td></tr><tr><td>Michiel Veldhuis</td><td>Utrecht University (NL)</td></tr></table>

Finally – we are already planning for FAME2 which will be held in Budapest in 2026 – I hope to see you all there!

Paola Iannone
President of the IPC of FAME1

The University of Edinburgh
United Kingdom

paola.iannone@ed.ac.uk
9. Enacting assessment accommodations in an inclusive formative classroom practice: The case of color-coding

Angeliki Dikarou and Chrissavgi Triantafillou .................................................................................. 90

10. Large language models as formative assessment and feedback tools? – A systematic report

Frederik Dilling .................................................................................................................................................. 98

11. From automatic diagnosis to lesson plannings: how teachers use elements of a digital formative assessment tool

Anica Eumann, Katrin Klingbeil and Bärbel Barzel ........................................................................ 106

12. Peer assessment in an undergraduate geometry course: Fostering proof competency in teacher students

Yael Fleischmann, Antoine Julien and Alexander Schmeding ................................................................. 110

13. Formative assessment: quick surprise quizzes online in class_in mathematics higher education

Sandra Gaspar Martins ........................................................................................................................................ 114

14. Assessment of students' learning mathematics with technology using video-based activities in an online course

Eirini Geraniou and Cosette Crisan ........................................................................................................ 122

15. Examining ChatGPT responses to TPCK assessment items

Peter Gonschwerowski¹, Edith Lindenbauer² and Benjamin Rott³ .................................................. 126

16. Open-ended items in digital formative assessments: Decision Trees as (AI-compatible) approach to reliably code students' understanding?

Corinna Hankeln.................................................................................................................................................. 134

17. Validity objections to comparative judgement

Ian Jones ........................................................................................................................................................................ 142

18. The students' use of visual representation for stimulating their metacognitive strategies: Teachers' perspectives

Amal Kadan-Tabaja........................................................................................................................................ 150

19. Using e-assessment for interactive example-generation tasks

George Kinnear and Igor' Kontorovich........................................................................................................ 158

20. Leveraging formative assessment to develop students' mathematical reasoning through images and student generated language

Richard Kitchen and Janet Lear ................................................................................................................................ 166
21. Unravelling (Mis)conceptions about Algebraic Letters: Exploring Response Patterns in the 'Meaning of Letters' SMART-test using Latent Class Analysis

Katrin Klingbeil and Filip Moons .................................................................................................................. 170

22. The potential of AI: generating answers for multiple-choice questions using ChatGPT

Laura Kuusemets .................................................................................................................................................. 178

23. Secondary mathematics teachers' experiences of using ChatGPT to design probability and statistics assessment items

Minsung Kwon and Inah Ko .......................................................................................................................... 186

24. Assessment by skills of an interdisciplinary project in which mathematics is the common threadwhich mathematics is the common thread

Maria Antonietta Lepellere and Marzia Toso.................................................................................................. 194

25. Advancing assessment in fractions: designing and implementing formative proficiency tasks

Hui-Chuan Li.................................................................................................................................................. 202

26. Elaborated feedback for interpreting graphs in Secondary Education ........................................ 210

María Sanz-Ruiz, Luis Miguel Soto-Sánchez, Zaira Ortiz-Laso and José Manuel Diego-Mantecón .................................................................................................................................................. 210

27. Development and validation of a three-dimensional framework for the classification of authentic tasks in mathematics

Marta Mikite, Ilze France and Ģirts Burgmanis.................................................................................................. 214

28. Fostering technology-enhanced formative assessment in Euclidean geometry proving through graded peer tutoring roles

Annamaria Miranda and Loredana Saliceto .................................................................................................. 218

29. CLASSIFICATION tasks as basis for assessment – Family resemblance as principle for choosing mathematical objects

Norbert Noster, Arnon Hershkovitz, Michal Tabach and Hans-Stefan Siller........................................ 226

30. Effects of a teacher professional development program in formative assessment on teachers' conceptions of feedback and assessment and their self-reported feedback practices

Torulf Palm, Gavin Brown and Björn Palmberg.................................................................................................. 234

31. Analytical readings' method as a tool to assess the word problem-solving process involving rational numbers

Sofia Peregrina Fuentes, Carlos Valenzuela García, María Teresa Sanz and Emilia López-Iñesta .................................................................................................................................................. 238

32. Exploring formative assessment and peer feedback in technology-enhanced mathematics learning environments using bar model virtual manipulatives

Claire Poh Hwee Sim .................................................................................................................................................. 246
33. Conception of two informative tutoring feedback strategies for mathematical tasks with STACK

Farhad Razeghpour..................................................................................................................254

34. Students' instrumentalizations of hints and automated feedback in their task solution process when learning mathematics with a digital curriculum resource

Sebastian Rezat....................................................................................................................262

35. Professionalising teachers with a digital formative assessment tool: a case study of the SMART tests

Fabian Rösken and Bärbel Barzel ....................................................................................270

36. Formative feedback in problem-solving lessons in German primary schools

Yasmin Theile and Benjamin Rott ....................................................................................274

37. Students' feedback process using automated post-submission report in their modeling process

Muna Touma and Shai Olsher................................................................................................278

38. Combining self-assessment and automatic-assessment – a mixed methods study ......286

Carina Tusche, Daniel Thurm and Shai Olsher........................................................................286

39. Facilitating teachers' in-the-moment feedback practices

Sumeyra Tutuncu....................................................................................................................290

40. Students' experiences with automated final answer diagnoses for mathematics tasks

Gerben van der Hoek, Bastiaan Heeren, Rogier Bos, Paul Drijvers and Johan Jeuring ......290

41. GeoGebra pop-up notifications as formative feedback for learning computational thinking in mathematics lessons

Wahid Yunianto, Theodosia Prodromou and Zsolt Lavicza ....................................................302

42. Mathematics teacher educators' microteaching implementations: selection of settings and approaches to assessment and feedback for prospective primary teachers

Bilge Yılmaz Aslan, Mehmet Fatih Özmantar and Bilge Kuşdemir Kayıran ........................310

43. Certainty-based marking as feedback in the context of formative assessment in mathematics lessons at school

Joerg Zender and Martina Geisen........................................................................................318

## Posters

1. Assessment in the mathematics classroom in relation to how students are in(ex)cluded in mathematics

Maria Silwer........................................................................................................................327

2. The importance of formative assessment in developing student teachers' teaching practice

Daniel de Oliveira Lima ........................................................................................................329
3. Perceptions of Effective Formative Feedback: A comparative Analysis Between Undergraduate Students and Mathematics Lecturers

Samah Taha, George Kinnear and Paola Iannone .................................................................................. 331

4. Design of tasks for assessing Diophantine equations on the new Mexican School

Alejandra Fabiola Huitrado Mora1 and Carolina de Haro Calderón1.................................................. 333

5. Exploring instrumental orchestration practices in the context of formative assessment with technology

Min Chen, Rogier Bos, Michiel Doorman and Paul Drijvers .................................................................. 335

6. Self-assessment in long-term problem solving STEM contexts

Julia Schäfer and Gero Stoffels .................................................................................................................. 337

7. The role of elicitation in formative assessment

Kristoffer Arvidsson1 and Torulf Palm........................................................................................................ 339
Plenary talks & Panel discussion
# Giving and receiving feedback on mathematics tasks: Insights gleaned from exploring teachers' and students' perspectives

Michal Ayalon

University of Haifa, Israel; mayalon@edu.haifa.ac.il

This paper explores the opportunities and challenges in cultivating feedback literacy among mathematics teachers and students through formative assessment practices. Drawing on findings from three distinct studies—Study A, which investigates pre-service teachers' engagement with feedback; Study B, which delves into students' engagement with feedback; and Study C, which examines the engagement of both teachers and students with feedback—this research provides a comprehensive analysis across varied educational contexts. The results demonstrate a significant enhancement in feedback literacy for both teachers and students, marked by a deeper understanding and more effective utilization of feedback. However, the findings also highlight the complexities and obstacles in implementing feedback practices, particularly within the context of mathematics education. The paper concludes by identifying key areas for further research, aiming to advance the effective integration of feedback literacy in educational practice.

## 1. Introduction

Feedback plays a crucial role in the learning process (Hattie & Timperley, 2007). It not only enhances understanding but also encourages metacognitive skills such as self-regulation and reflection (Shute, 2008). Despite its importance, there is significant variability in how feedback is provided, received, and utilized in the classroom. This paper aims to share findings from research focused on feedback literacy among mathematics teachers and students. It also discusses the feasibility of implementing effective feedback processes in the mathematics classroom.

## 2. Literature Review

### 2.1 Feedback

According to the Oxford Dictionary, feedback is defined as "advice, criticism, or information about how good or useful something or somebody's work is." When examining educational research on feedback, opinions vary on several key aspects, such as who should provide feedback (teachers or peers), how and when feedback should be delivered (e.g., written, spoken, planned, unplanned), and what the focus of feedback should be (e.g., whether it should address the correctness of work or focus on the processes and strategies underlying the task) (e.g., Carless & Winstone, 2020; Hattie & Timperley, 2007; Rakoczy et al., 2019; Sadler, 1989). Despite these differing viewpoints, there is some consensus on certain aspects of feedback that are effective for student learning, such as focusing on thinking processes and self-regulation, providing clear and specific feedback, and teaching students about assessment criteria (Fujita et al., 2018; Smit et al., 2023; van der Kleij, 2019; William & Thompson, 2008).

Traditionally, feedback has been viewed as a one-way transmission of information from teacher to student, aimed at correcting errors and guiding future performance (Hattie & Timperley, 2007). However, more recent perspectives emphasize the role of feedback as a dialogic process that involves both the giver and the receiver. This shift towards a more interactive and student-centered approach
to feedback recognizes that learners are active participants in interpreting and using feedback to enhance their learning (Boud & Molloy, 2013). According to Carless and Boud (2018), feedback should be understood as "a process through which learners make sense of information from various sources and use it to enhance their work or learning strategies." (Carless & Boud, 2018, p. 1315). This definition goes beyond notions that feedback is principally about teachers informing students about strengths, weaknesses and how to improve. It highlights the centrality of the student role in sense-making and using comments to improve subsequent work.

Research indicates that student engagement in feedback enhances understanding, encourages the development of metacognitive skills such as self-regulation and reflection, boosts students' confidence, reduces math anxiety, contributes to a positive and supportive classroom culture, and ultimately improves learning outcomes. (Adarkwah, 2021; Carless & Winstone, 2023; Hattie & Timperley, 2007; Mahfoodh, 2017; Shute, 2008; van der Kleij et al., 2019). These studies highlight the importance of implementing and researching feedback processes in mathematics classrooms.

## 2.2 Student and teacher feedback literacy

A central concept in the work presented here is feedback literacy. In terms of students' feedback literacy, Carless and Boud (2018) define it as “The understandings, capacities and dispositions needed to make sense of information and use it to enhance work or learning strategies.” (Carless & Boud (2018, p. 1316). They identify four key features of feedback literacy: first, students must appreciate their active role in the feedback process, recognizing that they are not merely passive recipients but active participants who need to engage with and act upon the feedback they receive, viewing it as a learning tool rather than just a judgment of performance. Second, feedback literacy involves the ability to make sound judgments about the quality of work, enabling students to assess both their own and others' work against established criteria, which supports self-regulation and the ability to independently enhance their performance. Third, it encompasses the ability to manage affect in constructive ways, as feedback can sometimes evoke strong emotional reactions, particularly when perceived as negative or critical; thus, feedback literacy helps students manage these emotions positively and see feedback as an opportunity for growth rather than a source of discouragement. Finally, feedback literacy is about taking action, where students must be able to convert feedback into actionable steps for improving their learning, which requires a clear understanding of the feedback and how to apply it in future tasks (Carless & Boud, 2018).

Teachers play a crucial role in designing and delivering feedback processes that support student learning in general, and students feedback literacy in particular (Carless & Winstone, 2023). Carless and Winstone (2023) define teachers' feedback literacy as “The knowledge, expertise and dispositions to design feedback processes in ways which enable student uptake of feedback and seed the development of student feedback literacy.” They identify three dimensions of teacher feedback literacy: the design dimension, which focuses on the teacher's ability to create feedback processes that encourage student engagement and foster the development of student feedback literacy by designing tasks that generate meaningful feedback and rubrics that clearly define success criteria (Brookhart, 2013); the relational dimension, which emphasizes the significance of the interpersonal relationship between teacher and student, requiring teachers to be supportive and sensitive in their delivery of feedback, thereby influencing students' motivation and self-esteem (Wiliam, 2011) and ensuring that students feel comfortable and confident in receiving and applying feedback; and the
pragmatic dimension, which addresses the practical challenges of delivering feedback within the constraints of the classroom and broader educational context, where teachers must balance the need for individualized feedback with the limitations posed by time constraints and large class sizes (Looney et al., 2018). Carless and Winstone (2023) also note that feedback literacy is closely tied to the particular discipline in which one works.

## 2.3 Feedback in practice

Research on feedback has shown that many students do not act on the feedback they receive, often lack effective strategies to utilize it, and struggle to recognize what constitutes a high-quality solution (Black and William 2009; Jönsson, 2013; van Gennip et al., 2010). Both students and teachers frequently view feedback primarily as information on strengths, weaknesses, and ways to improve student work (Little et al., 2024). This narrow perspective can limit the effectiveness of feedback as a tool for deeper learning and development. However, there is evidence that targeted interventions can successfully enhance features of feedback literacy (Ketonen et al., 2020; Little et al., 2024). Despite these promising findings, most studies in this area have been conducted at the university level (Little et al., 2024), with limited attention given to feedback practices in school mathematics. This gap is particularly concerning because mathematics often involves complex problem-solving and abstract reasoning, making effective feedback both more challenging to provide and more critical for student success (Schoenfeld, 2007; Teledahl, 2017). The research presented here addresses these gaps by focusing on the opportunities and challenges in developing feedback literacy among mathematics teachers and students, with a particular emphasis on the school mathematics context.

## 3. Three studies

This paper draws on three studies focusing on feedback literacy in different educational contexts: Study A, which examines pre-service teachers' (PSTs) engagement with feedback; Study B, which explores students' engagement with feedback; and Study C, which investigates teachers' and students' engagement with feedback. Findings from Study A have been published in Ayalon and Wilkie (2020). Studies B and C were conducted with a group of MA students and are currently in the process of being written. The common goal across these studies is to investigate whether and how experiencing formative assessment processes can support the development of feedback literacy among teachers and students. The research design for each study involved cycles of working on math tasks, generating and assessing feedback using rubrics, refining solutions and rubrics, and reflecting on the process.

The studies employ Looney et al.'s (2018) framework for Teacher Assessment Identity to examine various aspects of feedback literacy in both teachers' and students' reflections on their experiences with feedback. Looney et al. (2018) conceptualized teachers' assessment work as being deeply intertwined with their professional identity, beliefs about assessment, disposition towards enacting assessment, and their perceptions of their role as assessors. The framework they proposed encompasses five interconnected aspects—articulated in the first person to emphasize their self-reflective nature: *I know, My role, I believe, I am confident* (self-efficacy), and *I feel*. In our analysis of teachers' and students' reflections across three studies, we applied these categories through iterative processes of data sorting, continual comparison between the data and the evolving categories, and cross-category analysis. The findings are presented separately for each study below.
**3.1 Study A. Pre-service teachers' engagement with feedback**

**Goal.** Study A focused on investigating whether and how the experience of approximations of formative assessment practices could support the development of feedback literacy among pre-service teachers (PSTs).

**Population.** The study involved 97 pre-service teachers across two cohorts.

**Research design.** The research design incorporated a cycle that included small group work, the creation of rubrics, the assessment of student solutions, the refinement of these rubrics, and individual reflection (Figure 1).

<figure><img src="image_4.png" /><figcaption>Figure 1. Study A cycle</figcaption></figure>

**Data collection** included (1) PSTs' written responses, including (i) initial and revised assessment criteria and accompanying explanations, and (ii) reflections on their noticed differences between their initial and final assessment criteria and their learning about assessment practices, and (2) group interview that focused on the PSTs' experiences, their perceived strengths and difficulties, and their learning gains.

**Data analysis** included (1) iterative and comparative process for generating categories for the assessment criteria created by the PSTs, and (2) Drawing on Looney et al.'s (2018) categories for analyzing the PSTs' reflections: feelings, beliefs, self-efficacy, role enactment, and knowledge.

**Findings.** Overall, the PSTs reflected that the formative assessment (FA) processes significantly enhanced their understanding of feedback. They reported gaining valuable skills in building rubrics, identifying strengths and weaknesses, and proposing improvements. Additionally, there was evidence of a shift in their perception of their role in the feedback process, moving from a narrow view to recognizing the student as an integral part of the feedback cycle. Below are the main emerging themes, some of which are accompanied with examples.

Theme 1. *Awareness of the need for clear feedback and for students to understand the criteria.*

Theme 2. *Uncertainty in applying some assessment criteria, considering the student's perspective.*
For example, one of the PSTs wrote:
When I tried to assess the students' responses for their *quality of communication*, I felt confused [I feel]... there were some cases in which the solution 'spoke for itself' and I did not need further explanation to understand its underlying thinking. So, should I give a lower assessment for not including an explanation? [I am not confident]. For example, a student drew three parabolas to answer the functions task [Figure 2]. I like the solution; it reflects understanding, but I still wonder, should I also require a descriptive explanation from him? Is the solution perfect regarding communication [I am not confident]? I am thinking of the student, I prefer not to burden him with unnecessary work and then step back from the feedback process [I believe]." (PST3)

<figure><img src="image_5.png" /><figcaption>Figure 2. The functions task</figcaption></figure>

In this example, the PST considered the student's perspective, aiming to avoid unnecessary frustration.

Theme 3. *Awareness of the limitations of their own interpretations of students' solutions*. For example, one of the PSTs wrote:

What was difficult for me sometimes was being sure about my understanding of a student's way of thinking based on reading his answer [I am not confident]. The task is open-ended, and the solution is not obvious in its presentation or reading, which makes it hard to assess. Indeed, there was a case where my peer and I interpreted a student's solution differently. The student presented her solution but obtained the wrong answer. The explanation was very vague. I thought she had made a calculation error, but my colleague thought the student had misunderstood the task questions. So, to ensure the student receives a fair score and constructive comments, it's beneficial not to depend solely on my perspective [I believe]. Engaging the student for additional
explanations about their answer can enhance my understanding of their knowledge, moving closer
to an accurate assessment. [My role]" (PST26)

In this example, the PST considered the student's perspective by prioritizing fairness, providing
productive feedback, and involving the student in the process.

Theme 4. *Understanding the importance of providing supportive and motivational feedback.*

Theme 5. *Recognition of the importance of engaging students in exploring their solutions through feedback.* For example, one of the PSTs wrote:

My colleague and I used the Truck task [finding the maximum number of pallets that a truck can carry under a bridge] with 7th-grade students and chose these criteria to assess their solutions: the model used, manipulations, inferring, communication, and creativity. When I first evaluated their work, I assessed each student's quality level for each criterion and provided accompanying explanations. However, in discussions with my colleague, we reflected on how engaged the students were with the task and that we must keep them engaged with the feedback so they don't just glance at their assessments and dismiss them [My role]. So, instead of directly telling students what was good or bad, we activated them in the feedback process. For instance, some students provided valuable solutions but overlooked some of the constraints given in the situation. Here, my feedback was related to the "Inferring" criterion. I wrote: "Dan, the man in the task, is very worried. Try to convince him that your solution perfectly meets his requirements." This exemplifies how we engaged students in the feedback to motivate them to improve [I believe]." (PST 45)

In this example, the PST considered the student’s perspective, aiming to engage them in exploring
the solution with the feedback.

**Summary.** The summary of findings indicates a notable shift among PSTs toward recognizing the student as an integral part of the feedback process. This shift aligns with the dimensions of feedback literacy as outlined by Carless and Winstone (2023), particularly in the areas of design for uptake and the relational dimension. The change is evident across various aspects, including knowledge, feelings, beliefs, roles, and confidence, as described by Looney et al. (2018). However, some potential clashes emerged, such as uncertainty in applying criteria ("I know the criteria, but am unsure if I need to use them"), difficulty in understanding their role ("I see it as my role, but I don't know how to apply it"), and a lack of confidence despite their beliefs ("I believe, but I am not confident"). The main explanation for this shift, from the PSTs' perspective, was the opportunity to encounter and discuss different perspectives on student responses, which emphasized the need for a more nuanced, open, and responsive approach to providing feedback.

**3.2 Study B. Students' engagement with feedback**

**Goal.** Study B focused on investigating if and how a process of experiencing formative assessment practice might support students’ development of feedback literacy.

**Population.** The study involved 147 10th-grade students.
**Research design.** The research design incorporated a cycle that involved small groups solving rich mathematical tasks, building rubrics, assessing solutions, refining the rubrics, and individually reflecting on the process (Figure 3).

<figure><img src="image_6.png" /><figcaption>Figure 3. Study B cycle</figcaption></figure>

**Data collection** included (1) students’ written responses, which comprised (i) task attempts, (ii) initial and revised assessment criteria with accompanying explanations, and (iii) final reflections on the differences they observed between their initial and final assessment criteria and their learning about assessment practices, and (2) Individual interviews that focused on the students’ experiences, perceived strengths and difficulties, and learning gains.

**Data analysis** included (1) an iterative and comparative process for generating categories for the assessment criteria created by the PSTs and examining the distribution of these criteria, and (2) utilizing Looney et al.’s (2018) categories to analyze the students’ reflections, focusing on feelings, beliefs, self-efficacy, role enactment, and knowledge.

**Findings.** Data analysis revealed four main emerging themes, which are presented below, with some accompanied by examples.

Theme 1: *Increased, but still restricted, knowledge about using criteria for assessing solutions.* For example, one of the PSTs wrote:

> At the beginning I did not fully understand the assessment criteria. But while using them to assess one more solution and another different solution etc., I better understood what they mean. Some criteria were more difficult for me to use. For example, I wasn't sure whether to give preference to a certain solution strategy over another, such as algebraic, visual, verbal. I also wasn't sure how to determine whether a solution is original or not – if I did not think about it? [I know]**” (S18)

Theme 2: *A shift in perspectives related to goals and the importance of feedback.* Some PSTs emphasized that feedback should include the reasoning behind it to help students understand why and how to proceed. For example,
I tried to get into the heads of the students for whom I wrote the feedback and think about how best I could support them [My role]. I wanted the feedback to, like, connect with the student. I understand now that feedback shouldn't simply highlight errors and their corrections but also provide reasoning behind the mistakes and guide the student towards another attempt [I know]." (S18)

Additionally, some PSTs highlighted that feedback criteria can teach us about the nature of doing mathematics. For example,

Assessing the students' solutions through the rubric criteria made me realize that the feedback highlights various mathematical aspects, teaching us that there's much more to mathematics than just the final answer. For example, consider whether our mathematical strategy is good or whether it is worth trying an alternative solution, perhaps using a graph or algebra. Or if we used the data appropriately... I understand now that the feedback provides us with cues to consider in advance when approaching a task [I know]." (S43)

Theme 3: Experience anxiety initially but ultimately felt satisfaction from generating feedback. For example,

Writing the feedback is challenging. At first, I felt nervous [I feel] because I thought I couldn't do it [My confidence]. However, through various experiences and discussions with friends, I felt more confident [My confidence]... I still feel a bit anxious, especially because sometimes I am not sure that I understood the other student's thinking (I know). But it also feels worthwhile. It is like "getting out" of myself and looking at others' work, and then, when I "return" to look at my work, I can better assess it [I know]. It feels good [I feel]." (S52)

Theme 4: Awareness of their role as students in acting on feedback. For example,

As I wrote feedback for the students, I realized that simply reading the feedback isn't sufficient for the student to learn and improve. He should go over his solution and improve it with my feedback. I realized that when my teacher gives me feedback, it's the same. I really need to rethink my solution with the feedback. It's up to me if I want to learn from it [My role]". (S3)

**Summary.** The summary indicates a shift among the participants toward recognizing themselves as integral parts of the feedback process. This shift aligns with key aspects of feedback literacy, as outlined by Carless and Boud (2018), including appreciating feedback, making judgments, and managing affect. These changes were evident across various dimensions, such as knowledge, feelings, beliefs, roles, and confidence, as described by Looney et al. (2018), though potential clashes emerged. For instance, participants expressed sentiments like, "I see it as my role, but I feel anxious about it," and "I know the criteria, but I am unsure how to apply them." The main explanations for this shift include the experience of assessing imaginary students' solutions, which varied across different criteria and helped ease pressure, as well as the group's collective discussion and negotiation on the feedback.

## 3.3 Study C. Teachers' and students' engagement with feedback

**Goal.** Study B focused on investigating if and how a process of experiencing formative assessment practice might support teachers and students' development of feedback literacy.
**Population.** The study involved 20 secondary teachers and 200 students.

**Research design.** The teachers participated in a university course focusing on formative assessment, consisting 12 sessions. In the first session, the teachers discussed how feedback is used in their classrooms. Following this session, they administrated a feedback questionnaire to their students, focusing on Looney et al.'s (2018) categories to of feelings, beliefs, self-efficacy, role enactment, and knowledge. In the next session, the teachers shared their students' responses and concluded that, generally, the students did not appreciate feedback. As a result, they decided to plan an intervention aimed at engaging students more actively in the feedback process. The following two sessions were dedicated to learning about formative assessment and feedback, as well as planning the intervention. The final eight sessions focused on three cycles of experiencing feedback. Each cycle involved designing tasks and rubrics, implementing them in the classrooms, and sharing experiences to inform further decisions (Figure 4).

<figure><img src="image_7.png" /><figcaption>Figure 4. Study C design</figcaption></figure>

In this paper, I focus on the students’ perspective. The students participated in three cycles of experiencing feedback, which included solving a rich mathematics task, receiving feedback from their teacher, refining their solutions, providing "feedback on the teacher’s feedback, and class discussion (Figure 5).
<figure><img src="image_8.png" /><figcaption>Figure 5. Study C cycle (students)</figcaption></figure>

**Data collection** included pre-questionnaire and post-questionnaire (administrated after the three cycles) on feedback, focusing on Looney et al.'s (2018) categories to of feelings, beliefs, self-efficacy, role enactment, and knowledge.

**Data analysis** utilized Looney et al.'s (2018) categories to analyze the students' responses to the pre- and post- questionnaires on feedback.

**Findings.** Data analysis revealed four main emerging themes that reflect shifts in students' perspectives to feedback from the pre-questionnaire to the post-questionnaire. These themes are presented below, accompanied by examples from the responses of the same student, Lia, to illustrate a main profile identified among the students.

Theme 1: *A shift in perspectives related to goals and the importance of feedback.* For example, in the pre-questionnaire, Lia wrote:

> Feedback clarifies why points were deducted [I know]. The teacher provides feedback so we can correct our mistakes. [My role].

In the post-questionnaire, Lia wrote:

> The assessment criteria in the feedback are crucial for highlighting what's important in the task, and what can aid in finding a successful solution [I know]. I believe that the strategies I learn from the feedback will help me in solving other problems in the future [I believe] ... For example, the feedback about the strategy I used in my solution to the functions task taught me not to rush into the first strategy that comes to mind, but to pause and consider whether the strategy is truly appropriate for the task [I know].

In her response, Lia refers to the feedback she received from her teacher to her solution to the functions task [Figure 2]. Figure 6 presents Lia's solution in Hebrew. Lia used an algebraic approach to solve the solution and got stuck.
<figure><img src="image_9.png" /><figcaption>Figure 6. Lia's solution to the functions task</figcaption></figure>

Figure 7 presents the teacher's feedback, which was structured using a rubric with four criteria: using mathematical models, manipulations, inferring and looking back, and communication. For each criterion, the teacher provided an assessment of Beginning, Developing, or Proficient, accompanied by an explanation.

<table><caption>Figure 7. The teacher's feedback to Lia's solution</caption><thead><tr><th>Criteria</th><th>Beginning / Developing / Proficient</th></tr></thead><tbody><tr><td>Using mathematical models</td><td><strong>Developing</strong><br>You wrote that you got stuck. Think: did the strategy you used help you to solve the problem? If it didn't, can you think of why it might not have worked? Maybe there's a different model or approach that could be more effective?<br>Don't hesitate to discuss this with a friend too; sometimes a fresh perspective can make all the difference 😊</td></tr><tr><td>Manipulations</td><td><strong>Proficient</strong></td></tr><tr><td>Inferring & looking back</td><td><strong>Developing</strong><br>Your solution is well-structured, with each statement logically following from the previous one, which is excellent. As you reflect on your work, consider the questions I wrote above. They might help you delve deeper into understanding and improving your approach.</td></tr><tr><td>Communication</td><td><strong>Proficient</strong></td></tr></tbody></table>

As evidenced by Lia's responses to the pre- and post-questionnaires presented above, there is a shift in her perception of feedback from merely correcting mistakes to recognizing it as a valuable source of learning.

Theme 2: *Increased understanding of feedback*. In the post-questionnaire, Lia wrote:

> Over time, I've realized that my teacher gives me feedback by asking questions and making comments to help me figure out my solutions and see what I did right or wrong. But sometimes, it
is tough for me to grasp what the teacher is thinking. Like, for instance, when I got feedback on a geometry problem about making inferences, the teacher asked me a question to think about, but I just couldn't understand what she was getting at. It's like she had a clear idea, but I couldn't quite see things from her perspective. [I know]

In her response, Lia refers to the feedback she received from her teacher to her solution to a geometry task [Figure 8].

<figure><img src="image_10.png" /><figcaption>Figure 8: The geometry task, Lia's solution, and the teacher's (partial) feedback</figcaption></figure>

Lia's responses to the post-questionnaire presented above reflects an increased understanding of feedback overtime, alongside some struggles to discern the teacher's intended line of thought.

**Theme 3: Enhanced interest and motivation to engage with feedback.** For example, in the pre-questionnaire, Lia wrote:

> I usually feel frustrated because I don't understand what I am supposed to do with the feedback. Or when the comments are not clear. It makes me less inclined to look at the feedback [I feel].

In the post-questionnaire, she wrote:

> I really felt involved, like it was my own thing. I was trying to figure out what went wrong and what I could do differently. It was tough, but I actually enjoyed the challenge. Take the feedback I got, for example, which pushed me to try a different approach. I struggled with it for a while because it was really hard. Then, out of nowhere, I thought to drop equations and use a graph instead. After that, everything started to make more sense and flowed better. So maybe I felt frustration, but also satisfaction.” [I feel]

**Theme 4: Enhanced self-confidence to engage in feedback, including handling critique.** For example, in the pre-questionnaire, Lia wrote:
I'm unsure how to use the feedback and usually don't feel capable of applying the comments to improve [My confidence].

In the post-questionnaire, she wrote:

> I now believe more strongly that I can use feedback to improve. My confidence in myself has grown, and I'm gradually becoming more comfortable even when receiving critique [My confidence].

As evidenced by Lia's responses to the pre- and post-questionnaires, there is a shift to a sense of being capable of utilizing feedback effectively.

**Summary.** The summary indicates a shift among students toward recognizing themselves as integral parts of the feedback process. This shift aligns with key aspects of feedback literacy, as outlined by Carless and Boud (2018), including appreciating feedback, making judgments, managing affect, and taking action. These changes were evident across various dimensions such as knowledge, feelings, beliefs, roles, and confidence, as described by Looney et al. (2018), though potential clashes emerged. For example, students expressed sentiments like, "My role is..., but I feel unconfident," "I have motivation, but partial knowledge," and "I know the criteria, but I am unsure how to apply them." The main explanations for this shift from the students' perspective include explicit discussions between the teacher and students about feedback and the students' role in utilizing it, as well as empowering students to take an active role in using feedback, providing feedback on the teacher's feedback, and engaging in reflection.

## 3.4 Synthesis of findings

Across the studies, the findings indicate that engaging in feedback activities led to a significant shift toward feedback literacy among both teachers and students. For teachers, this shift was observed across various dimensions, including knowledge, feelings, beliefs, roles, and confidence, as described by Looney et al. (2018). This shift manifested in teachers providing more opportunities for students to develop their own feedback literacy. Similarly, students also experienced a shift toward feedback literacy through their engagement in feedback activities, which was expressed in their increased appreciation and utilization of feedback. This, in turn, motivated teachers to continue these efforts, suggesting potential for sustainable feedback practices.

However, some challenges were also identified. While both teachers and students showed greater appreciation for feedback and motivation to use it, there was a notable difficulty and lack of confidence in implementation. Specifically, there were hesitations in applying mathematics assessment criteria, both for teachers and students. These hesitations included questions such as whether there is a “preferred” mathematical model (e.g., algebraic, graphical, verbal), whether explanations and mathematical language should always be prioritized, and how to judge creativity. Teachers and students noted that applying criteria in open-ended math tasks felt more subjective compared to traditional tests.

Additionally, there was evidence of challenges in understanding someone else's thinking, both for teachers and students. This challenge was particularly pronounced in the context of open-ended math tasks and complex solutions, where understanding thinking that differs from one's own proved
difficult. These findings highlight the complexities of implementing effective feedback practices, particularly in the context of mathematics education.

## 4. Conclusion

Our research has provided some valuable evidence regarding the potential for developing feedback literacy among both students and teachers. By applying Looney et al.'s categories, we were able to identify nuanced complexities within the dimensions of feedback literacy, highlighting how these dimensions interact and manifest differently depending on various factors. Importantly, the challenges specific to mathematics emphasize that feedback literacy is deeply connected to the particular discipline in which it is applied, as noted by Carless and Winstone (2023). This reinforces the need for tailored approaches to developing feedback literacy that consider the unique demands and characteristics of each subject area.

To build on these findings, further research is necessary to explore the interconnections between the different categories related to feedback literacy. Understanding how these categories influence and interact with one another could provide deeper insights into how feedback literacy develops and operates in educational contexts. Additionally, there is a need to investigate the implementation of systematic changes in mathematics classroom-feedback practices. Such research could explore how to effectively integrate feedback literacy into everyday teaching practices in a way that is sustainable and impactful. Finally, it is essential to explore feedback literacy across different contexts and on a larger scale. Large-scale applications and studies across various educational settings could provide a more comprehensive understanding of how feedback literacy can be developed and supported in diverse environments, ultimately contributing to more effective teaching and learning practices.

## References

Adarkwah, M. A. (2021). The power of assessment feedback in teaching and learning: a narrative review and synthesis of the literature. *SN Social Sciences, 1*(3), 75.

Ayalon, M., & Wilkie, K. (2020). Developing assessment literacy through approximations of practice: Exploring secondary mathematics pre-service teachers developing criteria for a rich quadratics task. *Teaching and Teacher Education, 89*.

Black, P., & William, D. (2009). Developing the theory of formative assessment. *Educational Assessment, Evaluation and Accountability, 21*(1), 5e31.

Boud, D., and E. Molloy. 2013. “Rethinking Models of Feedback for Learning: The Challenge of Design.” *Assessment & Evaluation in Higher Education, 38*(6): 698–712.

Brookhart, S. M. (2013). How to create and use rubrics for formative assessment and grading. Ascd.

Carless, D., & Boud, D. (2018). The development of student feedback literacy: enabling uptake of feedback. *Assessment & Evaluation in Higher Education, 43*(8), 1315–1325.

Carless, D., & Winstone, N. (2020). Teacher feedback literacy and its interplay with student feedback literacy. *Teaching in Higher Education, 1-14*.
Fujita, T., Jones, K., & Miyazaki, M. (2018). Learners’ use of domain-specific computer-based feedback to overcome logical circularity in deductive proving in geometry. *ZDM – Mathematics Education, 50*(4), 699-713.

Hattie, J., & Timperley, H. (2007). The power of feedback. *Review of educational research, 77*(1), 81-112.

Jonsson, A. (2013). Facilitating productive use of feedback in higher education. *Active Learning in Higher Education, 14*(1), 63-76.

Ketonen, L., Nieminen, P., & Hähkiöniemi, M. (2020). The development of secondary students' feedback literacy: Peer assessment as an intervention. *The Journal of Educational Research, 113*(6), 407-417.

Little, T., Dawson, P., Boud, D., & Tai, J. (2024). Can students' feedback literacy be improved? A scoping review of interventions. *Assessment & Evaluation in Higher Education, 49*(1), 39-52.

Looney, A., Cumming, J., van Der Kleij, F., & Harris, K. (2018). Reconceptualising the role of teachers as assessors: teacher assessment identity. *Assessment in Education: Principles, Policy & Practice, 25*(5), 442-467.

Mahfoodh, O. H. A. (2017). “I feel disappointed”: EFL university students’ emotional responses towards teacher written feedback. *Assessing Writing, 31*, 53-72.

Rakoczy, K., Pinger, P., Hochweber, J., Klieme, E., Schütze, B., & Besser, M. (2019). Formative assessment in mathematics: Mediated by feedback's perceived usefulness and students' self-efficacy. *Learning and Instruction, 60*, 154-165.

Sadler, D. R. (1998). Formative assessment: Revisiting the territory. *Assessment in education: principles, policy & practice, 5*(1), 77-84.

Smit, R., Dober, H., Hess, K., Bachmann, P., & Birri, T. (2023). Supporting primary students' mathematical reasoning practice: the effects of formative feedback and the mediating role of self-efficacy. *Research in Mathematics Education, 25*(3), 277-300.

Teledahl, A. (2017). Mathematics teachers' assessment of accounts of problem solving. In Dooley, T. & Gueudet, G. (Eds.) (2017). *Proceedings of the Tenth Congress of the European Society for Research in Mathematics Education* (CERME 10, February 1–5, 2017). DCU Institute of Education and ERME.

Schoenfeld, A. H. (2007). What is mathematical proficiency and how can it be assessed? In A. H. Schoenfeld (Ed.), *Assessing Mathematical Proficiency* (pp.59-73). New York: Cambridge University Press.

Shute, V. J. (2008). Focus on formative feedback. *Review of educational research, 78*(1), 153-189.

Van der Kleij, F. M. (2019). Comparison of teacher and student perceptions of formative assessment feedback practices and association with individual student characteristics. *Teaching and Teacher Education, 85*, 175-189.
van Gennip, N. A. E., Segers, M. S. R., & Tillema, H. H. (2010). Peer assessment as a collaborative learning activity: the role of interpersonal variables and conceptions. *Learning & Instruction, 20*(4), 280e290.

William, D. (2011). What is assessment for learning? *Studies in educational evaluation, 37*(1), 3-14.

William, D., & Thompson, M. (2007). Integrating assessment with instruction: What will it take to make it work? In C. A. Dwyer (Ed.), *The future of assessment: Shaping teaching and learning* (pp. 53e82). Mahwah, NJ: Lawrence Erlbaum Associates.
# Summative Assessment across three countries

Francesca Morselli¹, Paul Drijvers², Alice Lemmo³, Hans-Georg Weigand⁴

¹DIMA University of Genova, Italy; morselli@dima.unige.it

²Utrecht University, the Netherlands; p.drijvers@uu.nl

³Università dell'Aquila, Italy; alice.lemmo@univaq.it

⁴University of Würzburg, Germany; weigand@mathematik.uni-wuerzburg.de

## Introduction

This article builds upon the themes discussed during the panel held at the conference, which focused on summative assessment. During the panel, three researchers—Drijvers, Lemmo and Weigand— presented information on three countries (Italy, the Netherlands, and Germany, respectively) and offered insights, reflections, and proposals for future directions. In preparation for the panel, a set of questions was shared with conference participants weeks in advance, allowing them to upload their responses to a Padlet platform.

This first section provides a brief summary of the key themes that emerged from the padlet. Researchers from six countries posted their contributions: China, Greece, Mexico, Singapore, Spain, and Turkey. Their comments helped to sketch a first overview of summative assessment practices, their purposes, and emerging trends in each context.

The subsequent sections present the contributions of the three experts.

### First theme: how summative assessment is carried out

In most countries, summative assessment predominantly relies on written tests conducted with traditional methods, such as paper and pencil. These assessments are usually administered at the end of the semester or academic year, with final scores calculated as the average of results from multiple tests or assignments throughout the year. We may note that this approach may limit the exploration of alternative assessment methods.

### Second theme: the purpose of summative assessment

The primary objectives of summative assessment across the surveyed countries include: ability streaming and entrance to schools and universities.

Summative assessment for ability streaming is used in contexts like Singapore: this practice aims to group students based on their academic abilities. However, research highlights potential drawbacks, such as stigmatization and a narrowing of curricula for students in lower ability streams (Kramer-Dahl & Kwek, 2010).

Summative assessment is used for entrance to schools and universities in countries such as Turkey, where summative evaluation scores play a critical role in determining student progression to higher education levels. Secondary school scores are used to assign students to high schools, while university placements depend on standardized entrance exams.

### Third theme: emerging trends and key issues
The survey highlighted the following emerging trends and key issues related to summative assessment.

The first trend concerns equity and access. Several countries face disparities in assessment outcomes due to socioeconomic and geographical inequalities. For instance, students from rural or underprivileged urban areas in China often encounter additional challenges in accessing quality education. In Greece, efforts to create an inclusive public education system have led to a focus on evaluating students with learning difficulties under the general education framework, emphasizing the social dimensions of assessment.

The second trend is related to assessment stress and mental health. High-stakes assessments contribute to student stress and excessive test preparation. In response, countries like China are adopting new curriculum standards that emphasize holistic assessment and performance-based evaluations. Similarly, Singapore is exploring strategies to balance traditional examinations with broader educational goals, gradually reducing their weight while maintaining accountability.

The third trend concerns the use of Artificial Intelligence (AI). Spain has highlighted the growing influence of AI on educational assessments, particularly regarding the quality and integrity of student-submitted tasks. This technological shift raises questions about maintaining fairness and adapting assessment frameworks to the digital age.

The last trend refers to teacher professional development. Effective implementation of summative assessment reforms requires well-trained educators. However, countries such as Mexico report that teachers often lack the necessary training to adopt new methods, leading to the continued use of traditional approaches. Greece has also emphasized the need for professional development programs to support innovative assessment strategies.

# Summative Assessment in the Netherlands

Paul Drijvers

## The present state

Summative assessment at the national level in secondary education in the Netherlands has the form of final national examinations at the age of 16 years (vocational education), 17 years (pre-higher education), or 18 years old (pre-university education). For all school types, students' final grades are the average of their grade on the final national examination and the grade for school-based examinations, the latter having different possible formats depending on the schools' policy.

The national examinations for some tracks in vocational education are fully digital (assessment *through* technology, Olsher et al., 2023), but are graded manually by the teachers. For other school types, the final national examination is a paper-and-pen test, in which students can use a calculator or, in case of pre-higher and pre-university education, a non-CAS graphing calculator (assessment *with* technology).

## Some reflections

The aim of the 2016 curriculum reform for pre-higher and pre-university education is to give higher-order mathematical skills, called mathematical thinking activities, a more prominent place in both teaching and assessment has proven to be hard to implement in the final examinations (Drijvers et
al., 2019). How to assess such higher-order skills in the traditional assessment format, in combination with their positions in formative assessment, remains a challenge; a challenge that is not limited to the Netherlands but is widely recognized in our community (Olsher et al., 2023).

## Further Developments

At present, a curriculum reform process for all educational levels is taking place in the Netherlands. Of course, the role of digital technology in summative assessment is part of the debate. The common opinion seems to be that graphing calculators are no longer up-to-date technology. One option is to switch to examinations through technology, or to examinations with digital tooling on a tablet or laptop in a so-called bootable client lockdown environment. Content wise, the question is whether or not to include CAS facilities, and if yes, to what school types. Once these questions will be answered and the new curricula will be implemented, the next question of how to deal with artificial intelligence as mathematical tools will be begging to be addressed as well.

# Summative Assessment in Italy

Alice Lemmo

## The present state

The aspects that principally characterize the Italian assessment policy concern teacher responsibility. The Italian national guidelines state that the responsibility for both periodic and final assessment lies with teachers (MIUR, 2012). In addition, the certification of competences at the end of middle (grade 9) and secondary school (grade 13) is also provided by the teachers.

The Italian standardised assessment (INVALSI) involves all grade 2, 5, 8, 10 and 13 Italian students. INVALSI test is in no way part of school assessment, but it is compulsory for participating in the final examination. It has no formal impact on the teaching-learning and assessing process, each teacher decides independently how to analyse, interpret and use the data collected from his/her students in teaching practice. INVALSI test from grade 8 to grade 13 is computer based, in the next two years this will also be the case for grade 2 and 5 (primary school). Italian standardised assessment therefore involves the use of technology (calculators, digital tools) and is administered through technology (computers/tablet), paper and pen are available if the student needs them.

At the end of middle and high school, all students take a final examination that consists in both written and oral assessment for all disciplines, including mathematics. For grade 8 students both written and oral test are designed and administered by teachers. The final mark consists in the performance in the exams, assessed by teachers. Differently, the final mark at the end of grade 13 is composed by both final exam performance and the assessments of the last three years of school career. In particular, 40% concerns the school career, the other 60% is divided into 1/3 for oral discussion and 2/3 for written examination, in turn divided into two tests. The two written exams are statewide, but teachers administer and evaluate them. Only the second test concerns mathematics, but it is administered only for the scientific high schools' students.

## Some reflections

The main aspect of the Italian policy concerns the responsibility and autonomy of teachers in the assessment process at each school level. This could be interpreted because teachers' support guides
and assists students in their school career in the same manner as the assessment process should do in a formative perspective. Unfortunately, this may reveal as a point of deep weakness: such autonomy leads to several obstacles and critical issues, if it is not supported and trained.

The use of tests and written exams at school is becoming more and more common (Shepard, 2000); in Italy, written exams and tests seem to represent the main tool for class assessment in mathematics (Amado & Morselli, 2023). This tendency could be perceived as a remedy for inconsistencies in the evaluation or grading practices proposed to teachers and/or as a quest for evaluative objectivity (Shepard, 2000). My personal experience as teacher trainer revealed that Italian teachers seem confused formative and summative assessment with subjective and objective practices. This misconception endorses the assessment of procedural at the expense of conceptual understanding (Hiebert & Lefevre, 1986). This has a strong impact on teaching and learning whereby teaching to the test and consequently learning to the test are increasing. This is confirmed by grade 13 INVALSI test results. Students' data is presented divided in five levels: Level 1 represents students that only deal with elementary knowledge and procedural items, while Level 5 denotes students who tackle problem solving and argumentation items. About 50% of grade 13 students is in Level 1 and 2, only 15% of them belong to Level 5 (INVALSI, 2024). This is also confirmed by OECD PISA 2022 results: more than 50% of Italian grade 10 students belong to level 3 and below (OECD, 2023).

## Further Developments

Several Italian researchers are engaged in research on formative assessment (e.g. Cusi, Morselli & Sabena, 2017); others on standardised assessment (e.g. Ferretti, Giberti & Lemmo, 2018). Little is yet done on summative assessment, in particular with tests or written exams (Lemmo, 2023). Concerning this last issue, Italian research is exploiting teaching-learning to the test, proposing tests designed upon processes and assessment grids that aim at competencies (Niss & Højgaard, 2019).

This perspective does not impose to drastically change tasks' design but to modify the way of assessing them. In a procedural approach, teachers could choose that the correct answer in a task counts a certain number of points, and he/she could subtract some points for each error or missing information. The feedback students and teachers receive focus on errors and/or missing answers. In a competency approach, each single task of the test contributes for identifying the level for each competency. For example, even with a wrong answer in some tasks, students could demonstrate ability in using algebraic calculations or properties; in other cases, they could show high communication competency and so on. In this latter case, the focus of the assessment is not on error or omission but on the level of the competencies the student employs.

In conclusion, there is a great gap between the development of research in mathematics education and assessment (Niss, 1992). Research needs to overcome this gap remembering that tests and written exams has an impact on the teaching-learning process and regulates it. In the Thirteenth Congress of the European Society for Research in Mathematics Education, into the Assessment in mathematics education working group, “there was consensus on the fact that assessment is a crucial part of the teaching and learning process and not a mere final “appendix”” (Morselli et al., 2023, p. 5).

# Summative Assessment in Germany

Hans-Georg Weigand
## The present state

Concerning the education system, there is no uniform system in Germany. There are 16 states, and each state is responsible for the education system on its own. The consequence is that there is a great variety of school systems and also of final examinations. Although there exist National Standards since 2004, and there is a national “task pool” for exams without technology (Fig. 1), the states can take different tasks from this pool for the first part of the final examination and the second part of the examination is completely different in different states. Concerning digital technologies there are states, which do not allow any digital technology in examinations (except arithmetic calculators), in some other states schools can choose whether they want to do the exam with advanced (usually scientific CAS-calculators) or without technology, and states where CAS-calculators are mandatory for all students.

The federal government tries to control the compliance of the national standards. There are different nationwide annual tests. VERA<fnref n="1" /> is a comparative test in grade 3 and 9 (Fig 2 and 3). “IQB-Trends in student achievement”<fnref n="2" /> is a test every 5 years in grade 4 and every 3 years in grade 9. It has been a comparative test between the states from 2008-2012, however since 2015 it should “only” give feedback to students, teachers and the policy makers.

<figure><img src="image_11.png" /><figcaption>Fig. 2 VERA - grade 3: How many small boxes does the rectangle have?</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 3 VERA 8 – grade 9</figcaption></figure>

<footnote n="1">¹ https://www.iqb.hu-berlin.de/vera</footnote>

<footnote n="2">² https://www.iqb.hu-berlin.de/bt</footnote>
Give reasons why there is no triangle with the sizes

<figure><img src="image_13.png" /><figcaption>Fig. 4 Bavarian statewide test - grade 8</figcaption></figure>

Simplify as much as possible!

$$a) 3x - 7y + 2y =$$

$$b) 0,1b^3 \cdot 20b^2 =$$

$$c) \left(\frac{1}{2}x^2\right)^3 =$$

## German's reaction to the (bad) results of TIMMS, PISA, VERA, ...

There are – again – quite different reactions to the (bad) results of the international and national tests. E.g., Hamburg or Berlin assigned a scientific working group (2018, 2020). There have been many suggestions for changes concerning the goals, contents and classroom work and the states working on the implementation of some of these suggestions. Bavaria introduced statewide annual – quite classical – tests for grade 8 and 10 at the beginning of each school year (Fig. 4). Last year the nationwide professional development programme “QuaMath” for the next 10 years and for teacher of 10.000 schools was established and at the moment there is just starting a nationwide programme for low achieving students. Sometimes, reactions from the policy makers are done without any empirical basis. In Baden-Württemberg, since 2001 graphing calculators were allowed in examinations, however in 2014, there was a ministry ban and since then no advanced technology is allowed in final examinations. This is an example how arbitrary decisions concerning final examinations are taken.

## Further Developments

There are some very important questions and emerging areas for further development concerning summative assessment that are discussed intensively (of course, not only) in Germany. Summative assessment ...

* ... has a crucial influence on classroom teaching (e.g., how digital media are used).
* ... cannot be a 1:1 copy of classroom work. There are some goals which are not – and cannot be – addressed in summative assessment, e.g., modeling or inquiry-based learning.

* ... must keep the balance between procedural and conceptual understanding (CU).

* ... but must be seen in relation to the development of adequate tasks for CU in the classroom.

* ... requires follow-up research on the reasons for good/poor performance.

* ... needs strategies to react – in the short and long term – to assessment results.

* ... needs to think about alternatives, e.g., oral exams, project work or portfolios.

* ... has to be open for up-coming developments like virtual reality, augmented reality or artificial intelligence.

## References

Amado, N., & Morselli, F. (2023) Teachers' beliefs about assessment: a study in Italy and Portugal. In In Drijvers, P., Csapodi, C., Palmér, H., Gosztonyi, K., & Kónya, E. (Eds.). (2023). *Proceedings of the Thirteenth Congress of the European Society for Research in Mathematics Education (CERME13)*. Alfréd Rényi Institute of Mathematics and ERME. https://hal.science/hal-04413032v1

Cusi, A., Morselli, F. & Sabena, C. Promoting formative assessment in a connected classroom environment: design and implementation of digital resources. *ZDM Mathematics Education* 49, 755-767 (2017). https://doi.org/10.1007/s11858-017-0878-0

Drijvers, P., Kodde-Buitenhuis, H., & Doorman, M. (2019). Assessing mathematical thinking as part of curriculum reform in the Netherlands. *Educational Studies in Mathematics*, *102*(3), 435–456. https://doi.org/10.1007/s10649-019-09905-7

Ferretti, F., Giberti, C., & Lemmo, A. (2018). The Didactic Contract to interpret some statistical evidence in mathematics standardized assessment tests. *EURASIA Journal of Mathematics, Science and Technology Education, 14*(7), 2895-2906. https://doi.org/10.29333/ejmste/90988

Hiebert, J., & Lefevre, P. (1986). Conceptual and procedural knowledge in mathematics: An introductory analysis. In J. Hiebert (Ed.), *Conceptual and procedural knowledge: The case of mathematics* (pp. 1–28). Routledge https://doi.org/10.4324/9780203063538

Kramer-Dahl, A., & Kwek, D. (2010). ‘Reading’ the home and reading in school: Framing deficit constructions as learning difficulties in Singapore English classrooms. In C. Wyatt-Smith, J. Elkins, & S. Gunn (Eds.), *Multiple perspectives on difficulties in learning literacy and numeracy* (pp. 159–178). Dordrecht: Springer.

INVALSI (2024). Rapporto Invalsi 2024. Retrieved from https://invalsi-areaprove.cineca.it/docs/2024/Rilevazioni_Nazionali/Rapporto/Rapporto%20Prove%20INVALSI%202024.pdf

Lemmo, A. (2023). Mathematical written tests as formative assessment practice. In In Drijvers, P., Csapodi, C., Palmér, H., Gosztonyi, K., & Kónya, E. (Eds.). (2023). *Proceedings of the Thirteenth*
*Congress of the European Society for Research in Mathematics Education (CERME13).* Alfréd Rényi Institute of Mathematics and ERME. https://hal.science/hal-04413550

MIUR (2012). *National guidelines for the curriculum for pre-school and first cycle of education*. [National guidelines for the curriculum for grades K-8]. Annali della Pubblica Istruzione, Numero Speciale. Le Monnier.

Morselli, F., Drüke-Noe, C., Giberti, C., Kaplan-Can, G. & Rämö, J. (2023). An introduction to TWG21: Assessment in mathematics education. *Proceedings of the Thirteenth Congress of the European Society for Research in Mathematics Education (CERME13)*. Alfréd Rényi Institute of Mathematics and ERME. https://hal.science/hal-04412990v1

Niss, M. (Ed.). (1992). *Investigations into assessment in mathematics education: An ICMI study* (Vol. 2). Springer Dordrecht. https://doi.org/10.1007/978-94-017-1974-2

Niss, M., & Højgaard, T. (2019). Mathematical competencies revisited. *Educational Studies in Mathematics*, 102, 9-28. https://doi.org/10.1007/s10649-019-09903-9

OECD (2023), PISA 2022 Results (Volume I): *The State of Learning and Equity in Education*, PISA, OECD Publishing, Paris, https://doi.org/10.1787/53f23881-en.

Olsher, S., Chazan, D., Drijvers, P., Sangwin, C., Yerushalmy, M. (2023). Digital assessment and the “machine”. In B. Pepin, G. Gueudet, & J. Choppin (Eds.), *Handbook of Digital Resources in Mathematics Education*. Springer. https://doi.org/10.1007/978-3-030-95060-6_44-1

Shepard, L. A. (2000). The role of assessment in a learning culture. *Educational Researcher*, 29(7), 4-14. https://doi.org/10.3102/0013189X029007004
# Student agency in mathematics assessment: passive targets or active agents?

Juuso Henrik Nieminen<fnref n="1" />, <fnref n="2" />

There is no shortage of examples of innovative practices in mathematics assessment research. However, surprisingly little scholarly attention has been given to the presumed role students play in assessment. Students have traditionally been seen as targets of mathematics assessment, deriving from a conceptualisation of assessment as measurement. On the other hand, assessment policies have increasingly emphasised the values of 'Assessment for Learning' that portray students as active agents. This paper proposes a research agenda for understanding student agency in mathematics assessment amid these contradictory trends. By taking a sociocultural and -political approach, I discuss how students largely remain as the 'objects' of assessment, as designed and implemented by others, for the purposes of others. I provide potential ways further if students are to be seen as meaningful agents in mathematics assessment.

Keywords: Mathematics assessment, classroom assessment, student agency, sociology of assessment.

## Setting the scene: discursive tensions in how students are positioned in assessment

Mathematics assessment research has vastly unpacked the power of classroom assessment on students' learning of mathematics. There is no shortage of exciting, innovative approaches to assessment in published research literature, particularly when it comes to digital technologies in assessment. A glance at the FAME and CERME TWG21 conference contributions from previous years confirms this: these proceedings include various impressive studies on digital assessment, summative and formative assessment, and standardised assessment, just to name a few key themes. Despite such rich developments in how mathematics assessment should best be designed and implemented, surprisingly, little scholarly attention has been given to the presumed role students play in assessment. Should students be seen as relatively passive targets of assessment who should benefit from mathematics assessment in ways that are determined primarily by educators? Or might it be more beneficial to portray students as active agents who take control of their own learning processes? While these ideas are surely discussed implicitly in a lot of the published work on mathematics assessment, they are commonly overlooked and under-theorised. How could the research community make sense of the student role – students' *agency* – amid the structures of mathematics assessment?

To answer this question, I discuss two global megatrends that surround mathematics assessment, based on Nieminen et al. (2023). These trends find their way into mathematics classrooms at many educational levels, from early to higher education. They shape not only the daily practices of mathematics but the ways in which we think and talk about assessment and student roles within.

First, how mathematics is assessed by teachers at the classroom level is vastly influenced by global testing cultures in which tests are “accepted as foundational practice in education and shape how education is understood in society and used by its stakeholders” (Smith, 2016, p. 10). Indeed, few

<footnote n="1">¹The University of Hong Kong, Faculty of Education, Hong Kong SAR; juuso@hku.hk</footnote>

<footnote n="2">²Ontario Tech University, Mitch and Leslie Frazer Faculty of Education, Oshawa, Canada</footnote>
other school subjects are tested as intensively as mathematics. There is not a similar societal urgency to screen, measure, track and predict students’ skills in music, history or geography, even though these subjects are taught widely in schools around the world. Mathematics has particular power in high-stakes testing due to its objective, *testable* nature, which perhaps partly explains why the “measurement paradigm” (Goos, 2020, p. 572) has been so prevalent in mathematics in particular. The rankings produced by international comparison studies, such as the Programme for International Student Assessment (PISA) and Trends in International Mathematics and Science Study (TIMSS), produce national media spectacled around the world (of which the Finnish shock of *losing* to Estonia in mathematics skills in PISA2022 is an excellent example!). Testing and summative examinations are frequent at the classroom level of mathematics, too, which is often attributed to the washback effect of high-stakes testing. However, the “era of testing times” (Marinho et al., 2017, p. 196) is seen in how examinations commonly dominate mathematics assessment in contexts with no high-stakes testing – nor any obligations towards testing – such as higher education (e.g., Iannone & Simpson, 2022) and low-stakes assessment contexts (e.g., Nieminen & Atjonen, 2022).

In the era of testing, students have been seen as “the objects of assessment processes with teachers or external testing bodies controlling the field” (Adie et al., 2018, p. 1). In other words, students rarely have agency over assessment policies and practices, namely, over *how, why and when* assessment is conducted. This is a particular issue in mathematics due to the high stakes of the discipline (see Nieminen et al., 2023). This was noted by Anne Watson, who, in her excellent book about care in mathematics education, portrayed assessment as an antagonist that *restricts* students’ engagement with mathematics:

> ... assessment and accountability systems push schools into a fairly limited range of practices so that many students are trained primarily to pass tests rather than being educated to become competent and confident students and users of mathematics assessment. (Watson, 2021, p. 1)

However, at the same time, recent decades have seen a strong global push towards ‘Assessment for Learning’ (AfL) policies and practices that emphasise diverse assessment ecosystems stemming from self- and peer-assessment to portfolios to digital innovations to oral assessment (Volante et al., 2024). These ‘student-centred practices’ are now at the core of mathematics curriculum documents, teacher education programmes, and professional development materials (Goos, 2020). Likewise, such practices are vastly represented in the mainstream of mathematics assessment research.

This push towards understanding mathematics assessment not only as a technology of measurement but as a tool to promote learning seems to promote an *active role* for students. ‘Student-centred’ assessment aims to activate the students themselves in their own learning processes, enabling them to take more control of their own learning of mathematics. In doing so, students are prepared for the societies of the future, as phrased by Gravemeijer et al. (2017). This idea is present in recent research initiatives on topics such as student agency, activity, engagement, self-regulation, self-efficacy, motivation, control, responsibility, ownership of learning, reflection, higher-order skills, assessment literacy, feedback literacy...

These megatrends may at first seem to contradict each other. From the student point of view, they seem to convey two messages. The era of testing portrays students as compliant objects of tracking and testing: as industrial subjects who should demonstrate their mathematical abilities in identical
and comparable ways. On the other hand, AfL positions students as active and empowered 21st century citizens who are unique and self-determined. However, a closer look shows that these two trends intertwine and strengthen each other (for the full story, please refer to Nieminen et al., 2023). Both trends ultimately see students as objects of assessment systems that are defined by experts. For example, many digital formative assessment systems train students to self-regulate their learning of mathematics exactly in ways that are defined by educators, for the purposes defined by educators.

In this way, both megatrends aim to shape students from afar, leaving little room for students' own goals, aspirations, and insights. Students are then seen as objects that can be shaped and moulded through mathematics assessment, which is far from a neutral view (Nieminen & Yang, 2024). In the remainder of this paper, I will propose an alternative by calling for a research agenda on better understanding student agency in and through mathematics assessment.

## Towards a research agenda: student agency in mathematics assessment

In my view, if mathematics assessment aims to prepare students for the unknown futures (see e.g., Gravemeijer et al., 2017), there is a need to critically examine how assessment shapes students and their agency as users of mathematics. Given the strong presence of assessment and testing in the day-to-day practices of mathematics education, it is urgent for the mathematics education research communities to unpack the issues of agency in assessment.

Overall, student agency refers to students' capacity to act autonomously and direct in one's life. The idea of agency follows long philosophical, sociological and theological traditions of mapping the relationship between human action and the surrounding social structures. Contemporary conceptualisations of agency tend to differ regarding how much emphasis they place on individuals' own agency amid such structures (see Matusov et al., 2016). Adie and colleagues (2018) contextualised broader discussions around student agency in the context of classroom assessment: 'A focus on student agency in assessment acknowledges students as actors who make choices, and whose actions shape assessment practices in both anticipated and unexpected ways.' (Adie et al., 2018, p. 2) This idea of students shaping the structures of assessment indeed lies at the core of student agency, given that students are not simply 'receivers' of assessment cultures and practices but they actively shape how assessment is conducted and talked about in mathematics (Nieminen & Lahdenperä., 2024; Nieminen & Atjonen, 2023; Nieminen & Tuohilampi, 2020).

Assessment research in mathematics has so far largely focused on individualistic approaches to student agency, examining ideas such as student self-regulation, motivation, self-efficacy, control and choice (see Nieminen et al., 2023). While such views have accumulated an important knowledge base concerning the student viewpoints on assessment, there remains a need to understand the societal, social, cultural, historical, ethical and political underpinnings of the structures of mathematics assessment. Only in this way we can understand the agency of individual students amid the structures that constitute mathematics assessment: assessment policies, institutional practices, social discourses, norms, values, and so forth.

Let me illustrate this point with an anecdote. Consider a rather traditional situation, namely, that a student crams for hours for a mathematics examination during the previous day before the test (you may well imagine this student in your own context, be it primary education in Greece or university mathematics in Australia). The student takes part in the examination and then, regardless of the final
test result, forgets most of the covered material immediately. While this may be detrimental to the student's learning of mathematics, life goes on: the teacher moves on to a new topic...

How could the mathematics assessment research community make sense of such (rather common) situations? Imagine further: you are given a massive research grant to study this phenomenon from the viewpoint of student agency. You might take an individualistic viewpoint to agency, since surely the student seems to be using her agency for the maladaptive purpose of cramming. Perhaps you could investigate the student's (apparently lacking) skills in self-regulation or her seemingly low motivation towards learning mathematics for the longer term. However, you might also reach 'beyond the individual' to investigate the structures around student agency. Maybe the immediate structures of assessment promoted such learning behaviours; maybe there was no formative assessment nor opportunities for feedback loops and cycles throughout the learning process? Investigations of social and socio mathematical norms might reveal how assessment cultures in mathematics education promote cramming. For students, cramming might be a completely normalised way of *doing mathematics*, amplified by popular culture and social media discourses around mathematics. Or, perhaps you would choose to reach even further to investigate assessment policies and testing cultures and how national and international agendas promote certain kinds of learning behaviours at the classroom level. In fact, each of these approaches provides very specific information about students' agency in assessment – and hopefully, if the grant only allows this, all these different viewpoints could come together and supplement each other!

My main point, then, is that the assessment research community should widen its repertoire to understand the social, cultural, political, historical and ethical aspects of mathematics assessment – particularly when it comes to students’ agency in assessment. Addressing such diverse viewpoints is by no means a simple task, and indeed Matusov and colleagues (2016) remind us that investigations of agency should draw on complex and multifaceted rather than on one-dimensional approaches. Next, I will discuss two theoretical ideas on how student agency could be conceptualised in mathematics assessment (research). This is, of course, not an exhaustive list but it provides some illustrations of what might be possible.

## Authorial agency over the materials of mathematics assessment

It is typically seen as the responsibility of educators, publishers and testing agencies to design the assessment materials and practices of mathematics. For example, there is plenty of academic literature that aims to produce valid and reliable tools for summative and formative assessment of mathematics, be it in the form of scalable tests, digital worksheets, or learning analytics and trajectories (see Nieminen et al., 2023). Based on this approach, assessment materials should be based on expert knowledge. While this is obviously desirable in many cases, at the same time, students are portrayed as non-experts whose voices have little value over how, when and why mathematics is assessed.

Matusov et al.’s (2016) concept of *authorial agency* is helpful in unpacking how students could be seen as meaningful contributors in mathematics assessment design. When students are provided with authorial agency, they are seen as co-creators of assessment cultures and cultural objects (e.g., test papers, self-assessment forms, assessment instructions, rubrics, syllabus documents...). Thus, rather than indoctrinating students into existing assessment cultures, the idea of authorial agency reminds us that students can be given ownership of such cultures through practices steeped in democratic
values. Mathematics classrooms are then seen as learning communities in which all members
contribute to the daily practices and materials of teaching, learning and assessment.

There is an increasing knowledge base regarding student partnership in assessment design that might be helpful here (Deeley & Bovill, 2017). Assessment partnership means that students co-design assessment practices together with educators, albeit supported and scaffolded appropriately by educators. As Deeley and Bovill (2017) explain, assessment co-design provides tangible ways to meet the abstract goals of democratic education. This approach turns assessment from mechanical practices into cultural objects in mathematical communities: students may feel ownership of the materials in such communities. In other words, a co-designed rubric is not only a tool for learning, but a cultural object that denotes the values of cultural ownership and democracy. Moreover, when students co-design, for example, formative assessment practices, they might also increase their awareness of how and why assessment is conducted in mathematics (so-called ‘assessment literacy’).

Since most assessment is at least partly digital now, the lens of authorial agency might also enable
educators to foster students’ digital agency; their capacity to shape the digital circumstances they live
in and to “control and adapt to a digital world” (Passey et al., 2018, p. 426). Students rarely have
digital agency in the context of mathematics education. Students are rarely heard when high-stakes
testing procedures are digitised; or when new digital technologies are implemented in mathematics
classrooms; or when digital platforms are designed to track student learning through learning
analytics and learning trajectories. In many ways, students are seen as consumers of technologies in
mathematics, particularly when it comes to assessment. In this context, co-designing digital
classroom assessment practices – such as digital self-assessment forms, multimodal learning
portfolios or annotated rubrics – might result into broader authorial agency when it comes to digital
technologies.

At the moment, assessment partnership remains at the margins of mathematics assessment literature. However, Tina Rapke (2016) provides an intriguing example of co-designing a test paper in the context of university mathematics. Students produced test items in groups and, in doing so, felt ownership over the design of this important learning material. What is striking about Rapke’s account is that it describes how assessment partnership may be implemented in an institutional context that does not value ‘authorial agency’ but instead accountability and compliance:

> The idea of involving students in developing the final exam emerged from an attempt to satisfy administrative expectations that students sit a traditional exam (i.e. students would sit an exam individually with only access to a pen or pencil) in my mathematics course, while holding to my philosophy that students and their instructor should play significant roles in the direction of each class and in all assessment practices. (Rapke, 2016, p. 30)

Epistemic agency over mathematical knowledge

Conversation about student agency in mathematics assessment should not disacknowledge the role
of mathematics itself in shaping students. Indeed, there is a risk of keeping the discussions around
student agency in assessment too general and universal without paying enough attention to the
disciplinary knowledge structures of mathematics. From the viewpoint of *epistemic agency*, it could
then be asked: how does assessment shape students as mathematical knowers?
The viewpoint of epistemic agency portrays assessment practices as epistemic practices. This means that assessment is conducted in ways that fit the knowledge structures of a given discipline (e.g., mathematical and musical skills are assessed in different ways) and, at the same time, assessment upholds specific epistemologies, or, ways of knowing (Nieminen & Lahdenperä, 2024). Following this approach, the test-driven culture of mathematics assessment could be explained as reflecting the *testable* epistemologies of school mathematics. Moreover, tests and examinations could be seen as upholding certain ways of knowing mathematics as tests commonly rely on individualistic, timed and standardized situations that leave little room for, for example, embodied forms of doing mathematics. In other words, assessment signals students what mathematical knowledge is and is not, and how such knowledge can be assessed.

As epistemic practices, assessment shapes students as epistemic knowers. Here, I rely on the philosophical framework for epistemic justice and injustice by Tanswell and Rittberg (2020) (see also Nieminen & Lahdenperä, 2021). They discuss the ethics of how certain forms of mathematics education shape students as thinkers and doers of mathematics. Broadly, epistemic injustice concerns the injustices of educational institutions that “affect humans specifically in their roles as epistemic agents” (Tanswell & Rittberg, 2020, p. 1200). Epistemic injustice may occur if assessment systematically and institutionally portrays students as ‘non-knowers’ of mathematics, such as by encouraging cramming cultures as discussed in the anecdote above. Likewise, epistemic injustice may occur if certain ways of knowing are systematically excluded and marginalised through mathematics assessment. For example, what is commonly called ethnomathematics is rarely represented in the assessment ecosystems of assessment, but instead, these ecosystems are built on particular ways of knowing and doing mathematics.

An apt example of epistemic injustice is introduced in Nieminen and Lahdenperä (2021). In this study, one university student criticised examinations as an assessment practice because they promoted only particular ways of being a mathematician, or, an epistemic agent in mathematics:

> A dissertation is a good way of learning, because solving problems and writing about them is what mathematicians actually do. It's not a mathematician's job to sit in a closed exam hall giving answers based on facts and solving procedures learned by heart. So it makes no sense that the education of future mathematicians should prepare you for this kind of activity. (a student in Nieminen & Lahdenperä, 2024, p. 310)

A powerful example of alternative epistemologies in assessment comes from early education in New Zealand (Anthony et al., 2015). This study introduces narrative assessment that captures mathematical learning through *stories*. These stories by children captured elements of mathematical learning that were not otherwise visible and thus promoted students' roles as knowers of mathematics. This example is illuminating since it disrupts the individualistic epistemology of mathematics: the learning stories not only make visible the learning of individuals but supported “educative partnerships with family/whanau” (Anthony et al., 2015, p. 398).

## Final words

In this paper, I have argued that the communities of research and practice must pay closer attention to student agency in mathematics assessment. This requires not only psychological but also sociological and ethical explorations of ‘agency’ that are specific to both mathematics and
assessment. I have briefly discussed two potential frameworks for student agency in mathematics assessment – authorial and epistemic agency – but these are only examples of the rich literature on agency waiting to be utilised in the context of mathematics assessment (see Matusov et al., 2016).

Considering student agency in mathematics assessment demands us educators to ask uncomfortable questions from ourselves. How could mathematics assessment *hinder* students' agency in mathematics education? Currently, many assessment practices might simultaneously promote students' learning of mathematics *and* restrict their agency as learners, humans and citizens. To unpack such paradoxical situations, the assessment research community may need to supplement the individualistic and pedagogical paradigms with social, cultural, political, sociological and critical ones, too. For example, well-designed digital platforms for formative assessment may well foster students' learning outcomes in summative tests, but at the same time, they might hinder students' authorial agency by leaving students with no choice other than to partake assessment tasks as designed by educators, test designers or perhaps private companies.

Assessment is commonly cited as the engine of mathematics education reforms (Barnes et al., 2000). This is why I see it imperative to centre the idea of student agency in *assessment* if mathematics education is to prepare students for the societies of the future. That said, student agency is not a normative concept but a risky one since students may exercise their agency in assessment in surprising ways. What kinds of opportunities might emerge if the student agency was placed at the centre of assessment? Exploring this idea might be a risk worth taking.

## References

Adie, L. E., Willis, J., & Van der Kleij, F. M. (2018). Diverse perspectives on student agency in classroom assessment. *The Australian Educational Researcher, 45*(1), 1-12.

Anthony, G., McLachlan, C., & Lim Fock Poh, R. (2015). Narrative assessment: Making mathematics learning visible in early childhood settings. *Mathematics Education Research Journal, 27*(3), 385-400.

Barnes, M., Clarke, D., & Stephens, M. (2000). Assessment: the engine of systemic curricular reform?. *Journal of Curriculum Studies, 32*(5), 623-650.

Deeley, S. J., & Bovill, C. (2017). Staff student partnership in assessment: enhancing assessment literacy through democratic practices. *Assessment & Evaluation in Higher Education, 42*(3), 463-477.

Goos, M. (2020). Mathematics Classroom Assessment. In S. Lerman (Ed.), *Encyclopedia of Mathematics Education* (pp. 572–576). Springer.

Gravemeijer, K., Stephan, M., Julie, C., Lin, F. L., & Ohtani, M. (2017). What mathematics education may prepare students for the society of the future?. *International Journal of Science and Mathematics Education, 15*, 105-123.

Iannone, P., & Simpson, A. (2022). How we assess mathematics degrees: the summative assessment diet a decade on. *Teaching Mathematics and its Applications: an International Journal of the IMA, 41*(1), 22-31.
Marinho, P., Leite, C., & Fernandes, P. (2017). Mathematics summative assessment practices in schools at opposite ends of performance rankings in Portugal. *Research in Mathematics Education, 19*(2), 184-198.

Matusov, E., von Duyke, K., & Kayumova, S. (2016). Mapping concepts of agency in educational contexts. *Integrative Psychological and Behavioral Science, 50*, 420-446.

Nieminen, J. H., & Yang, L. (2024). Assessment as a matter of being and becoming: Theorising student formation in assessment. *Studies in Higher Education, 49*(6), 1028-1041.

Nieminen, J. H., & Lahdenperä, J. (2024). Assessment and epistemic (in) justice: how assessment produces knowledge and knowers. *Teaching in Higher Education, 29*(1), 300-317.

Nieminen, J. H., Bagger, A., Padilla, A., & Tan, P. (2023). Student positioning in mathematics assessment research: A critical review. *Journal for Research in Mathematics Education, 54*(5), 317-341.

Nieminen, J. H., & Atjonen, P. (2023). The assessment culture of mathematics in Finland: A student perspective. *Research in Mathematics Education, 25*(2), 243-262.

Nieminen, J. H., & Tuohilampi, L. (2020). ‘Finally studying for myself’–examining student agency in summative and formative self-assessment models. *Assessment & Evaluation in Higher Education, 45*(7), 1031-1045.

Passey, D., Shonfeld, M., Appleby, L., Judge, M., Saito, T., & Smits, A. (2018). Digital agency: Empowering equity in and through education. *Technology, Knowledge and Learning, 23*, 425-439.

Smith, W. C. (2016). An introduction to the global testing culture. In Smith, W. (Ed.), *The Global Testing Culture: Shaping Educational Policy, Perceptions, and Practice* (pp. 7-24). Oxford: Symposium Books.

Tanswell, F. S., & Rittberg, C. J. (2020). Epistemic injustice in mathematics education. *ZDM, 52*(6), 1199-1210.

Watson, A. (2021). *Care in mathematics education: Alternative educational spaces and practices*. Springer Nature.

Volante, L., DeLuca, C., Barnes, N., Birenbaum, M., Kimber, M., Koch, M., & Wyatt-Smith, C. (2024). International trends in the implementation of assessment for learning revisited: Implications for policy and practice in a post-COVID world. *Policy Futures in Education, 14782103241255855*.
Full papers
# A mathematics teacher's implementation of formative assessment: Overcoming obstacles with adaptive professional development support

Catarina Andersson¹ and Torulf Palm¹

¹Umeå University, Sweden; catarina.andersson@umu.se; torulf.palm@umu.se

This paper focuses a mathematics teacher's implementation of formative assessment (FA) when helping students solve mathematics tasks. Such FA practice has great potential, but is non-trivial, and teachers will need substantial support for developing their beliefs and practices. We have studied why an engaged and experienced mathematics teacher who had participated in a comprehensive professional development program made certain changes but not others and how additional support helped her overcome obstacles she experienced. The study exemplifies the significance of first-hand information from teachers' classroom practices together with adapted feedback when providing professional development support for their FA development.

Keywords: Formative assessment, professional development, teacher-student interaction.

## Introduction

This paper focuses a mathematics teacher's implementation of formative assessment (FA) when helping students who work individually with mathematics tasks. Such work is frequent in mathematics education in many countries (Hiebert et al., 2003), also in Sweden (Boesen et al., 2014), but providing adequate help to students in this situation is challenging. In this study we follow one experienced primary school teacher who had participated in a comprehensive professional development (PD) program in FA and volunteered to additional individual PD support. We sought to understand why even a committed and experienced mathematics teacher who participated in a comprehensive PD did not make the changes in practice that she desired. We study the changes made, the reasons for making certain changes but not others, and features of the additional PD support. The study contributes to knowledge about crucial features of PD in FA.

## Background

### Formative assessment, its implementation, and PD support

FA is a classroom practice in which teachers and/or students elicit evidence of students' learning needs through assessment and then adapt teaching or learning to these needs. A large body of research conducted in many different subjects at all educational levels has shown that classroom practices that adhere to the principles of FA can accomplish large gains in student achievement, regardless of whether the teachers or the students were the proactive agents in the FA processes (e.g. Lee et al., 2020). This holds true also for mathematics education (Palm et al., 2017). With this background, not surprisingly, FA has been promoted in many countries.

Despite these promotions, high-quality FA is commonly not enacted in schools. Since such practices include non-trivial aspects of classroom practice, teachers need substantial support for developing both their beliefs about teaching and learning and their practices. Research on PD has identified a number of program features important for attaining desired teacher and student outcomes. Examples are instructional resources, hands-on practice, interactive feedback and discussions, time, and
engagement of school leaders and external expertise (e.g. Heitink, 2016). Also, a formative process
orientation is pointed out to be a crucial feature of PD programs in FA (Andersson & Palm, 2018).

## High-quality formative assessment and its implementation in mathematics education

The present study focuses FA practices where the teacher, as main actor: (1) elicits evidence of student knowledge and skills, (2) interprets the evidence and makes inferences about student learning needs, and (3) gives feedback adapted to these learning needs. The quality of those practices regards the quality of the evidence elicited and the process of using it as information to provide feedback that can support learning. Such FA practices are difficult and complex (Black & William, 2009), and the use of FA in mathematics accompany specific challenges and changes that rarely come easily (Burkhart & Schoenfeld, 2019). Changes may be giving up intuitive responses such as re-teaching or funneling of students' thinking toward a particular strategy or answer to be replaced by responsive actions which involve taking up and building on students' ideas and thinking (e.g. Jacobs et al., 2022).

## Context of the study

The study is part of a three-year intervention project about FA in mathematics. The research group conducted a PD for 25 mathematics teachers at two schools. It was based on previous identified important features of successful PDs (e.g. Heitink, 2016). In the second year, classroom observations revealed that even not all experienced and motivated teachers had implemented all FA practices that was supported in the PD with the quality they aimed for. Additional individual PD support was offered with the intention to target the teachers' specific needs when using FA to help students who work individually with mathematics tasks. The present study focuses on one of these teachers and the following research questions guided the study: Which aspects of the suggested FA practices did the teacher implement and which did she not?; What were the reasons for making certain changes but not others?, and What support helped her to overcome obstacles she experienced?

## Methods

### Participants

Elsa (fictious name) volunteered to participate in the study, keen to develop her teaching. She was a committed and experienced mathematics teacher and at the time teaching in Year 6.

### Procedure

Elsa audio-recorded her individual help to students during two lessons and sent the recordings to the researchers. From analyzes of the recordings the researchers formulated feedback to Elsa (with examples from the recordings), which was then discussed in a digital meeting between Elsa and one researcher as part of the additional PD. Elsa then used the feedback and attempted to improve her FA practices of her own choice, before a new cycle of co-operation started. The meetings included discussions of Elsa's views of her practice and her beliefs, experiences, difficulties, and successes. The discussions had two purposes: 1) To support her FA implementation; 2) to collect research data.

### Data collection

Five teacher-researcher meetings were audio recorded and verbatim transcribed. Other sources of data were the feedback prepared for the meetings, and the recordings from the classroom practice.
## Data analysis

To characterize the development of Elsa's FA practices, her classroom recordings were analyzed in relation to the principles and qualitative aspects of FA outlined in the PD. To identify why Elsa made certain changes but not others and to characterize the additional PD support, transcripts from the meetings were analyzed in iterative cycles to identify common themes (see Braun & Clarke, 2006).

## Results

### Elsa's developed use of suggested FA practices

Before the additional PD support, Elsa struggled to make the students share their thinking and found it difficult to provide feedback that engaged the students in their learning. In her attempts to assess students' needs, several students just answered "I don't know". Elsa tried to push them, using questions such as "What is it that is difficult?". She also used leading questions and asked series of questions with scant room for students to answer. Through her feedback, Elsa tried to encourage the students to take more responsibility but met resistance from the students. During the period of additional PD support, Elsa made progress. For example, she started to use other questions that were easier for the students to answer. She more often insisted on and provided time for students to share their thinking, and occasionally used follow-up questions. Her previous use of feedback pointing out students' successes also got more specific. In addition, from previously taking a leading role in solving the tasks based on her own ideas, she increasingly began to incorporate the students' thinking in her feedback and after giving them hints about how to proceed in their task solving leaving them alone for a while to try to use the feedback.

### Reasons for choosing to implement or not implement FA practices

The most salient reasons for not making desired changes had to do with the reactions of the students that Elsa experienced or expected. For example, several students got upset when she insisted that they should share their thinking and Elsa found these students' reactions hard to handle. Moreover, Elsa cared for students' wellbeing and wanted them to enjoy mathematics, which she feared would not be achieved if she demanded too much of them in terms of sharing their thinking or taking responsibility for solving the task. In relation to difficulties of making the students share their thinking Elsa also referred to reasons that had to do with the group constellation. The students did not trust each other and were afraid to reveal shortcomings. Many students with low self-confidence and varying needs made it difficult for Elsa to consistently use the FA practices. Another reason for not using desired FA practices was that she simply forgot, and she referred to the difficulties of abandoning old habits.

### Support helping the teacher overcome obstacles

The additional PD provided Elsa with feedback identifying successful uses of the FA practices suggested in the PD, as well as hints about how to develop. Moreover, during the feedback sessions difficulties and reasons (see above) for not using desired FA practices were identified and explored in the dialogue between Elsa and the researcher, and viable ways to implement FA practices were negotiated. Examples of such ways were: introducing changes step by step starting with students who were receptive to changes, providing support and space for students to practice new behaviors, and feedback reinforcing the students' experiences of the benefits of, for example, sharing their thinking.
## Conclusion

The research-based PD program provided insufficient support for the use of high-quality FA when helping students individually. The additional PD helped the teacher to make progress regarding both assessment and providing feedback. The study exemplifies the significance of first-hand information of teachers' classroom practices together with adapted feedback for conducting PDs that accomplish large gains in teacher development of FA in the mathematics classroom.

## References

Andersson, C., & Palm, T. (2018). Reasons for teachers' successful development of a formative assessment practice through professional development—a motivation perspective. *Assessment in Education: Principles, Policy & Practice, 25*(6), 576–597.
https://doi.org/10.1080/0969594X.2018.1430685

Black, P., & William, D. (2009). Developing the theory of formative assessment. *Educational Assessment, Evaluation and Accountability, 21*, 5–31. https://doi.org/10.1007/s11092-008-9068-5

Boesen, J., Lithner, J., & Palm, T. (2010). The relation between types of assessment tasks and the mathematical reasoning students use. *Educational studies in mathematics, 75*(1), 89–105.
https://doi.org/10.1007/s10649-010-9242-9

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. *Qualitative research in psychology, 3*(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Burkhardt, H., & Schoenfeld, A. (2019). Formative assessment in mathematics. In H. L., Andrade, R. E. Bennett, & G. J. Cizek, (Eds.). *Handbook of formative assessment in the disciplines* (pp. 35–67). Routledge. https://doi.org/10.4324/9781315166933

Heitink, M. C., Van der Kleij, F. M., Veldkamp, B. P., Schildkamp, K., & Kippers, W. B. (2016). A systematic review of prerequisites for implementing assessment for learning in classroom practice. *Educational Research Review, 17*, 50–62. http://dx.doi.org/10.1016/j.edurev.2015.12.002

Hiebert, J. (2003). *Teaching mathematics in seven countries: Results from the TIMSS 1999 video study*. DIANe Publishing.

Jacobs, V. R., Empson, S. B., Jessup, N. A., Dunning, A., Pynes, D. A., Krause, G., & Franke, T. M. (2022). Profiles of teachers’ expertise in professional noticing of children’s mathematical thinking. *Journal of Mathematics Teacher Education, 1–30*. https://doi.org/10.1007/s10857-022-09558-z

Lee, H., Chung, H. Q., Zhang, Y., Abedi, J., & Warschauer, M. (2020). The effectiveness and features of formative assessment in US K-12 education: A systematic review. *Applied Measurement in Education, 33*(2), 124–140. https://doi.org/10.1080/08957347.2020.1732383

Palm, T., Andersson, C., Boström, E. & Vingsle, L. (2017). A review of the impact of formative assessment on student achievement in mathematics. *Nordic Studies in Mathematics Education, 22*(3), 25–50.
# Enhancing Teacher Training in Mathematics Education: A Model for a Semiotic Approach to Feedback and Interpretative Knowledge

Miglena Asenova<fnref n="1" />, Agnese Del Zozzo<fnref n="2" /> and Marzia Garzetti<fnref n="3" />

This research introduces a comprehensive model for a teacher training course centered on Semiotic Interpretative Knowledge (SIK) in mathematics education. Highlighting the critical need for specialized training, the course is designed to refine teachers’ abilities to interpret student’s responses through a semiotic lens, especially when conceptual knowledge remains hidden behind difficulties related to patterns of sign use. It focuses on equipping educators with advanced semiotic interpretation skills, thereby enhancing their capability to offer deeper, more meaningful, and effective feedback in mathematics classrooms. The model not only delineates the key features of the designed course but also lays a foundation for future investigations into its feasibility and impact.

Keywords: Teacher training, feedback, semiotic functions, Semiotic Interpretative Knowledge.

## Research problem and rationale of the paper

In mathematics education research, the types of feedback that teachers spontaneously provide in their mathematics classrooms are investigated by several studies (e.g., Galleguillos & Ribeiro, 2019; Santos & Pinto, 2010; Stovner & Klette, 2022). The main aspects of feedback that emerge from such research concern conceptual, strategical, or procedural features, while the semiotic aspects related to sign use and production are never explicitly considered. As research shows (e.g., D'Amore & Fandiño Pinilla, 2007; Iori, 2018; Santi, 2011), interpreting student's reasoning requires a strong semiotic competence on patterns of sign use and production. Asenova et al. (2023a) define the notion of Semiotic Interpretative Knowledge (SIK) as "the knowledge needed by teachers in order to interpret students' answers (...), and to give appropriate feedback to them, when conceptual knowledge is hindered, and thus remains hidden behind difficulties related to patterns of sign use and production" (p. 11). In Asenova et al. (2023b) an investigation carried out with 180 Italian prospective primary school teachers shows that prospective teachers spontaneously use a wider web of semiotic resources when they are asked to provide feedback to students, rather than when they are asked to interpret the student's solutions for themselves. Prospective teachers seem to implicitly assume that feedback effectiveness grows with increasing use of semiotic resources, but the above-mentioned study shows also that they often fail in this because of a lack of awareness on the semiotic transformations involved in their feedback. The necessity to consider SIK as a kind of mathematical knowledge for teaching (Ball et al., 2008) and as an explicit content area in teacher training, specifically in reference to feedback effectiveness, goes with a lack of research on how to implement teacher training focused on SIK. This paper presents a model for teacher training courses which address this issue.

<footnote n="1">¹Free University of Bolzano, Faculty of Education, Bolzano, Italy; miglena.asenova@unibz.it</footnote>

<footnote n="2">²University of Trento, Department of Mathematics, Trento, Italy; agnese.delzozzo@unitn.it</footnote>

<footnote n="3">³University of Genoa, Department of Mathematics, Genoa, Italy; garzetti@dima.unige.it</footnote>
# Theoretical framework

## The semiotic dimension of mathematical knowledge for teaching

Starting from research related to the conceptualization of Mathematical Knowledge for Teaching (MKT) (Ball et al., 2008), Ribeiro and co-authors introduce the notion of Interpretative Knowledge (IK) as the part of the mathematical knowledge "that allows teachers to give sense to pupils' non-standard answers (i.e., adequate answers that differ from the ones teachers would give or expect) or to answers containing errors" (Ribeiro et al., 2016, p. 9).

The semiotic aspects of IK are still little explored, but at the same time research shows that a strong semiotic competence is indispensable for a cognitively meaningful mathematical activity. According to Duval (2017), in mathematics, ostensive references are impossible, as we cannot directly access mathematical objects through our senses. Conceptualization itself, called noesis, is identified in mathematics with a complex coordination of several semiotic systems (called semiosis), rooted in semiotic transformations within the same semiotic system (treatments) and semiotic transformations between different semiotic systems (conversions) (Duval, 2017). A semiotic system (or register) is defined by Duval (1995) and Ernest (2006) as composed by: (1) a set of basic signs that only have meaning when set against or in relation to other basic signs (e.g., the meaning of the digits within the decimal number system); (2) a set of rules for the production of signs, starting from basic signs, and for their transformation. According to Duval, D'Amore (2003) identifies conceptualization with the following semiotic functions, specific to mathematics: (1) choice of the distinctive features of a mathematical object and its representation in a semiotic system; (2) treatment in the same semiotic system; (3) conversion between semiotic systems. The management of such semiotic complexity, within the structure of semiotic systems and the processing of semiotic functions, comes up against Duval's famous cognitive paradox (Duval, 2017): on the one hand, the student comes to know the abstract mathematical objects only through the semiotic activity; on the other hand, such a semiotic activity requires the student's conceptual knowledge of the mathematical objects involved in it. According to Duval, a mathematical object, intended as an epistemic object, arises as the invariant behind treatments and conversions and thus requires the interplay of at least two semiotic registers.

For the reasons displayed above, taking into account the intrinsically semiotic nature of mathematical thinking, Asenova et al. (2023a) introduce the theoretical construct of Semiotic Interpretative Knowledge (SIK) as "the knowledge needed by teachers in order to interpret students' answers (be they standard or non-standard), as well as students' behavior, and to give an appropriate feedback to them, when conceptual knowledge is hindered, and thus remains hidden behind difficulties related to patterns of sign use and production, including individual creativity in sign use" (p. 11). SIK is a kind of MKT that is both subject- and pedagogy-related because the control of the semiotic functions is intertwined both with mathematical knowledge (noesis and semiosis are overlapped) and their implementation in the teaching-learning activity driven by the teacher.

## The feedback dimension

Feedback is defined by Hattie and Timperley (2007) as "information provided by an agent (e.g., teacher, peer, book, parent, self, experience) regarding aspects of one's performance or understanding" (p. 81). Galleguillos and Ribeiro (2019) investigate prospective teachers' ability to use IK in giving feedback: teachers were asked to solve a task in small groups and then provide
feedback to chosen solutions given by students to the same task. These authors classify the provided feedback into four categories: (a) Feedback on how to solve the problem; (b) Confusing feedback: When the feedback seems to be correct, but it can be confusing for the student; (c) Counterexample as feedback; (d) Superficial feedback: The content of such feedback was insufficient (too broad or inconsistent) to allow the solver to understand its meaning. In Asenova et al. (2023b) the authors develop the kinds of feedback introduced by Galleguillos and Ribeiro, consistently with the notion of SIK. In particular, starting from answers given by prospective teacher to similar tasks as the ones proposed by Galleguillos and Ribeiro (2019), the authors categorized the collected feedback according to the implementation of the semiotic functions: type (i) - no mention of semiotic functions, which is framed by Galleguillos and Ribeiro's categories; type (ii) - use of semiotic representations confined to the recognition of the distinctive features; type (iii) - use of distinctive features and treatments; type (iv) - use of distinctive features, treatments, and conversions. The semiotic categorization of feedback does not provide levels of effectiveness per se, but it represents a tool able to tune sign use in producing and evaluating feedback, by identifying levels of complexity of semiotic activity. In this sense, it reduces the gap "between what is understood and what is aimed to be understood" (Hattie & Timperley, 2007, p. 82) on feedback in teacher training and it allows to consider SIK within the categories of mathematical knowledge for teaching.

**Research questions and aim**

A strong SIK is needed by teachers both to interpret student's solutions, especially when conceptual aspects are hindered by difficulties related to patterns of sign use and production, as well as to provide effective feedback (Asenova et al., 2023a, 2023b). Starting from this assumption and building upon the work of Ribeiro et al. (2016), the present study aims to present a design for a teacher training course on SIK. More specifically, the research question addressed here is: *What are the characteristics of a teacher training course which develops SIK in relation to feedback exchange and production?*

The characteristics of the course will be described, emphasizing their link to the theoretical framework introduced. Additionally, it will be discussed how these characteristics correlate with expectations regarding the processes engaged by the participants throughout the course. The development of a strong SIK on the part of the prospective or in-service teachers is a significant step towards enabling them to understand and be effective in the management and support of student's learning and strategies, particularly in relation to the use of semiotic functions. This proposal serves as a theoretical and methodological basis for future research on the feasibility and effectiveness of the training course described.

**The model for SIK operationalization in teacher training courses**

The basic structure of the proposed model is composed of five phases, each associated to an explicit goal (Table 1).
<table><caption>Table 1: Model of a cycle of SIK operationalization in feedback in teacher training courses</caption><thead><tr><th></th><th>Course phases</th><th>Main goal</th></tr></thead><tbody><tr><td>1</td><td>Introduction to semiotic functions and their noetic correspondences (step 1) and to semiotic transformations (step 2)</td><td>Introduction of tools for the development of SIK</td></tr><tr><td>2</td><td>Task 1: Giving written feedback to student's solutions working in small groups</td><td>First implementation of SIK</td></tr><tr><td>3</td><td>Task 2: Written peer-to-peer evaluation to the feedback provided by another group according to criteria related to semiotc functions</td><td>Metareflection on SIK implementation in relation to semiotic functions</td></tr><tr><td>4</td><td>Exchange of feedback and reflection in the groups</td><td>Metareflection on SIK evaluation: analysis on one's own work</td></tr><tr><td>5</td><td>Feedback of the teacher educator during whole class discussion</td><td>Institutionalization</td></tr></tbody></table>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

In the following the five phases are analyzed into detail and the relationships between the proposed tasks and expectations about the course are unraveled.

**Phase 1.** During the first phase the participants are introduced to the aspects related to the semiotic functions and their role in the recognition of the mathematical object as invariant behind treatments and conversions. In this phase the definition of conceptualization given by D'Amore (2003) referring to the semiotic functions is strongly used and exemplified. During step 1, focusing on D'Amore's definition of conceptualization, some examples are discussed with the participants recalling their attention to the properties of mathematical objects and the representations of their distinctive features. In Figure 1 four such examples are presented. In the first example (a), choosing the representation on the left highlights the distinctive feature of a fraction as related to equal areas; choosing the representation on the right highlights the distinctive feature of a fraction as related to areas of congruent figures. In the second example (b), only one of the representations ($\frac{2}{3}$ of $\overline{AB}$) highlights the distinctive features of a fraction as an operator (on a magnitude). In the third example (c), choosing the representation on the left brings out the distinctive features of the parallelogram as a quadrilateral with two pairs of parallel sides and, consequently, two pairs of congruent angles; choosing the representation on the right brings out the distinctive feature of the parallelogram as a quadrilateral, but by looking at it as an icon rather than recalling its properties as a bidimensional geometric figure. In the fourth example (d), the initial representation ($\frac{18}{36}$) emphasizes the distinctive features of probability as the ratio of favorable outcomes to total outcomes; the subsequent representation ($\frac{1}{2}$) still portrays probability as a ratio, but does so in a more abstract way: for each of the favorable outcomes, there are two possible outcomes; the third representation (50%) still carries some distinctive feature of probability as a ratio, but only mediated by the meaning of percentage ('per cent' as 'per hundred', from the Latin word 'cento', 'hundred'): there are 50 favorable cases out of 100 possible cases; the fourth representation (0.5) highlights the distinctive feature of probability as a real number in the interval [0;1]. This introduction is especially significant in raising teachers' awareness about the critical attention needed when selecting one representation over another, and the unique characteristics that such representations can either emphasize or conceal.
<figure><img src="image_19.png" /><figcaption>Figure 1: Examples of choice and representation of distinctive features of mathematical objects and of semiotic transformations to be discussed with the participants</figcaption></figure>

Step 2 of phase 1 focuses on semiotic transformation and the participants are introduced to the meaning of treatments and conversions. For this purpose, examples such as those displayed in Figure 1e are presented and discussed with the participants. A conversion between 4 in the decimal arithmetic semiotic system and an iconic representation belonging to the semiotic system of pictograms is accomplished. Then, two treatments within the decimal arithmetic semiotic system are accomplished; in this system there are established rules (how to transform 4 in $\frac{4}{1}$ and $\frac{4}{1}$ in $\frac{12}{3}$) that allow to perform the transformations within the system. In the third row, a conversion between the algebraic semiotic system and the cartesian semiotic system is represented. It allows to evidence the correspondences between the distinctive features of the mathematical object in the semiotic systems: the coefficient ‘3’ corresponds to the slope in the cartesian system; the constant ‘-1’ corresponds to the intercept on the y-axis; each ordered pair (x; y) of numbers that satisfies the equation $y = 3x - 1$ corresponds to a point on the line in the cartesian representation. It is important to point out these aspects because, according to Duval (2017), to support conceptualization, it is not enough to provide representations in different semiotic registers, but it is necessary to make explicit the correspondences between the distinctive features in the different registers. Furthermore, Duval (2017) stresses that conversions are necessary for the conceptualization of the mathematical object, but that the change of semiotic system often makes lose the meaning behind the performed transformations. Teachers often overlook this crucial aspect: they are already familiar with the concept that remains constant across various representations in different systems. However, students frequently miss this invariant and perceive the representations as distinct entities (e.g., students do not recognize a representation of the same object ‘line’ behind the algebraic and the cartesian representations). But not only conversions lead to a loss of meaning; as D’Amore and Fandiño Pinilla (2007) show, also treatments often lead to a loss of the invariant behind the transformations. Furthermore, it is important to stress that the choice of the distinctive features (properties of the object to be represented) is not independent on the choice of the representation system, as the possibility to represent strictly depends on the semiotic resources provided by the system (D’Amore & Fandiño Pinilla, 2007).

In summary, it is crucial for educators to offer representations in various semiotic systems for two reasons: firstly, identifying an invariant (mathematical object) across different contexts (semiotic systems) requires at least two such systems; and secondly, a single semiotic system is often insufficient to depict all the key features of a mathematical object that students need to understand to
fully grasp the concept. The subsequent stages of the model are designed to assist participants in reflecting upon the semiotic perspective introduced within the context of exchanging feedback.

**Phase 2.** During the second phase of the model, the participants are asked to work in small groups on Task 1 that requires to provide written feedback to student's solution of a task. No specific guidelines are provided on the criteria to be adopted when giving feedback, allowing participants the freedom to use the methods they deem most suitable. It is important to choose tasks that drive the use of semiotic functions, particularly conversions involving symbolic language, natural language, and figural representations. Usually, solutions that contain errors are selected as they more effectively motivate teachers to offer feedback. However, correct yet unconventional solutions that encourage the application of semiotic functions are also suitable. For this purpose, the example proposed by Ribeiro et al. (2016) is particularly suitable (Figure 2), because it involves a nonstandard procedure and the use of only an iconographic register not involving symbolic or verbal ones. Similar problems and their solutions provided by students can be given, involving different registers and strategies.

<figure><img src="image_20.png" /><figcaption>Figure 2: Mariana's solution to the task presented in Ribeiro et al. (2016)</figcaption></figure>

**Phase 3.** In the third phase of the model, the groups are asked to exchange their feedback with another group and to work on Task 2. This requires them to give a peer-to-peer assessment to the feedback provided by the other group in Task 1. In this case, criteria for the evaluation-feedback are provided. These criteria are the following: (1) Does the choice of distinctive features to represent in the feedback seem appropriate? (2) Does the feedback involve treatments, i.e. semiotic transformations within the same semiotic system? (3) Does the feedback use conversions, i.e. transformations between different semiotic systems? (4) Are the representations chosen in the feedback related to the representation used by the student in the solution? Some examples are provided of how each criterium could be concretely exemplified in relation to the solution provided by the student: for instance, if the teachers want to support the use of a symbolic register in relation to fraction, feedback to Mariana's strategy must go toward this direction. Criteria are provided at this stage to directly foster meta-reflection on using semiotic functions. Additionally, evaluating others' feedback is believed to prompt a detachment from the content, resulting in a more genuine reflection (Grion & Tino, 2018).

**Phase 4.** In the fourth phase of the model, the groups return their peer-to-peer-feedback to the other group and reflect on the evaluation received from their peers. Providing feedback to another group fosters reflection and offers an opportunity to gain a fresh perspective on the task, enabling to revise and enhance one's own original solving strategies. This is accomplished by reflecting on the evaluation of the appropriateness of the semiotic resources used in providing feedback to the student's solution and on their functionality in supporting the development of the semiotic functions. The feedback provided by others helps to bridge the gap "between what is understood and what is aimed to be understood" (Hattie & Timperley, 2007, p. 82) regarding the appropriateness of one's SIK.
**Phase 5.** In the fifth and final phase, feedback is provided to the participants by the teacher educator. This occurs in two distinct manners: firstly, through addressing questions that emerge during the activities of the previous phase, and secondly, through the discussion of general issues related to providing suitable feedback to students. Although it is recognized that there is no universally 'correct' feedback and 'ideal solutions' are intentionally not offered, an evaluation on the appropriateness of the implemented semiotic functions is carried out. The objective of this phase is the institutionalization of the new knowledge that has been developed. The focus should here be posed particularly on a comparison of the effectiveness of the chosen semiotic systems and semiotic representations, as well as on the semiotic transformations carried out in providing the feedback and their role in improving one's SIK.

The five phases of the basic model of SIK operationalization in relation to feedback have been presented. The model is conceived as cyclic: the repetition of the model on different problems allows work on semiotic functions in relation to various mathematical objects and varied strategies.

**Conclusive remarks**

A cohesive model aligned with the theoretical foundations of semiotic functions and IK has been crafted: the characteristics of the course have been presented allowing to work in the direction of defining structured design principles. The course design supports prospective and in-service teachers in consciously referring to semiotic functions while interpreting and providing feedback to students. This approach promotes openness to unconventional or incorrect strategies, highlighting effective conceptual or strategic elements that may be obscured by atypical or incorrect representation usage. The course structure demands focused communicative effort, especially regarding feedback mechanisms and representation choices. It underscores the importance of developing the ability and willingness to understand others' mathematical viewpoints. This comprehension is linked to feedback and interpretations within the SIK framework, recognizing that the selection of a semiotic register reflects an individual's conceptual categories for interpreting reality.

Overall, this contribution lays the groundwork for defining a set of design principles and assessing the effectiveness of courses aimed at developing SIK. It defines the crucial aspects in the foundation of the knowledge needed by the teacher for its development and thus the indicators for its assessment.

**References**

Asenova, M., Del Zozzo, A., & Santi, G. (2023a). Unfolding Teachers' Interpretative Knowledge into Semiotic Interpretative Knowledge to Understand and Improve Mathematical Learning in an Inclusive Perspective. *Education Sciences, 13*(1), 65. https://doi.org/10.3390/educsci13010065

Asenova, M., Del Zozzo, A., & Santi, G. (2023b). From Interpretative Knowledge to Semiotic Interpretative Knowledge in prospective teachers' feedback to students' solutions. In M. Ayalon, B. Koichu, R. Leikin, L. Rubel, M. Tabach (Eds.), *Proceedings of the 46th of the International Group for the Psychology of Mathematics Education (PME46) (Vol. 2, pp. 51–58)*. University of Haifa and PME.

Ball, D.L., Thames, M.H., & Phelps, G. (2008). Content knowledge for teaching: What makes it special? *Journal for Teacher Education, 59*(5), 389–408. https://doi.org/10.1177/0022487108324554
D'Amore, B. (2003). La complexité de la noétique en mathématiques ou les raisons de la dévolution manquée. *For the Learning of Mathematics, 23*(1), 47–51.

D'Amore, B., & Fandiño Pinilla, M.I. (2007). How the sense of mathematical objects changes when their semiotic representations undergo treatment and conversion. *La matematica e la sua didattica, 21*(1), 87–92.

Di Martino, P., Mellone, M., & Ribeiro, M. (2019). Interpretative Knowledge. In S. Lerman. *Encyclopedia of Mathematics Education* (pp. 1–5). Springer. https://doi.org/10.1007/978-3-319-77487-9_100019-1.

Duval, R. (1995). *Sémiosis et Pensée Humaine: Registres Sémiotiques et Apprentissages Intellectuels*. Peter Lang.

Duval, R. (2017). *Understanding the Mathematical Way of Thinking: The Registers of Semiotic Representations*. Springer.

Ernest, P. (2006). A semiotic perspective of mathematical activity: The case of number. *Educational Studies in Mathematics, 61*(1-2), 67–101. https://doi.org/10.1007/s10649-006-6423-7

Galleguillos, J., & Ribeiro, M. (2019). Prospective mathematics teachers’ interpretative knowledge: Focus on the provided feedback. In U. T. Jankvist, M. Van den Heuvel-Panhuizen, & M. Veldhuis (Eds.), *Proceedings of the Eleventh Congress of the European Society for Research in Mathematics Education (CERME11)* (pp. 3281–3288). Freudenthal Group & Freudenthal Institute, Utrecht University and ERME.

Grion, V., & Tino, C. (2018). Verso una “valutazione sostenibile” all’università: percezioni di efficacia dei processi di dare e ricevere feedback fra pari. *Lifelong Lifewide Learning, 14*(31), 38–55. https://doi.org/10.19241/lll.v14i31.104

Hattie, J., & Timperley, H. (2007). The Power of Feedback. *Review of Educational Research, 77*(1), 81–112. https://doi.org/10.3102/003465430298487

Iori, M. (2018). Teachers’ awareness of the semio-cognitive dimension of learning mathematics. *Educational Studies in Mathematics, 98*(1), 95–113. https://doi.org/10.1007/s10649-018-9808-5

Ribeiro, C.M., Mellone, M., & Jakobsen, A. (2016). Interpreting students’ non-standard reasoning: Insights for mathematics teacher education. *For the Learning of Mathematics, 36*(2), 8–13.

Santi, G. (2011). Objectification and semiotic function. *Educational Studies in Mathematics, 77*(2-3), 285–311. https://doi.org/10.1007/s10649-010-9296-8

Santos, L., & Pinto, J. (2010). The evolution of feedback practice of a Mathematics teacher. In M.F. Pinto, & T.F. Kawasaki (Eds.), *Proceedings of the 34th Conference of the International Group for the Psychology of Mathematics Education (PME34)* (Vol. 4, pp. 145–152). PME.

Stovner, R.B., & Klette, K. (2022). Teacher feedback on procedural skills, conceptual understanding, and mathematical practices: A video study in lower secondary mathematics classrooms. *Teaching and Teacher Education, 110*(1), 1–12. https://doi.org/10.1016/j.tate.2021.103593
# Exploring metrics: elementary mathematics teachers' evaluation of digital geometry assessment activities

Hassan Ayoob¹ and Shai Olsher²

University of Haifa, Faculty of Education, Israel; ¹hassan.ayoob@gmail.com;

²olshers@edu.haifa.ac.il

This study concentrates on metrics that guide teachers in perceiving Digital Formative Assessment (DFA) as suitable for formative assessment for students and aligning with the curriculum. This is part of a larger study that explores the efficiency of a digital formative assessment platform for elementary school geometry. The research utilized open coding to analyze responses from a teacher questionnaire, exploring elementary math teachers' perspectives on DFA. This research was conducted with nine mathematics teachers from diverse Israeli elementary schools. Using a teacher questionnaire and 12 DFA activities, the study identified 11 codes categorized into three categories: Information provided, type of task, and student interaction.

Keywords: Digital formative assessment, elementary math teachers' perspectives, curriculum alignment.

## Introduction and theoretical background

Formative assessment is a continuous, multifaceted process integrated into the daily dynamics of teaching and learning. It unfolds through ongoing interactions between teachers and students, wherein teachers adjust their instructional methods and activities based on assessment information (Black & William, 2009). The primary goal is to enhance the learning processes and improve student outcomes. According to Black et al. (2004), the essence of formative assessment lies not only in the assessments themselves, but also in the roles they play in supporting student learning and providing evidence for adapting teaching methods to address specific learning needs. Approaching formative assessment from this functional perspective underscores that its successful implementation hinges on the learning approach adopted and the adept use of knowledge, skills, and strategies by teachers in executing intricate pedagogical processes (Webb & Jones, 2009). For effective formative assessment, teachers' proficiency in regularly collecting and interpreting student learning data is crucial. This complex task extends beyond simple data collection, incorporating timely and constructive feedback, understanding of student learning goals, and tailoring teaching methods accordingly (Black & William, 1998; Heritage, 2007). Central to formative assessment is the role of feedback, which should be continuous, precise, and focused to guide both educators and learners. Feedback is vital for reflective learning, helping students recognize their progress and areas for improvement. Feedback also provides teachers with valuable insights about their teaching methods, by identifying areas where students struggle and understanding students' learning styles. This process helps teachers refine their instructional strategies and make informed decisions to enhance their teaching (Black & William, 1998; Hattie & Timperley, 2007; Heritage, 2007). While many researchers and teachers are aware of the importance of formative assessment for improving the quality of learning and increasing achievement, several factors, such as lack of time for implementation and challenges in adapting tasks, negatively influence formative assessment's implementation in the classroom (Brown, 2003).
In addition, many researchers agree that teachers' perceptions are one of the main factors impacting implementation in the classroom (Brown, 2003; Black & William, 2009). Assessment, especially formative assessment, is important in the mathematics teaching and learning process, as it helps realize the curriculum's objectives by adapting teaching (Black et al., 2003; Millar, 2016). In recent years, teaching with digital environments in general and geometry in particular has increasingly relied on technology. By offering dynamic forms of mathematical representation of different concepts, dynamic geometry environments (DGEs) provide students with access to mathematical concepts that they have not previously perceived (Leung, 2008; Butler et al., 2010). The suggested DFA activities are designed to effectively support the identification, classification, and analysis of student work methods to the relevant stakeholders (Ayoob & Olsher, 2023). The activities include rich tasks with an infinite number of correct solutions, in the form of the example-eliciting task (EET) (Olsher et al., 2016). These (DFA) tasks feature interactive feedback that highlights the students' exploration and reasoning beyond right or wrong answers. Such design promotes an in-depth interaction with geometric concepts, enabling students to articulate and reflect on their thought processes (Olsher et al., 2016), thereby facilitating a critical examination of their reasoning strategies (Stacey & William, 2013). This type of task involves student-centered assessment that can provide teachers and students with the characteristics of each student's work (Olsher, 2022).

## Methodology

The qualitative methodology in this research involves using open coding to analyze responses from an open-ended teacher questionnaire. The focus of this study is to systematically examine elementary mathematics teachers' perspectives regarding computerized activities, specifically digital formative assessment (DFA).

This study is part of a larger research project that explores the efficiency of DFAs for elementary school geometry. Our goal in this report is to describe the metrics according to which elementary mathematics teachers perceive digital formative assessment activities as appropriate for their students and the content of the curriculum. To achieve this goal, we seek to answer the following research questions: What metrics do math teachers use when evaluating the suitability of a digital formative assessment (DFA) to the content and skills detailed in the curriculum?

## Research setting

In this study, we analyzed the responses gathered through an open-ended questionnaire. Using an open coding qualitative method, we systematically examined teachers' insights to identify and categorize the metrics they provide. We aimed to address our research question by delineating emerging patterns and themes. This approach enabled us to uncover rich, context-specific information about teachers' perspectives on the effectiveness and alignment of DFA's with curriculum goals and student skills.

## Population

Nine mathematics teachers from five different Arabic-speaking elementary schools in northern Israel. All schools teach according to Israel's national curriculum. The participants are a diverse group of teachers with varying educational backgrounds and levels of experience. The majority held master's degrees, with teaching experience ranging from 8 to 25 years, in the 3rd to 6th grades. Collectively,
the teachers implemented 69 digital formative assessment activities (and answered the accompanying questionnaires), showing active participation in the research.

## Research tools

To examine the metrics according to which elementary mathematics teachers evaluate DFA activities as appropriate for students and the curriculum content, we used a teacher questionnaire and DFA units designed in the STEP-MFA system.

The **12 activities** were designed according to the design principles set out in previous research (Ayoob & Olsher, 2023). The activities align with the textbooks and with the recommended instruction and distribution of the topic to teaching hours according to the curriculum. The tasks were designed in the STEP-MFA environment, and the automatically assessed characteristics of student submissions were designed according to the misconceptions and perceptions in previous literature and the common errors for each subject.

<figure><img src="image_21.png" /><figcaption>Figure 1: Three tasks of the obtuse triangle altitude activity: (a) construct altitudes from the marked vertex, (b) construct three different examples of obtuse triangles and their altitudes, and (c) determine how many altitudes are outside the triangle?</figcaption></figure>

The following are examples of DFA activity: The mathematical topics studied in this study were the triangle's altitude and polygon area. The activity about the altitudes of an obtuse triangle is shown in Figure 1. The first task (figure 1a) shows three instances of the same obtuse triangle. In each instance, the student is asked to drag the bold vertex to construct an altitude from that vertex and submit it. The triangles are static, and students can drag only the bold vertices. In the second task (Figure 1b), students were asked to construct three different examples of obtuse triangles and then construct one altitude by dragging a point from a vertex. Students can drag the vertices of a triangle to create obtuse triangles of their choices. In the third task (Figure 1c), students are asked to specify how many altitudes can be constructed that are external to the static obtuse triangle and to construct them again by dragging points from the vertices to “stretch” segments. On the left side of each task, students have the option to select and use several tools, such as the option to display the lines extending the sides or a right-angled triangle ruler.

The **questionnaire** consisted of 12 questions, each featuring a dual structure. The first part involves a query with responses based on a linear or multiple-choice format. The subsequent section required the participants to explain their selection. For this study's purpose, we focused on six open-ended
questions pertaining to qualitative analysis, concentrating solely on verbal explanations. Question 3 directly addressed indices, inquiring about the metrics used to determine the suitability of a task as a formative assessment activity. Additionally, Question 6 sought an explanation for choosing the tasks deemed most suitable for the students. Question 7 sought an explanation of the rationale behind selecting tasks that were considered unsuitable for students. Question 8 sought an explanation for choosing tasks that provided information about students. Question 9 aimed to understand the reasoning behind choosing tasks that did not offer information about the students. Finally, Question 12 explored the preference for a specific report among the four reports available on the platform.

## Research setting

Preparatory stages were undertaken before introducing activities to the classroom for both teachers and students on the STEP platform. Throughout the academic year, our focus was on ensuring that teachers promptly conducted assessments after teaching each sub-topic. The system diligently recorded all the submissions for every student. After the assessment and after any discussions with students about their submissions, the teachers completed the questionnaire. In this questionnaire, they elaborated on their perspective regarding the tasks as formative assessment tasks and explained why they deemed them appropriate.

Following the completion of all activities and the collection of teachers' responses, we initiated a coding process for teachers' answers, categorizing them based on their characteristics. The coding process and questionnaire were validated by the Mathematics Education Research and Innovation Center (MERI) team comprising graduate students, curriculum developers, and researchers in mathematics education.

## Data collection and analysis

The data for this study were obtained from the responses of nine teachers to six open-ended questions in the questionnaire, following the completion of each of the 12 research activities.

The data analysis process in this study followed a qualitative approach, primarily utilizing open coding to extract meaningful insights from the responses obtained through a teacher questionnaire. Initially, teachers' answers were transcribed and organized into sheets by question. Open coding involves systematically examining and labeling data segments with descriptive codes, capturing the essence of each unit. Through constant comparison, new data were compared with existing codes to refine the categories and identify patterns and variations. Categories were developed to represent common themes emerging from teachers' insights, progressing from specific codes to more generalized concepts. To ensure the validity of our analysis, a validation process was implemented, cross-checking preliminary findings with the MERI team and validating over 15% of the data. In all cases, disagreement discussions led to consensus and, in some cases, modification of the coding scheme.

## Results

In our exploration of teachers' responses to the questionnaire, a comprehensive dataset of 69 statements for each question was collected. The coding process yielded 11 distinct codes.
Subsequently, these codes were aggregated and structured into three overarching categories, as detailed in Table 1. **Information provided**, **type of task**, and **student interaction**. A detailed breakdown of these categories is provided below.

<table><caption>Table 1: Categories and codes for Teachers' responses on the suitability of activities as formative assessment</caption><thead><tr><th>category</th><th>codes</th></tr></thead><tbody><tr><td rowspan="4">Information provided</td><td>Classroom or group information</td></tr><tr><td>Information about the student (Individual)</td></tr><tr><td>Correctness</td></tr><tr><td>Student work methods and misconceptions</td></tr><tr><td rowspan="4">Type of task</td><td>Curriculum objectives</td></tr><tr><td>Difficulty levels</td></tr><tr><td>Variety of tasks</td></tr><tr><td>Suitability for a diversity of students</td></tr><tr><td rowspan="3">Student interaction</td><td>Task modification</td></tr><tr><td>Number of submitted examples</td></tr><tr><td>Thought development</td></tr></tbody></table>

Following table 1, and to elucidate the categories and their corresponding sub-categories, we will outline each one followed by its sub-categories. For each sub-category, we'll provide an example of a teacher's verbal response. This approach aims to clarify the rationale behind our categorization and selection process, offering insights into the specific teacher feedback that influenced our decisions.

## Information provided

This category captures the depth and variety of information that DFA activities offer teachers about student learning and comprehension. Teachers looked at the information provided by the activity for insights about their class or a group of students. “In my view, if I need to teach the topic again because of students' mistakes, it suggests that the tasks are fitting for assessment activity” (Teacher A, answer to Question 3). Capturing nuanced details specific to individual students. “The activity gives me an in-depth idea of the student's level, especially after completing all the tasks in the activity” (Teacher N, Q 3).

This activity allows teachers to obtain information on the correctness or errors in students' responses. “As a teacher, I will have the knowledge of who has answered correctly and who has not, and I will also be aware of how often certain answers are given. This will enable me to effectively deal with the outcomes” (Teacher B, Q 12). Analysis of the varied methods employed by students in their work methods. “It offers the opportunity to understand all the misconceptions” (Teacher L, Q 8).
## Type of task

Teachers assess the relevance of DFA activities based on their alignment with curriculum goals, their challenge level, and the variety they offer. The activity matches and checks the curriculum objectives in every aspect. "The activity effectively checks a student's understanding of whether two segments are perpendicular, not only the segments themselves but also their extensions. It assesses comprehension across various levels: basic understanding, knowledge, identification skills, and application proficiency" (Teacher L, Q 3). Perceived difficulty levels of the assigned tasks. "The activity is suited as it caters to both advanced and less proficient students" (Teacher L, Q 3). Diversity and range of tasks. "The activity includes a variety of tasks designed to cater to and challenge the different skill sets of the students" (Teacher D, Q 3). Appropriateness of tasks for specific student groups. "In this task, students are required to provide three examples. However, from my experience, asking for ten examples is challenging. It is probable that only a few students would be able to present ten examples, but those who do show an ability to generalize" (Teacher D, Q 8).

## Student interaction

This category reflects how DFA activities facilitate student engagement through interactive and thoughtful task design. Tasks that enable students to modify the given mathematical context while constructing an example. "The second task allows the student also to build and vary the types of triangles he chooses" (Teacher A, Q 3). Preferences and considerations surrounding the number of prompted examples within a given task. "When a student submits numerous examples, their response becomes clearer and more comprehensible" (Teacher R, Q 8). Tasks for the potential to foster the development of critical thinking skills. "Different examples develop students thinking, so we will have an assessment tool that also enables learning" (Teacher R, Q 6).

## Summary and discussion

The categorization of teachers' responses into three primary categories provides an understanding among mathematics teachers of the multifaceted considerations involved in effectively leveraging formative assessment tools, reflecting a deep understanding of the multifaceted considerations required to effectively leverage these tools in education.

Exploring the **information provided** by the assessment task category, teachers acknowledged the importance of designing formative assessments for all students in the classroom or student groups, recognizing the diverse learning needs inherent in different educational stages. The focus on individual student characteristics, especially technological proficiency, underscores commitment to personalized approaches in formative assessment. Additionally, the acknowledgment of addressing and correcting misconceptions highlights the role of formative assessment in fostering accurate understanding and facilitating future teaching and learning.

Within the **type of task** category, teachers articulate their perspectives on the design and structure of formative assessment tasks. Emphasis on task structure indicates an awareness of how instructional design influences student engagement. Awareness of developing students' thinking skills reflects an approach that extends beyond measurement to actively cultivate critical thinking abilities. Furthermore, the acknowledgment of the challenge in balancing difficulty levels underscores the commitment to providing tasks that appropriately challenge students while ensuring accessibility.
The incorporation of a variety of tasks and the consideration of task suitability for specific student populations reveals the aim of creating diverse and inclusive learning experiences.

The **student interaction** category shows that teachers recognize the value of providing flexibility in both the number of examples and the mathematical context that can be altered to fit the interest or level of difficulty preferred by the student. Consideration of the number of examples indicates a thoughtful approach to balancing the assessment's depth and breadth.

These insights reflect an understanding of formative assessment in line with the principles discussed by Black and William (1998) and Heritage (2007).

In general, these categories contribute valuable insights into the ongoing discourse on best practices in formative assessment, particularly in the context of digital tools and technology integration. These insights reveal a holistic perspective on formative assessment practices. Some of the teachers also showed awareness of the importance of receiving information about the student beyond his mistakes, but also of the work method, which is aligned with the capacities of the STEP platform (Olsher et al., 2016).

In addition to the identified categories, it is important to explore teachers' attitudes towards students' automatic feedback and their integration into formative assessments. Teachers did not refer to this type of feedback in the form of personal reports. This observation is significant considering the established role of timely feedback in formative assessment (Hattie & Timperley, 2007). The absence of a focus on teachers' responses may indicate an area for further exploration and emphasis on teacher training. Additional aspects that did not receive attention in this study included the use of technology to facilitate collaborative work among students, including pairings and peer learning experiences. Understanding how teachers harness digital tools to foster collaborative learning environments and whether they endorse peer-driven feedback processes can offer additional insight into digitally enhanced formative assessment practices. Furthermore, the observation that some teachers showed a tendency to focus primarily on identifying where students went wrong underscores a critical area for development in teacher training. Recognizing and addressing the underlying thought processes behind both correct and incorrect student responses can lead to more nuanced teaching strategies that better support student learning.

## References

* Ayoob, H., & Olsher, S. (2023, July). Curriculum-aligned digital formative assessment for elementary school geometry. In Thirteenth Congress of the European Society for Research in Mathematics Education (CERME13) (No. 4). Alfréd Rényi Institute of Mathematics; ERME.

* Black, P., & William, D. (1998). Inside the black box: Raising standards through classroom assessment. *Phi Delta Kappan*, 80(2), 139–148.

* Black, P., Harrison, C., Marshall, L. B. & William, D. (2004). Working inside the black box: assessment for learning in the classroom. *Phi Delta Kappan*, 86(1), 8–21.
https://doi.org/10.1177/003172170408600105

* Black, P., & William, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability (formerly: Journal of personnel evaluation in education), 21, 5-31. https://doi-org.ezproxy.haifa.ac.il/10.1007/s11092-008-9068-5
Brown, G. T. (2003, November). *Teachers' instructional conceptions: Assessment's relationship to learning, teaching, curriculum, and teacher efficacy*. At a joint conference of the Australian and New Zealand Associations for Research in Education (AARE/NZARE), Auckland (Vol. 28).

Butler, D., Jackiw, N., Laborde, J. M., Lagrange, J. B., & Yerushalmy, M. (2010). Design for transformative practices. In C. Hoyles & J.-B. Lagrange (Eds.), *Mathematics Education and Technology-Rethinking the Terrain* (pp. 425–437). Springer.

Hattie, J., & Timperley, H. (2007). The power of feedback. *Review of educational research, 77*(1), 81-112. https://doi-org.ezproxy.haifa.ac.il/10.3102/003465430298487

Heritage, M. (2007). Formative assessment: What do teachers need to know and do?. *Phi Delta Kappan, 89*(2), 140-145. https://doi-org.ezproxy.haifa.ac.il/10.1177/003172170708900210

Millar, R. (2017). Using assessment materials to stimulate improvements in teaching and learning. In K. Hahl, K. Juuti, J. Lampiselka, A. Uitto, & J. Lavonen (Eds.), *Cognitive and Affective Aspects in Science Education Research* (pp. 31-40). Springer. https://doi.org/10.1007/978-3-319-58685-4_3

Leung, A. (2008). Dragging in a dynamic geometry environment through the lens of variation. *International Journal of Computers for Mathematical Learning, 13*, 135-157.

Olsher, S., Yerushalmy, M., & Chazan, D. (2016). How might the use of technology in formative assessment support changes in mathematics teaching? *For the Learning of Mathematics, 36*(3), 11–18. http://www.jstor.org/stable/44382716

Olsher, S. (2022). Te(a)Ching to collaborate: Automatic assessment-based grouping recommendations and implications for teaching. *Proceedings of the 15th International Conference on Technology in Mathematics Teaching (ICTMT 15)*, (pp. 214–223). Danish School of Education, Aarhus University. https://doi.org/10.7146/aul.452

Stacey, K., & William, D. (2013). Technology and assessment in mathematics. In M. A. Clements, A. Bishop, C. Keitel, J. Kilpatrick, F. Leung (Eds.) *Third International Handbook of Mathematics Education* (Vol.27, pp. 721–751). Springer. https://doi.org/10.1007/978-1-4614-4684-2_23

Webb, M., & Jones, J. (2009). Exploring tensions in developing assessment for learning. Assessment in Education: *Principles, Policy & Practice, 16*(2), 165-184. https://doi.org/10.1080/09695940903075925
# Contextuality of Application Tasks in Large-Scale Summative Assessments at Lower Competence Levels for Lower Secondary Education

Sven Basendowski<fnref n="1" /> and Gilbert Greefrath<fnref n="2" />

This article investigates the item pool of measurements of mathematical competencies of students at or below the lowest proficiency level through a qualitative analysis of Germany's 2018 IQB-Trends in Students Achievement. Contrary to conventional views, this article emphasizes a nuanced understanding of the difficulty of application tasks in large-scale assessments, with a focus on contextualization. By introducing a novel task pool designed for students with special educational needs, the study addresses previous limitations in accurately assessing their performance. Qualitative analysis reveals improved authenticity and relevance in the new tasks, particularly in private contexts. The findings highlight the importance of refining authenticity and relevance criteria for application tasks at lower proficiency levels, and provide valuable insights for inclusive education contexts.

Keywords: *Applications*, *test items*, *authenticity*, *relevance*, *special education*.

## Introduction

The results of large-scale summative assessments (LSAs) in mathematics are crucial for the further development of the education system. In recent years, there has been research-based debate about whether and how LSAs can accurately measure the mathematical competencies of students at or below the lowest competency level, as found in assessments such as the Organization for Economic Cooperation and Development's (OECD) Programme for International Student Assessment (PISA). This article presents the results of a qualitative analysis on the contextualization of application tasks in the lower competence range from a German national LSA, the 2018 Trends in Student Achievement, conducted by the Institute for Quality Development in Education (IQB). Contrary to the often-discussed general assessment in literature (Knoche & Lind, 2004), which suggests that application tasks are inherently more difficult than the same mathematical tasks without an application reference, the analysis by Mahler et al. (2020) concludes that the contextualization of application tasks must be considered in a more differentiated way. This finding is crucial for the further development of application tasks in LSAs for an inclusive education system aligning with the principles of the United Nations Convention on the Rights of Persons with Disabilities (UN CRPD). Application-related tasks in LSAs serve an educational goal in an inclusive education system. For this purpose, we analysed characteristic contextual features within the current 2018 IQB-Trends' task pool in Germany.

## Relevance of the application reference in LSA

Application reference is of particular importance in many LSAs. International LSAs focus on real-world references when assessing mathematical competencies in adolescence and adulthood. In the

<footnote n="1">¹University of Rostock, Germany; sven.basendowski@uni-rostock.de</footnote>

<footnote n="2">²University of Münster, Germany; greefrath@uni-muenster.de</footnote>
PISA study, mathematical competencies are assessed "in authentic application situations" (Baumert et al., 2001, p. 19). This applies equally to the PISA framework for 2022, which aims to emphasize the relevance of mathematics for students and continues to set tasks in authentic contexts. The application reference is a fundamental feature of tasks in PISA studies and other internationally known LSAs, representing a central educational goal for all students. However, there are indications that application-related tasks are empirically more difficult than comparable inner-mathematical tasks Knoche & Lind, 2004). Contrary to the relevance of application tasks as an educational goal for all students, there is a tendency in Germany that the lower the level of completion of an educational program at lower secondary level, the less often modelling tasks are set (Neubrand, 2007).

## Operationalization of the application reference in LSAs

The extent to which the applied tasks vary for different target groups in LSAs has not been analyzed to date. This applies, for example, to PISA tasks, which can be divided into contexts relevant to young people according to different areas of life (OECD, 2023). A distinction is made between contexts that address a personal, professional, social and scientific domain (Reinhold et al., 2019). In addition to the range of relevant contexts to be considered, the LSA frameworks emphasize the authenticity of the test items as a requirement (OECD, 2001). Authenticity refers to an extra-mathematical context that needs to be addressed in the situation using mathematical means. The extra-mathematical context should be authentic and not just constructed for the particular mathematical task. Thus, the use of mathematics in this situation should not be limited to mathematics lessons. Authenticity of tasks is included in various classification schemes and descriptions of modelling tasks (e.g. Maaß, 2010).

Authenticity can be operationalized in terms of different dimensions such as the situation, the question or the information and tools provided (Palm, 2007). When multiple dimensions of a task are authentic, students are more likely to make the necessary real-world considerations for the solution (Palm, 2008). Palm (2008) therefore considers a situation to be authentic if it represents a real task situation and if important aspects of that situation are simulated to an appropriate degree. An authentic task is therefore credible to the learner and realistic in terms of the environment (Palm, 2007). A focus on the authenticity of key dimensions of tasks, such as situation, question and methods, is useful (Turner et al., 2022) and will be used in this article.

## Tasks in LSAs at the lower competence levels of the 2018 IQB-Trends

In Germany, there is an institutionalized category of special educational needs for students with learning difficulties and disadvantages (SEN Idd students) as distinct from SEN for students with disabilities (OECD, 2007). The special educational needs of students are "considered to arise primarily from problems in the interaction between the student and the educational context" (OECD, 2007, p. 20) and are manifested by a general and persistent failure to achieve school standards, such as in mathematics. As a result, students with learning difficulties perform in the lower proficiency levels of PISA: 61.5% below proficiency level I, 27.9% at proficiency level I and 10.6% above proficiency level I (Müller et al., 2017). Due to this institutionalization in the German education system, it is possible to specifically examine a group of students that perform within the lower competence range among application tasks.
As part of the 2018 IQB-Trends, a target group-specific task pool for SEN Idd students was developed (Mahler et al., 2020). The IQB-Trends examines a representative sample of all students in grades 4 and 9 without SEN every six years. In 2018, the representative sample also included SEN Idd students and was intended to provide information about the extent to which learners in grade 9 have achieved the competence expectations formulated in the national educational standards (Stanat et al., 2019). Competence level Ib is associated with the minimum requirements to gain the lowest formal secondary school diploma (Kölm & Mahler, 2019).

The new target group-specific development in the 2018 IQB-Trends was deemed necessary, because the task pools and competence structure models used nationally (e.g., Südkamp et al., 2015) and internationally (Müller et al., 2017) had not proven to be empirically suitable for adequately recording the performance of SEN Idd students. In the national studies cited, items from pools for the 4th or 6th grade level were used without taking contextual features into account. As a result, there was no task pool or competence structure model that could measure the performance of SEN Idd students in grade 9 with precision (Müller et al., 2017). The new development was able to empirically demonstrate a satisfactorily valid instrument for comparisons between all learners in secondary I programs except for the highest level (Mahler et al., 2020). In doing so, it focused conceptually on the lower competence areas. Further adaptations focused on processing times and the reduction of language and context barriers in the test material, i.e. by gaining a high degree of authenticity (Mahler et al., 2020). The results suggest that the general valuation that application tasks are empirically more difficult per se needs to be differentiated. The closeness to the real world and the authenticity of the context plays a decisive role here (Mahler et al., 2020). However, no analyses have been conducted to determine how these adaptations were reflected in the final task pool or how they influenced performance. Nevertheless, the results are relevant because modelling competencies are central to the subject of mathematics (KMK, 2004), and this area of competence in LSAs must be ensured for each student in an inclusive education system (UN CRPD).

**Research question**

The state of the art has highlighted the conceptual, didactic and participatory relevance of application-
related items in LSAs in mathematics. The new development of an LSA in the educational trend 2018
provides important indications that, considering specific adaptations to authenticity and the relevance
of the application reference, it is possible to develop a valid and reliable task pool that ensures the
recording of the performance of young people from different educational pathways in the lower
competence area. Given the documented test quality of the newly designed task pool for young people
from almost all lower secondary programmes except the highest level, it can be assumed that the
adaptations of the newly designed task pool can be applied to any student performing at the lower
levels of competence in the IQB-Trends studies. Similarly, there is no consensus in the research on
whether, due to their complexity, application-related tasks in mathematical LSAs should be
eliminated at the lower proficiency levels for students in lower secondary education.

The aim of this analysis is therefore to characterize the features of context embedded in a task pool
that can demonstrably and validly measure performance in application tasks at the lowest competence
levels. For this purpose, the items of the newly designed task pool from the 2018 IQB-Trends are
compared with those of the previous task pool in terms of the authenticity and relevance of the
application reference for each lower competence level. This is of interest to be able to provide
indications as to which features of the context are suitable for the improvement of future task pools
for LSAs. The research question arises:

To what extent do empirically suitable LSA items with an application reference at the lower competence levels differ from previously used LSA items in terms of authenticity and relevance of the application reference per competence level?

Study Design and Material

A structured qualitative content analysis is recommended for the intended qualitative analysis of the
selected didactic criteria and characteristics of context embedded in application tasks (Gläser-Zikuda
et al., 2020). For the present study, the standard procedure based on Mayring (2010) was adapted.
The deductive category system was scrutinized and modified after the first run of analysis of the
material. For this first run through the material, approximately half of the items were randomly
selected and prepared for coding with the initial deductive category system in MAXQDA. Coding
was carried out independently by two coders.

In this analysis of the 2018 IQB-Trends, the entire task pool, including information on item difficulty and assigned competence level, was provided by the IQB (Basendowski & Greefrath, 2024).
However, this data can only be used on the condition that no items are published.

The subject of this structured qualitative content analysis is the set of test items used in the 2018 IQB-
Trends (Stanat et al., 2019), which includes both the items and the coding guide. The items of the
2018 IQB-Trends were developed by teachers under the guidance of the IQB. Some items were reused
from the 2012 test, while others were supplemented with newly designed items specifically for SEN
Idd students. The redesign was prompted by the insufficient measurement accuracy for SEN Idd
students identified in previous item pools, as explained above. Before being used in the 2018 IQB-
Trends, the items were tested with several hundred SEN Idd students in both general and special
schools and then selected. In the 2018 IQB-Trends itself, the 188 newly developed items were used
as part of the task pool of a total of 521 items for all students, regardless of the level of the secondary
I programme they attended (Mahler, Schipolowski & Weirich, 2019).

In accordance with the given research interest, only items with an application reference were selected.
As the redesigned item pool is exclusively at levels Ia, Ib and II, only items at these levels were
selected. The total of 128 items selected consists of 88 items from the redesigned pool and 40 items
from the existing pool.

Revision of the category system and final material pass including intercoder reliability check

Gläser-Zikuda et al. (2020) identify testing through intercoder agreement as a common quality
criterion for qualitative content analyses and specify a target value of Kappa = 0.70 as a sufficient
indicator. The kappa values in the reported study for each supercategory in the second run are between
0.69 and 0.91, determined by MAXQDA. As a result, after the second run there is an outcome that
does not necessitate any further revisions of the deductive-inductive category system. All three
authenticity categories, i.e. question, situation, and method (Turner et al., 2022), could be coded in
"at least simulated authentic" (= credible and realistic), "simulated" (= untrustworthy) and "not
assessable". The relevance categories (private, social, professional, scientific) were coded once per item. The final deductive-inductive category system can be found in Basendowski & Greefrath (2024). For an exemplary explanation of the authenticity and relevance criteria, see Table 1 and Figure 1: "Indicate how many degrees C the measured body temperature in the figure is."

<table><caption>**Table 1: exemplary coding**</caption><tr><td rowspan="3">Authenticity</td><td>question</td><td>at least simulated authentic</td><td>the problem of measuring a body temperature by reading a clinical thermometer is not implausible</td></tr><tr><td>situation</td><td>simulated</td><td>reading of a thermometer is not credible without a person being present; in a problem situation, the measured body temperature may have very different consequences</td></tr><tr><td>tools</td><td>simulated</td><td>the clinical thermometer is virtually available; in a problem situation</td></tr></table>

<figure><img src="image_22.png" /><figcaption>**Figure 1: Example for previous concept pool items**</figcaption></figure>

**Teilaufgabe 1**

Gib an, wie viel °C die gemessene Körpertemperatur in der Abbildung beträgt.

_______ °C

(source: https://www.iqb.hu-berlin.de/vera/aufgaben/ma1/)

**Results**

A comparison between the previous concept and the new concept reveals the following striking differences: The authenticity of the tasks in both pools is generally low. However, there appears to be a trend that the newly designed tasks in the situation category are more consistently simulated authentic than the tasks used in the previous concept. No differences were found in the other authenticity categories. There are major differences in terms of the relevance of the tasks. In both groups, the relevance is essentially limited to scientific contexts, i.e. exemplary questions are described that are not relevant to the students' private, social and future professional lives. However, there are significantly more items among the newly developed tasks that can be classified as relevant in a private context (see Table 2).

<table><caption>**Table 2: Number of items in the previous concept and the new concept**</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="2">Competence Level Ia</th><th colspan="2">Competence Level Ib</th><th colspan="2">Competence Level II</th></tr><tr><th>Previous Concept</th><th>New Concept SEN Idd</th><th>Previous Concept</th><th>New Concept SEN Idd</th><th>Previous Concept</th><th>New Concept SEN Idd</th></tr></thead><tbody><tr><td rowspan="3">Authenticity: question / situation / tools</td><td>not assessable</td><td>11 / 0 / 13</td><td>36 / 0 / 44</td><td>0 / 0 / 0</td><td>33 / 0 / 31</td><td>25 / 0 / 31</td><td>15 / 0 / 14</td></tr><tr><td>simulated</td><td>11 / 14 / 8</td><td>32 / 41 / 26</td><td>2 / 0 / 2</td><td>23 / 26 / 22</td><td>21 / 31 / 19</td><td>6 / 22 / 8</td></tr><tr><td>at least simulated authentic</td><td>1 / 7 / 1</td><td>8 / 35 / 4</td><td>0 / 2 / 0</td><td>15 / 35 / 18</td><td>9 / 22 / 6</td><td>6 / 5 / 5</td></tr></tbody></table>
<table><tr><td rowspan="4">relevance</td><td>private</td><td>0</td><td>18</td><td>0</td><td>21</td><td>0</td><td>1</td></tr><tr><td>social</td><td>0</td><td>2</td><td>0</td><td>4</td><td>0</td><td>1</td></tr><tr><td>professional</td><td>0</td><td>2</td><td>0</td><td>2</td><td>0</td><td>0</td></tr><tr><td>scientific</td><td>22</td><td>54</td><td>2</td><td>47</td><td>50</td><td>25</td></tr></table>

Reading note for lines 2-4: The first number refers to the authenticity category of the question, the second to the authenticity category of the situation and the third to the authenticity category of the tool.

## Discussion

The study of Mahler et al. (2020) has revealed that it is indeed possible to develop suitable and valid application tasks at the lowest competence levels. The study indicated that the newly designed task pool allows for a valid assessment of the performance of adolescents at almost all levels of secondary I programs, including the one for SEN Idd students.

It was found that there are clear differences between the concepts. This suggests that the new conceptualisation (Mahler et al., 2020) can provide new insights for the development of LSA items with practical application relevance at lower proficiency levels. In comparison, the new conceptualisation of items for SEN students shows that the novel characteristics 'relevance to everyday life' and 'authenticity of question, situation and tools' of application-related items at lower proficiency levels Ia and Ib were not as prominent in the previous conceptualization of IQB-Trends. It is generally important to enhance the credible problem character of the situation to make it more authentic – even though the new item test pool deviate from the degree of authenticity and relevance intended by the theoretical concept (Mahler et al., 2020).

The absence of authentic contexts on all levels (question, situation, and tools) needs to be discussed in relation to the PISA framework as well (OECD, 2023). There may be a need for improvement. If the demand of the tasks was to increase through authentic and relevant contexts, differences could potentially be identified by comparing the lower competence level with those above. However, in this investigation, no significant differences could be found. This might imply that authentic and relevant contexts can be constructed across all competence levels. Nevertheless, there are indications that authenticity is somewhat less frequent at the lowest competence level, Ia. Therefore, these concerns cannot be entirely dismissed. Other limitations concern items at higher levels of proficiency and more detailed categories.

## References

* Basendowski, S., & Greefrath, G. (2024). Anwendungsbezug in Mathematik-Large-Scale-Assessments im Bildungsmonitoring für den Sekundarstufe I-Bildungsgang des sonderpädagogischen Schwerpunkts Lernen. Journal für Mathematik-Didaktik. https://doi.org/10.1007/s13138-024-00230-y

* Baumert, J., Klieme, E., Neubrand, M., Prenzel, M., Schiefele, U., Schneider, W., Stanat, P., Tillmann, K.-J., & Weiß, M. (Eds.). (2001). PISA 2000. Basiskompetenzen von Schülerinnen und Schülern im internationalen Vergleich. Leske + Budrich. https://doi.org/10.1007/978-3-322-83412-6
Gläser-Zikuda, M., Hagenauer, G., & Stephan, M. (2020). The Potential of Qualitative Content Analysis for Empirical Educational Research. Forum: Qualitative Social Research, 21(1), 17. https://doi.org/10.17169/fqs-21.1.3443

KMK (Ed.). (2004). Bildungsstandards im Fach Mathematik für den Mittleren Schulabschluss. Beschluss vom 4.12.2003. Luchterhand.

Knoche, N., & Lind, D. (2004). Bedingungsanalysen mathematischer Leistung: Leistungen in den anderen Domänen, Interesse, Selbstkonzept und Computernutzung. In M. Neubrand (Ed.), Mathematische Kompetenzen von Schülerinnen und Schülern in Deutschland: Vertiefende Analysen im Rahmen von PISA 2000 (pp. 205–226). VS Verlag für Sozialwissenschaften. https://doi.org/10.1007/978-3-322-80661-1_11

Kölm, J., & Mahler, N. (2019). Kompetenzstufenbesetzungen im Ländervergleich. In P. Stanat, S. Schipolowski, N. Mahler, S. Weirich, & S. Henschel (Eds.), IQB-Bildungstrend 2018. Mathematische und naturwissenschaftliche Kompetenzen am Ende der Sekundarstufe I im zweiten Ländervergleich (pp. 157–168). Waxmann.

Lehmann, R. H., & Hoffmann, E. (Eds.). (2009). BELLA: Berliner Erhebung arbeitsrelevanter Basiskompetenzen von Schülerinnen und Schülern mit Förderbedarf ‘Lernen’. Waxmann.

Maaß, K. (2010). Classification Scheme for Modelling Tasks. Journal Für Mathematik-Didaktik, 31(2), 285–311. https://doi.org/10.1007/s13138-010-0010-2

Mahler, N., Kölm, J., & Werner, B. (2020). Entwicklung von Mathematiktestaufgaben für Schüler*innen mit einem sonderpädagogischen Förderbedarf im Lernen – Konzeption und empirische Ergebnisse. In C. Gresch, P. Kuhl, M. Grosche, C. Sälzer, & P. Stanat (Eds.), Schüler*innen mit sonderpädagogischem Förderbedarf in Schulleistungserhebungen (pp. 109–146). Springer Fachmedien Wiesbaden. https://doi.org/10.1007/978-3-658-27608-9_5

Mahler, N., Schipolowski, S., & Weirich, S. (2019). Anlage, Durchführung und Auswertung des IQB-Bildungstrends 2018. In P. Stanat, S. Schipolowski, N. Mahler, S. Weirich, & S. Henschel (Eds.), IQB-Bildungstrend 2018. Mathematische und naturwissenschaftliche Kompetenzen am Ende der Sekundarstufe I im zweiten Ländervergleich (pp. 99–124). Waxmann.

Mahler, N., Weirich, S., & Becker, B. (2019). Auswertung, Trendschätzung und Ergebnisdarstellung. In P. Stanat, S. Schipolowski, N. Mahler, S. Weirich, & S. Henschel (Eds.), IQB-Bildungstrend 2018. Mathematische und naturwissenschaftliche Kompetenzen am Ende der Sekundarstufe I im zweiten Ländervergleich (pp. 125–130). Waxmann.

Mayring, P. (2010). Qualitative Inhaltsanalyse. In G. Mey & K. Mruck (Eds.), Handbuch Qualitative Forschung in der Psychologie (pp. 601–613). VS Verlag für Sozialwissenschaften. https://doi.org/10.1007/978-3-531-92052-8_42

Müller, K., Prenzel, M., Sälzer, C., Mang, J., Heine, J.-H., & Gebhardt, M. (2017). Wie schneiden Schülerinnen und Schüler an Sonder- und Förderschulen bei PISA ab? Analysen aus der PISA 2012-Zusatzerhebung zu Jugendlichen mit sonderpädagogischem Förderbedarf. Unterrichtswissenschaft, 2, 194–211. https://doi.org/10.3262/UW1702175
Neubrand, M. (2007). Professionelles Wissen von Mathematik-Lehrerinnen und Lehrern: Konzepte und Ergebnisse aus der PISA- und der COACTIV-Studie und Konsequenzen für die Lehrerausbildung. In: F. Kostrzewa (Ed.), Lehrerbildung im Diskurs (pp. 53–72). gata-Verlag.

OECD (Ed.). (2001). Knowledge and Skills for Life. First Results from the OECD Programme for international Student Assessment. OECD.

OECD. (2007). Students with Disabilities, Learning Difficulties and Disadvantages: Policies, Statistics and Indicators. OECD. https://doi.org/10.1787/9789264027619-en

OECD (2023). PISA 2022 Assessment and Analytical Framework. OECD. https://doi.org/10.1787/dfe0bf9c-en

Palm, T. (2007). Features and Impact of the Authenticity of Applied Mathematical School Tasks. In W. Blum, P. L. Galbraith, H.-W. Henn, & M. Niss (Eds.), Modelling and Applications in Mathematics Education. The 14th ICMI Study (Vol. 10, pp. 201–208). Springer US. https://doi.org/10.1007/978-0-387-29822-1_20

Palm, T. (2008). Impact of authenticity on sense making in word problem solving. Educational Studies in Mathematics, 67(1), 37–58. https://doi.org/10.1007/s10649-007-9083-3

Reinhold, F., Reiss, K., Diedrich, J., Hofer, S. I., & Heinze, A. (2019). Mathematische Kompetenz in PISA 2018 – aktueller Stand und Entwicklung. In K. Reiss, M. Weis, E. Klieme, & O. Köller (Eds.), PISA 2018: Grundbildung im internationalen Vergleich (pp. 187–209). Waxmann. https://doi.org/10.31244/9783830991007

Stanat, P., Schipolowski, S., Mahler, N., Weirich, S., & Henschel, S. (Eds.). (2019). IQB-Bildungstrend 2018. Mathematische und naturwissenschaftliche Kompetenzen am Ende der Sekundarstufe I im zweiten Ländervergleich. Waxmann.

Südkamp, A., Pohl, S., Hardt, K., Jordan, A.-K., & Duchhardt, C. (2015). Kompetenzmessung in den Bereichen Lesen und Mathematik bei Schülerinnen und Schülern mit sonderpädagogischem Förderbedarf. In P. Kuhl, P. Stanat, B. Lütje-Klose, C. Gresch, H. A. Pant, & M. Prenzel (Eds.), Inklusion von Schülerinnen und Schülern mit sonderpädagogischem Förderbedarf in Schulleistungserhebungen (pp. 243–272). Springer.

Turner, E. E., Bennett, A. B., Granillo, M., Ponnuru, N., Roth Mcduffie, A., Foote, M. Q., Aguirre, J. M., & McVicar, E. (2022). Authenticity of elementary teacher designed and implemented mathematical modeling tasks. Mathematical Thinking and Learning, 26(1), 1–24. https://doi.org/10.1080/10986065.2022.2028225