to a misalignment between their mathematical competence and the assessment's practice domain (Drijvers et al., 2016).

## References

Clark-Wilson, A. (2010). Emergent pedagogies and the changing role of the teacher in the TI-Nspire Navigator-networked mathematics classroom. *ZDM–The International Journal of Mathematics Education, 42*(7), 747–761.

Drijvers, P., Ball, L., Barzel, B., Heid, M. K., Cao, Y., & Maschietto, M. (2016). *Uses of Technology in Lower Secondary Mathematics Education: A Concise Topical Survey*. ICME-13 Topical Surveys, 1–34. http://doi.org/10.1007/978-3-319-33666-4

Grice, H. P. (1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), *Speech acts* (pp. 51–58). Brill. https://doi.org/10.1163/9789004368811_003

Hall, J. (2013). MathsBot.com. https://mathsbot.com/

Kho, T. H., Yeo, S. M., & Fan, L. (2014). Model method in Singapore primary mathematics textbooks. In K. Jones, C. Bokhove, G. Howson, & L. Fan (Eds.), *Proceedings of the International Conference on Mathematics Textbook Research and Development* (pp. 275–282). University of Southampton.

Kintsch, W., & Greeno, J. G. (1985). Understanding and solving arithmetic word problems. *Psychological Review, 92*(1), 109–129. https://doi.org/10.1037/0033-295X.92.1.109

Mayer, R. E. (2002). Multimedia learning. *Psychology of Learning and Motivation, 41*, 85–139. https://doi.org/10.1016/S0079-7421(02)80005-6

Mayer, R. E., & Wittrock, M. C. (1996). Problem-solving transfer. In D. Berliner and R. Calfee (Eds.), *Handbook of educational psychology* (pp. 45–61). Macmillan.

Nesher, P., Greeno, J. G., & Riley, M. S. (1982). The development of semantic categories for addition and subtraction. *Educational Studies in Mathematics, 13*(4), 373–394. https://doi.org/10.1007/BF00366618

Rittle-Johnson, B., Star, J. R., & Durkin, K. (2017). The power of comparison in mathematics instruction: Experimental evidence from classrooms. In D. C. Geary, D. B. Berch, and K. M. Koepke (Eds.), *Acquisition of complex arithmetic skills and higher-order mathematics concepts* (pp. 273–295). Elsevier. https://doi.org/10.1016/B978-0-12-805086-6.00012-6

Schmidt, J., Gröller, M. E., & Bruckner, S. (2013). VAICo: Visual Analysis for Image Comparison. *IEEE Transactions on Visualization and Computer Graphics, 19*(12), 2090–2099. https://doi.org/10.1109/TVCG.2013.213

Stacey, K., & William, D. (2012). Technology and assessment in mathematics. In M. A. Clements, A. J. Bishop, C. Keitel, J. Kilpatrick & F. K. S. Leung (Eds.), *Third international handbook of mathematics education* (pp. 721–751). Springer. https://doi.org/10.1007/978-1-4614-4684-2_23
# Conception of two informative tutoring feedback strategies for mathematical tasks with STACK

Farhad Razeghpour

Ruhr-University Bochum, Germany; farhad.razeghpour@ruhr-uni-bochum.de

The increasing prevalence of computer-based teaching and learning opportunities in recent years has led to the emergence of novel approaches for providing informative tutoring feedback (ITF). For mathematical tasks, the tool STACK represents a promising avenue for exploration. STACK enables the automated evaluation of open mathematical tasks and the provision of immediate feedback for learners. This paper will therefore investigate the potential of STACK in designing ITF-strategies for mathematical tasks. In light of this question, two distinct strategies are presented, which differ in the case of incorrect answers with unclear causes. While in one strategy learners are then provided with solution-specific hints for processing the task, in the other strategy they enter a task loop in which they must work out these hints on their own. Finally, the strategies are compared in terms of their characteristics and possible effects on students' learning processes.

Keywords: Computer assisted learning, education, informative tutoring feedback, mathematics.

## Introduction

Feedback plays a pivotal role in mathematics education, exerting a profound influence on enhanced academic performance, with a considerable impact on student learning (Hattie, 2010). When designing feedback, it is essential to determine whether it is incorporated into a formative or summative assessment. Summative assessments are provided at the end of the learning process and serve to evaluate the learning outcomes. In contrast, formative assessments are provided during the learning process with the intention of having a positive impact on it (Pals et al., 2023). This paper focuses on formative assessments as the objective is to design feedback that supports learning processes.

In recent years, computer-based teaching and learning tools have offered new ways of providing informative tutoring feedback (e.g. Erickson et al., 2020). This refers to feedback for digital learning environments that provide learners with solution- and error-specific hints rather than concrete solutions, enabling them to correct their incorrect answers independently. In the field of mathematics teaching, the STACK tool represents a promising approach for this. STACK enables the automatic assessment of open mathematical tasks and provides learners with immediate feedback (Knaut et al., 2022).

This theoretical paper aims to investigate the use of STACK for the realization of effective ITF-strategies for mathematical tasks. The theoretical background highlights the central role of STACK in automatizing the assessment process for mathematical tasks. With its integrated computer algebra system (CAS), STACK enables the evaluation of students' answers to various mathematical tasks. Furthermore, this section describes the relevant characteristics of ITF-strategies. Subsequently, the research question of how STACK could be used to design ITF-strategies for mathematical tasks is addressed. To answer this research question, two different implementations of ITF-strategies are presented and explained. The effectiveness of both strategies is discussed on a theoretical basis,
highlighting their potential impact on students' learning processes. The paper concludes with a
summary and an outlook on further related issues.

# Theoretical background

## STACK

Computer-based tools can be utilized to automatically evaluate tasks with even free input fields and to provide learners with feedback concerning their achievement. For mathematical tasks, this can be accomplished with STACK (System for Teaching and Assessment using a Computer algebra Kernel). Due to the implemented CAS, the digital evaluation of tasks with any number of correct solutions is possible (Alarfaj & Sangwin, 2022). Furthermore, emerging numerical values can be randomized. As a result, learners are offered numerous opportunities to practise by creating a single task. Meanwhile, STACK is available as a free plugin for Moodle, allowing the creation and use of STACK tasks in this learning platform.

For the digital evaluation and provision of feedback, a so-called response tree must be constructed in advance for each task (see Figure 1). Within the response tree, the learners' answers are checked for various mathematical characteristics in several nodes (Knaut et al., 2022). Through the underlying CAS, it is possible, for example, to check whether an entered function has certain zeros or extreme points. After a positive or negative result of the check, the path can be stopped or connected to another node for further analysis. In Figure 1, the paths are visualized by green and red lines. On each path, feedback can be stored and points for performance can be added or subtracted. Overall, the construction of a response tree enables the evaluation of various answers and the provision of feedback for learners (Sangwin, 2023).

<figure><img src="image_76.png" /><figcaption>Figure 3: Example of a response tree for a task</figcaption></figure>

# Informative tutoring feedback strategies

Narciss (2008) has classified various forms of feedback in terms of their content. A distinction is made between simple and elaborated forms. While simple forms of feedback only include information about the correctness of a task, elaborated forms include more extensive information. Examples of simple feedback are *knowledge of performance* (KP), which indicates the number or proportion of tasks solved correctly, and *knowledge of result* (KR), which contains a specification of which tasks were solved correctly. An example of an elaborated form is feedback that discusses specific errors and offers explanations for their occurrence, which is assigned to the category knowledge about errors (KM). If solution-specific hints are included in the feedback, it is referred to as knowledge on how to proceed (KH). Solution-specific hints contain information about strategies and necessary intermediate steps to solve the task without presenting a complete solution.

In the case of digital learning tools such as the STACK system previously presented, a feedback strategy can be employed to define which feedback forms are combined and presented to learners. Accordingly, a feedback strategy is a defined plan that determines the structure and presentation of feedback. One approach is to develop ITF-strategies, which combine simple KR-feedback with
elaborated forms such as KH- and KM-feedback (Narciss, 2012). Instead of providing learners with correct answers, the focus is on enabling them to use the feedback to correct their responses. As a result, learners should be allowed to retry tasks immediately after receiving feedback. The opportunity for immediate implementation increases the relevance of the feedback for the learners and leads to a more intensive engagement with it (Tärning et al., 2020). This is intended to facilitate the students' learning processes.

## Research question

In light of the potential for ITF-strategies to support learning processes, it is necessary to consider how these strategies could be implemented in practice with current technologies. While there have been a few studies on the implementation of ITF-strategies in digital learning environments for specific mathematical areas, such as written subtraction (Narciss & Huth, 2004), further research is needed on how ITF-strategies could be implemented in general for mathematical tasks. This research is essential for the wider use of these strategies in mathematics classes and for the investigation of them in experimental studies. The STACK system could provide a possible approach to this need. This paper will therefore investigate the research question of how STACK could be used to design ITF-strategies for mathematical tasks. To answer the research question, two possible approaches using STACK are developed, which both respect the presented characteristics of ITF-strategies. The two strategies are described in detail in the subsequent chapter.

## Conception of ITF-strategies with STACK

Firstly, we consider the two cases in which either a correct solution or an erroneous solution is entered where the cause of the error is recognizable. The design of the two strategies is identical for these two cases. In both strategies, students receive KR-feedback in the event of a correct input. The correct solution can be verified in STACK in two different ways due to the underlying CAS. The answer can be compared with a saved sample solution for algebraic equivalence or tested for relevant properties. These options ensure that every correct answer is recognized, even if it is entered in a different form (e.g. fraction instead of a decimal number) or if there are any number of correct answers. For certain incorrect answers, both strategies offer KM feedback. This necessitates the implementation of appropriate checks in advance within the response trees. Due to the CAS, it is sufficient to test the submitted answer for relevant mathematical characteristics and hence no explicit incorrect solutions need to be stored within the response trees. For example, it is possible to ascertain whether an inserted function has the requisite degree or whether the correct variable has been used. This approach enables the identification of numerous incorrect responses, all attributable to a common set of error sources, within individual nodes. In this way, differentiated KM-feedback can be formulated for several typical errors.

The described common components of both strategies are illustrated in Figure 2 in the form of a diagram for an exemplary integral task. As previously stated, KR-feedback is displayed if the input is correct. In the event of an incorrect answer, several different error-specific hints have been stored where the cause of the error can be deduced reliably enough to provide KM-feedback. For the sake of clarity, not all five KM-feedback cases are listed in full in Figure 2, but only two of them. In the first case shown (see top left in Figure 2), feedback is given if a solution has been entered that arises when the product of two integrals has been calculated instead of the integral of the product. This
scenario utilizes an example to explain why this method is typically not allowed. The second case, depicted in the top right of Figure 2, concerns a scenario where the correct approach of partial integration has been selected, but the terms in the formula have been added instead of subtracted. The formula is shown, with the minus sign highlighted in red.

<figure><img src="image_77.png" /><figcaption>Figure 2: Common feedback components of both strategies in an integral task</figcaption></figure>

Now the challenging question arises of how to deal with certain incorrect entries for which no specific error causes have been created in the response tree. This is a likely scenario, as task creators cannot consider every possible cause of error in advance. In addition, particularly in the case of more complex tasks, a combination of several error causes can result in incorrect answers that cannot be recognized based on the final solution. Nevertheless, it is of the utmost importance to ensure that students receive constructive feedback even in the event of such erroneous entries, in order to enable them to identify their mistakes and rectify their solutions. The two feedback strategies deal with this situation differently.

As part of the first feedback strategy, students receive KH-feedback in the event of an incorrect answer that was not saved in the response tree. The feedback contains solution-specific instructions for processing the task. However, the correct solution is not anticipated, as the students are asked to use the information presented to perform the last step independently. In this way, learners still have the opportunity to correct their answers. We describe this strategy as *summarizing* feedback.

This case is illustrated in Figure 3 for the integral task. In the event that an incorrect response cannot be attributed to an underlying error, we present the formula for partial integration in the KH-feedback. Furthermore, the choice of functions for the two factors and the resulting expression are presented. It is now the learner's task to calculate this new integral expression.

However, the second feedback strategy handles this case differently. In the event of an incorrect answer, for which the cause of the error is not stored in the response tree, the second feedback strategy does not provide any error-specific feedback. Instead, students can enter a task loop in which they
<figure><img src="image_78.png" /><figcaption>Figure 3: KH-component of the summarizing feedback in an integral task</figcaption></figure>

work their way through the entire task in a series of sub-steps. In these task loops, the solution-specific
hints from the summarizing feedback are now reformulated as questions which now should be
answered by the students. They also receive error-specific feedback while working through the sub-
steps. This feedback is usually more extensive than the feedback received when working through the
initial complex task. The reason for this lies in the fact that more precise insights into the causes of
errors can be drawn from the answers to the sub-steps. We refer to this strategy as *guiding* feedback.

In Figure 4, the case in which an incorrect response was entered that cannot be traced back to the cause of its error is now visualized for the guiding feedback. Concerning this situation, a button is displayed in the feedback that leads to the task loop (see Figure 4). The first sub-step relates to the necessary method for calculating the integral. In a drop-down menu, the correct solution (partial integration) among some distractors can be selected. If the answer is incorrect, KM-feedback is given for each answer option, explaining why the selected option is not appropriate or not permitted. If the answer to the first sub-step is correct, KR-feedback appears, which leads to the next sub-step. While the second sub-step still has a closed answer format, the third, fourth and fifth sub-steps employ an open input field format. For these three open input fields, a total of ten error-specific hints are provided if required. These error-specific hints are not explicitly shown in Figure 4 for reasons of clarity. In the fifth sub-step, learners are asked to specify a suitable antiderivative. Once this last sub-step has also been successfully completed, students return to the initial task. Subsequently, learners may re-engage with the task using the sub-solutions they have worked out. Their sub-solutions are displayed so that they can easily access them. The task loop described can be repeated if necessary. The creation of these task loops is not a regularly offered feature of STACK. This option was developed at the Ruhr University Bochum by incorporating an additional JavaScript program (Altieri et al., 2020).

It can be argued that both strategies have the potential to support students' learning processes. Both
strategies contain at least one important elaborated component in addition to the evaluative one
(Kulhavy & Stock, 1989). Moreover, both strategies encourage learners to become active and correct
their answers independently. In both cases, this could lead to a more intensive engagement with the
feedback and thus have a positive effect on the learning process (Tärning et al., 2020).
<figure><img src="image_79.png" /><figcaption>Figure 4: Task loop of the guiding feedback in an integral task</figcaption></figure>

One favor of summarizing feedback lies in the fact that solution-specific cues target correct solution strategies. Therefore it could have a more motivating effect (Fong et al., 2019). In addition, similar to the use of correct solutions (Renkl, 2014), the presentation of solution-specific hints can be assumed valuable. Solution-specific hints draw students' attention to the correct solution procedure, making it easier for them to focus on acquiring new knowledge (Große & Renkl, 2007). Furthermore, its design could contribute to a clearer overview when working on the task and therefore lead to a better understanding.

Nevertheless, it can be argued that the guiding feedback has a more profound effect on the learning process, as the students are required to comprehend and apply the content of the feedback within the task loops to a greater extent. The individual sub-steps that are displayed in the summarizing feedback must be worked out independently in the tasks with guiding feedback. This could have a direct influence on the self-efficacy of the students, as it provides more opportunities for mastery experiences (Bandura, 1997; Ramdass & Zimmerman, 2008). Another advantage that results directly from the task loops is that errors can be traced back more precisely to their causes when working through the sub-steps. In this manner, the feedback can be employed to ascertain the learning status and assist students in closing learning gaps (Hattie & Timperley, 2007).

## Conclusion and outlook

The presented tool STACK is appropriate for the creation of tasks with formative feedback due to its numerous technical possibilities. The response tree in STACK can be used to provide learners with elaborated feedback on their answers. For this purpose, task creators must consider in advance, based on literature or empirical observations, which mistakes learners might make. They can then create response trees with appropriate checks and corresponding KM-feedback. However, the causes of errors can only be anticipated in advance to a certain extent. To also handle incorrect answers that
cannot be traced back to specific errors by defined response trees, two feedback strategies were presented in this paper. In the summarizing feedback, solution-specific hints are presented. Despite this, in the guiding feedback, only error-specific hints are given, while intermediate solutions must be worked out in sub-steps.

The question of whether one of these feedback strategies offers greater support for students' learning processes requires empirical investigation. On the basis of theoretical considerations, reasons could be identified for both feedback strategies explaining their potential support for learning processes. Comparative studies are necessary to explore this further. Certainly, the effectiveness of the feedback strategies also depends on other factors to be investigated, such as the topic or the difficulty of the task and the individual prerequisites of the learners. A final decision between the two strategies may not be required by task creators, as strengths of the two feedback strategies may differ in cognitive, motivational and metacognitive areas.

## References

Alarfaj, M., & Sangwin, C. (2022). Updating STACK Potential Response Trees Based on Separated Concerns. *International Journal of Emerging Technologies in Learning (iJET)*, 17(23), 94–102.
https://doi.org/10.3991/ijet.v17i23.35929

Altieri, M., Horst, J., Kallweit, M., Landenfeld, K., & Persike, M. (2020). *Multi-step procedures in STACK tasks with adaptive flow control*. https://doi.org/10.5281/zenodo.3944786

Bandura, A. (1997). *Self efficacy: The exercise of control*. W. H. Freeman.

Erickson, J. A., Botelho, A. F., McAteer, S., Varatharaj, A., & Heffernan, N. T. (2020). The automated grading of student open responses in mathematics. *Proceedings of the Tenth International Conference on Learning Analytics & Knowledge*, 615–624.
https://doi.org/10.1145/3375462.3375523

Fong, C. J., Patall, E. A., Vasquez, A. C., & Stautberg, S. (2019). A Meta-Analysis of Negative Feedback on Intrinsic Motivation. *Educational Psychology Review*, 31(1), 121–162.
https://doi.org/10.1007/s10648-018-9446-6

Große, C. S., & Renkl, A. (2007). Finding and fixing errors in worked examples: Can this foster learning outcomes? *Learning and Instruction*, 17(6), 612–634.
https://doi.org/10.1016/j.learninstruc.2007.09.008

Hattie, J. (2010). *Visible learning: A synthesis of over 800 meta-analyses relating to achievement*
(Reprinted). Routledge.

Hattie, J., & Timperley, H. (2007). The Power of Feedback. *Review of Educational Research*, 77(1),
81–112. https://doi.org/10.3102/003465430298487

Knaut, J., Altieri, M., Bach, S., Strobl, I., & Dechant, K. (2022). A Theory-Based Approach of Feedback in STACK-Based Moodle Quizzes Taking into Account Self-Regulation and Different Proficiency of Learners. *International Journal of Emerging Technologies in Learning (iJET)*, 17(23), 38–55. https://doi.org/10.3991/ijet.v17i23.36425
Kulhavy, R. W., & Stock, W. A. (1989). Feedback in written instruction: The place of response certitude. *Educational Psychology Review, 1*(4), 279–308. https://doi.org/10.1007/BF01320096

Narciss, S. (2008). Feedback strategies for interactive learning tasks. In van Merrienboer, J. M. Spector, M. D. Merrill, & M. P. Driscoll (Eds.), *Handbook of research on educational communications and technology* (3rd ed., pp. 125–144). Lawrence Erlbaum.

Narciss, S. (2012). Feedback Strategies. In N. M. Seel (Ed.), *Encyclopedia of the Sciences of Learning* (pp. 1289–1293). Springer US. https://doi.org/10.1007/978-1-4419-1428-6_283

Narciss, S., & Huth, K. (2004). How to design informative tutoring feedback for multi-media learning. In H. M. Niegemann, D. Leutner, & R. Brünken (Eds.), *Instructional Design for Multimedia learning* (pp. 181–195). Waxmann.

Pals, F. F. B., Tolboom, J. L. J., & Suhre, C. J. M. (2023). Development of a formative assessment instrument to determine students’ need for corrective actions in physics: Identifying students’ functional level of understanding. *Thinking Skills and Creativity, 50*, 101387. https://doi.org/10.1016/j.tsc.2023.101387

Ramdass, D., & Zimmerman, B. J. (2008). Effects of Self-Correction Strategy Training on Middle School Students’ Self-Efficacy, Self-Evaluation, and Mathematics Division Learning. *Journal of Advanced Academics, 20*(1), 18–41. https://doi.org/10.4219/jaa-2008-869

Renkl, A. (2014). The Worked Examples Principle in Multimedia Learning. In R. E. Mayer (Ed.), *The Cambridge Handbook of Multimedia Learning* (2nd ed., pp. 391–412). Cambridge University Press. https://doi.org/10.1017/CBO9781139547369.020

Sangwin, C. (2023). Running an Online Mathematics Examination with STACK. *International Journal of Emerging Technologies in Learning (iJET)*, 18(03), 192–200. https://doi.org/10.3991/ijet.v18i03.35789

Tärning, B., Lee, Y. J., Andersson, R., Månsson, K., Gulz, A., & Haake, M. (2020). Assessing the black box of feedback neglect in a digital educational game for elementary school. *Journal of the Learning Sciences, 29*(4–5), 511–549. https://doi.org/10.1080/10508406.2020.1770092
# Students' instrumentalizations of hints and automated feedback in their task solution process when learning mathematics with a digital curriculum resource

Sebastian Rezat

Paderborn University, Institute of Mathematics, Germany; srezat@math.upb.de

Automated feedback is a characteristic feature of digital curriculum resources. Recently, there has been a growing interest in students' perspectives on feedback. Feedback is increasingly regarded as a dialogic process in which learners make sense of information from varied sources and use it to enhance the quality of their work or their learning strategies. This study aims to further contribute to the body of research on the learners' perspectives on feedback, by investigating for what purpose students make use of hints and automated feedback when learning mathematics with a digital curriculum resource. Students' use of hints and automated feedback is analyzed through the lens of instrumental genesis. Results from a qualitative study with eight 8th-grade students show which type of hint or feedback is used at what phase for what purpose in the process of solving tasks from a widely available online curriculum resource.

Keywords: Automated feedback, hints, user study, students, digital curriculum resources.

## Introduction

Automated feedback is a characteristic feature of digital curriculum resources (Choppin et al., 2014; Rezat, 2020). It aims to support students individually in their learning processes. The important role of feedback in learning is widely acknowledged (Hattie & Timperley, 2007). For a long time, feedback was regarded as a unidirectional process in which learners are viewed as receivers of information from an external source that they use to enhance their learning. Consequently, most research on feedback focuses on variables of the feedback message, such as the contents or the timing of when it is provided. Only in the past years, there has been a growing interest in students' perspectives on feedback (Esterhazy & Damşa, 2019; Molloy & Boud, 2014; Olsson, 2018). Feedback is increasingly regarded as a dialogic process in which learners make sense of information from varied sources and use it to enhance the quality of their work or their learning strategies (Carless, 2015). Consequently, the meaning of feedback is not only determined by the feedback message, but by both, the agent and the user (Esterhazy & Damşa, 2019). Accordingly, there is a growing interest in better understanding how students seek, interpret, and use information related to their learning and how programs are designed to foster this (Molloy & Boud, 2014).

Elsewhere, I have shown how feedback can afford or constrain students' conceptual development (Rezat, 2021) and that the interpretation of signs related to feedback on the artifact level imposes additional challenges on students (Rezat et al., 2021). This study aims to further contribute to the body of research on the learners' perspective on feedback, by investigating the research question: For what purpose do students make use of hints and automated feedback when learning mathematics with a digital curriculum resource? The focus here is on the purpose of using the supportive information provided by the digital curriculum resource.
# Theoretical framework

The instrumental approach (Rabardel, 2002) theorizes cognitive aspects of human interactions with digital artifacts and has proven useful in understanding students' learning of mathematics with Computer-Algebra and Dynamic Geometry Systems. In this paper, hints and feedback are considered artifacts developed to support students individually in their learning of mathematics. To develop a detailed account of how hints and feedback function in the learning process, students' use of these artifacts is analyzed through the lens of the instrumental approach.

## Instrumental approach

According to Rabardel (2002) an artifact is transformed into an instrument in use. An instrument is a psychological entity that consists of an artifact component and a scheme component. In using the artifact, the subject attributes functions to the artifact and develops or adjusts utilization schemes that are shaped by both, the artifact and the subject. Attributing functions and the development of utilization schemes are two opposite but intertwined processes, which Rabardel refers to as “instrumentalization” and “instrumentation”. In this paper, the focus is on the different functions that students attribute to hints and feedback in their learning process while solving tasks in an online learning platform and thus on their *instrumentalization*. Rabardel (2002, p. 106) defines *instrumentalization* as a “process in which the subject enriches the artifact’s properties”. Although this process is grounded in the artifact’s intrinsic characteristics and properties, it is mainly linked to the subject’s goals and conditions for action in a situation.

## Hints and feedback

Feedback is widely defined as “information provided by an agent (e.g., teacher, peer, book, parent, self, experience) regarding aspects of one’s performance or understanding” (Hattie & Timperley, 2007, p. 81). Therefore, it only relates to information provided to students after they have solved a task and entered a solution into the system. However, many digital curriculum resources also offer information that learners can access on their way to their first solution before they have entered it into the system. This information is widely referred to as hints, cues, prompts, or tips. Referring to conceptualizations of feedback as a dialogic process in which learners make sense of information from varied sources and use it to enhance the quality of their work or their learning strategies (Carless, 2015) it makes sense to include this information in the analysis. Therefore, this paper analyzes students’ use of supportive information provided by a widely used online curriculum resource that is offered in addition to the task itself before or after students have entered a solution to a task into the system. Consequently, clear conceptualizations of both, hints and feedback are necessary.

Whether a given task is a routine task or a problem does not only depend on task features but also students' knowledge and abilities. If students do not know how to find the solution to a given routine task immediately, the task becomes a problem for students. Zech (2002) suggests a taxonomy of five levels of hints that might support students in their problem-solving process: 1) motivational hints that motivate learners to continue the problem-solving process, 2) feedback that informs learners about the correctness of the selected solution strategy or achieved intermediate steps towards the solution, 3) general strategic hints that suggest a general problem-solving strategy to learners, 4) content specific strategic hints that provide learners with information about the solution strategy for the problem at hand, 5) content related hints provide learners with particular content that is relevant for
solving the problem. A comparison between these types of hints and the different types of feedback
in the next paragraph will show unmistakable overlaps in their content. Additionally, the digital
curriculum resource used in this study sometimes offers the same information before and after
entering the solution. Therefore, depending on when the information is presented it would be either
considered a hint or elaborated feedback. Consequently, I use the same terminology to distinguish
the different types of hints and feedback.

To differentiate different types of feedback offered by digital curriculum resources, the study presented in this paper refers to the classification of feedback according to Shute (2008). Shute distinguishes different types of feedback according to their complexity. The following types are relevant:

1. Knowledge of results feedback (KR) informs the learner about the correctness of an answer.

2. Knowledge of correct response (KCR) feedback informs the learner about the correct response.

3. Repeat-until-correct (RUC) feedback informs the learner about an incorrect response and offers the possibility of a new try to answer the task.

4. Location of mistakes (LOM) feedback informs the learner about the location of an error in the solution without giving the correct response.

5. Elaborated feedback (EF) offers further information regarding the solution of the task or the solution of the learner.

For the last type (EF), the literature distinguishes many different subtypes. For the study presented in this paper, the following types of EF are relevant that Shute (2008, p. 160) subsumes under “topic contingent” and “hints/clues/prompts”:

- knowledge about concepts (kac),

- knowledge or strategic information on how to proceed (kohp),

- a worked example or demonstration (we)

Types of hints and feedback in the used digital curriculum resource

As apparent from the theoretical framework, hints and feedback are characterized differently and use
different terminology. However, the contents of the messages seem to be equivalent in many cases.
Additionally, *bettermarks* shows the same message sometimes as a hint and sometimes as feedback.
To develop a clear terminology to denote the information provided by hints or feedback messages,
the types of hints according to Zech (2002), types of feedback according to Shute (2008), and the
features of the used digital curriculum resource (DCR) that contain these types of hints and feedback
are juxtaposed in Table 1. This juxtaposition reveals the overlaps in terms of the content of the
feedback message. However, the scope of some of the types is different. On the one hand, the
differentiation between general and content-specific strategic hints is not mirrored in different types
of feedback, on the other hand, both, elaborated feedback presenting knowledge on how to proceed,
or a worked example can be considered content-specific strategic hints. Comparing the types of hints
and feedback with the related features in the curriculum resource shows that a single feature may
present a variety of different types of hints or feedback. Especially the feature “Tip” may provide
knowledge about concepts, knowledge on how to proceed, or a worked example.
<table><caption>Table 1: Juxtaposition of hints, types of feedback, and related features in the used DCR</caption><thead><tr><td>Type of hint</td><td>Type of feedback</td><td>Appearance in the DCR</td></tr></thead><tbody><tr><td>1) motivational hints</td><td>-</td><td>-</td></tr><tr><td>2) hints that provide learners with feedback about the correctness of the selected solution strategy or achieved intermediate steps toward the solution</td><td>KR-feedback</td><td>KR-feedback after a solution was entered</td></tr><tr><td>3) general strategic hints</td><td>EF (kohp)</td><td>-</td></tr><tr><td>4) content specific strategic hints</td><td>EF (kohp)</td><td>Feature called “Tip” Sometimes shown automatically after entering a wrong solution</td></tr><tr><td></td><td>EF (we)</td><td>Feature called “Lookup” Feature “Example” sometimes appearing after a wrong solution</td></tr><tr><td>5) content related hints</td><td>EF (kac)</td><td>Feature called “Tip” Linked technical terms in the task</td></tr></tbody></table>

Methodology

This study aims to analyze students' use of hints and feedback from digital curriculum resources in an ecologically valid setting. Therefore, the widely used resource *bettermarks* (www.bettermarks.com) was used. *Bettermarks* offers a wide range of different types of hints and feedback while solving a task. It is licensed in several federal states in Germany by the federal ministries of education and thus offered for free to schools. In this study, eight students in eighth grade were working on a unit on percentages. The unit comprised a set of 17 tasks and problems that students worked through at the end of the unit on percentages as a preparation for a test. The tasks were carefully selected to comprise different kinds of problems covering all the content that was relevant for the test and offering to students possibly all the different kinds of hints and feedback available in *bettermarks* (at that time). The eight students worked on the set of tasks at home in their familiar setting in a video conference with the interviewer on a shared screen. The students were used to working with *bettermarks* at home. Thus, the situation of using *bettermarks* was kept as natural as possible. The only difference was the presence of the interviewer in the video conference.

The recordings of the video conferences provide the data for this study. The videos were analyzed using the qualitative data analysis software MAXQDA. In the first step, each video was coded for the different types of hints and feedback used by the students. In the second step, each episode in which the students used a hint or received feedback was analyzed in terms of the purpose that the students associated with its use. This was done based on the constant comparative method (Corbin & Strauss,
2015) until different instrumentalizations could be delineated and defined. As instrumentalizations refer to functions that students attribute to the hints and feedback by their goals and conditions for action in a situation, this was achieved by inferring students' motivations or reasons and their goals for using a hint or feedback from the data. These partly depend on the phase in the solution of the problem, in which students make use of the hint or feedback.

## Results

Table 2 provides an overview of the results of the analysis of students' instrumentalizations of the different types of hints and feedback that are offered by bettermarks. The left column of Table 2 is organized in chronological order and describes the different phases that students must go through when solving a task from bettermarks. These different phases characterize the situations in which hints and feedback are used. As described in the theoretical framework, these influence students' instrumentalization of hints and feedback. The second column contains the different types of hints or feedback that are offered by bettermarks in the different phases and were used by the students. In phases 1–3, the types characterize the contents of hints. Starting in phase 4, the types relate to feedback. The third column shows students' instrumentalizations of these types of hints and feedback in the respective phase.

Table 2 may be read in the following way: While reading the task (phase 1) bettermarks offers hints of kac-type. Students instrumentalize these hints to enhance their understanding of technical terms that appear in the tasks. In phase 2, when students aim to find the solution to the task, they have access to three different kinds of hints: They can open kac-type or kohp-type hints or they can ask for the complete solution of the task (KCR). Kac-type and kohp-type hints are instrumentalized in two different ways: Students either use them to get support in finding the solution or to resolve uncertainties about the expected input format when entering the solution into the system. KCR is used to understand the expected solution if students do not develop any solution on their own. Starting in phase 4, columns 2 and 3 are divided as there are two possibilities for feedback depending on the correctness of the entered solution. The same applies to the fields in phase 7. Fields shaded in grey denote that no more feedback or hints are accessible at these phases.

<table><caption>Table 2: Cumulated results of students' instrumentalizations of different types of hints and feedback in the different phases of the task solution process</caption><thead><tr><th>Phase in the solution process</th><th>Type of hint / feedback</th><th>Instrumentalization</th></tr></thead><tbody><tr><td>1. Reading task</td><td>kac</td><td>Enhancing understanding of technical terms in the task</td></tr><tr><td rowspan="2">2. Finding solution</td><td>kac</td><td>Getting support to find the solution</td></tr><tr><td>kohp</td><td>Resolving uncertainties about the expected input format</td></tr><tr><td></td><td>KCR</td><td>Understanding the expected solution</td></tr></tbody></table>
<table><tr><td>3. Entering solution</td><td colspan="2">kac<br/>kohp</td><td colspan="2">Confirming that the solution (procedure) is correct before entering</td></tr><tr><td rowspan="2">4. Evaluation of the entered solution</td><td>KR feedback: correct</td><td>KR feedback: incorrect</td><td>Reassurance of own solution</td><td>Removing uncertainties about which solution from two alternatives is the correct one</td></tr><tr><td>KCR</td><td></td><td colspan="2">Resolving uncertainties about the expected input format</td></tr><tr><td></td><td></td><td></td><td>Checking own / alternative solution</td><td></td></tr><tr><td>5. Rethinking solution</td><td colspan="2">KCR</td><td colspan="2">Understanding the expected solution<br/>Understanding own mistakes</td></tr><tr><td></td><td colspan="2">Hints as in phases 2 & 3</td><td colspan="2">Instrumentalizations as in phases 2 & 3</td></tr><tr><td>6. Entering adjusted solution</td><td colspan="2">KR correct<br/>KR incorrect</td><td colspan="2">No particular instrumentalization observed</td></tr><tr><td>7. Evaluation of solution</td><td colspan="2">KCR</td><td colspan="2">Understanding the expected solution<br/>Understanding own mistakes</td></tr></table>

## Discussion

Many of the instrumentalizations of hints and feedback can be expected and seem to fit the intended purpose of the type of hint or feedback. This is for example the case for the following hint/feedback/instrumentalization pairs: (kac&kohp/getting support to find the solution) or (KCR/understanding the expected solution). However, some instrumentalizations are particularly interesting as they indicate difficulties that students have with solving tasks from a DCR. This is especially the case for instrumentalizations related to issues with the input format. In these cases, students are not sure, what kind of input or input format (e.g., fraction or decimal, exact or rounded decimal) is expected by the system. They either instrumentalize hints to resolve these uncertainties
before entering the solution or the *KR* feedback helps to resolve the uncertainties after entering the solution. Another unexpected instrumentalization is that some students who have found the solution to a task instrumentalize use hints before entering the answer to ensure that their answer is correct instead of simply using the *KR*-feedback for this purpose. However, these utilizations are closely related to the constraints of the DCR. They are mostly not of mathematical relevance. For example, if a number is written as a fraction or a decimal is equivalent from a mathematical point of view, but the DCR only accepts one input as correct.

On the one hand, the results show that different kinds of hints and feedback are instrumentalized for the same purpose. For example, students use *kac* or *kohp* hints or *KR*-feedback to resolve uncertainties about the expected input format. However, as the results give a cumulated overview of the observed instrumentalizations of all participating students, nothing can be said about whether this is the case for one particular student or if this is an observation that only appears between different students. A deeper, case-sensitive analysis is necessary to reveal if a particular student shows a definite instrumentalization of a particular type of hint or feedback for a specific purpose. If this is not the case, i.e. if one student uses different types of hints and feedback for the same purpose it may be an indication of an incomplete instrumental genesis. However, it may also be an issue with the system, as it is not always clear what kind of hint is provided by *bettermarks* when students look for support while solving a task especially when they use the feature “Tip”. Furthermore, the analysis only focused on what type of hint or feedback was used for a particular purpose. It was not analyzed if the hint or feedback was actually supportive in the sense that it either helped to solve the problem or if students thought that the information was helpful. This would also be a matter of deeper analysis.

On the other hand, the results show that one type of feedback is instrumentalized for different purposes. For example, *KCR*-feedback is instrumentalized for two different purposes: First, for understanding the expected solution before or after entering a solution when students do not have a clue of how to solve the tasks. Second, students also instrumentalize *KCR*-feedback after entering the correct answer to compare their solution with the provided one to check their solution procedure or to see an alternative solution.

In summary, the results show that students instrumentalize the different types of hints and automated feedback offered by the system when solving tasks from a DCR for different purposes and that one type of hint or feedback may be instrumentalized at different phases of the solution process for different purposes. Consequently, these results underline the starting point of the study, namely that “the meaning of feedback is not only determined by the feedback message, but by both, the agent and the user” (Esterhazy & Damşa, 2019). A deeper analysis may reveal how students’ instrumentalizations of hints and feedback contribute to a successful solution of the task and their learning of mathematics. These insights could be helpful for the design of hints and automated feedback in DCR as they show students’ difficulties and needs during their individual learning processes.

## References

*Carless, D. (2015). *Excellence in university assessment: Learning from award-winning practice*. Routledge.
Choppin, J., Carson, C., Borys, Z., Cerosaletti, C., & Gillis, R. (2014). A typology for analyzing digital curricula in mathematics education. *International Journal of Education in Mathematics, Science and Technology, 2*(1), 11–25.

Corbin, J., & Strauss, A. (2015). *Basics of qualitative research: Techniques and procedures for developing Grounded Theory* (4th ed.). Sage.

Esterhazy, R., & Damşa, C. (2019). Unpacking the feedback process: An analysis of undergraduate students’ interactional meaning-making of feedback comments. *Studies in Higher Education, 44*(2), 260–274. https://doi.org/10.1080/03075079.2017.1359249

Hattie, J., & Timperley, H. (2007). The power of feedback. *Review of Educational Research, 77*(1), 81–112. https://doi.org/10.3102/003465430298487

Molloy, E. K., & Boud, D. (2014). Feedback models for learning, teaching and performance. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), *Handbook of research on educational communications and technology* (pp. 413–424). Springer. https://doi.org/10.1007/978-1-4614-3185-5_33

Olsson, J. (2018). The contribution of reasoning to the utilization of feedback from software when solving mathematical problems. *International Journal of Science and Mathematics Education, 16*(4), 715–735. https://doi.org/10.1007/s10763-016-9795-x

Rabardel, P. (2002). *People and technology: A cognitive approach to contemporary instruments* https://hal.archives-ouvertes.fr/hal-01020705/document

Rezat, S. (2020). Mathematiklernen mit digitalen Schulbüchern im Spannungsfeld zwischen Individualisierung und Kooperation. In D. M. Meister & I. Mindt (Eds.), *Mobile Medien im Schulkontext* (pp. 199–213). Springer Fachmedien. https://doi.org/10.1007/978-3-658-29039-9_10

Rezat, S. (2021). How automated feedback from a digital mathematics textbook affects primary students’ conceptual development: Two case studies. *ZDM – Mathematics Education, 53*, 1433–1445. https://doi.org/10.1007/s11858-021-01263-0

Shute, V. J. (2008). Focus on formative feedback. *Review of Educational Research, 78*(1), 153–189. https://doi.org/10.3102/0034654307313795

Zech, F. (2002). *Grundkurs Mathematikdidaktik. Theoretische und praktische Anleitungen für das Lehren und Lernen von Mathematik* (10 ed.). Beltz.
# Professionalising teachers with a digital formative assessment tool: a case study of the SMART tests

Fabian Rösken and Bärbel Barzel

University of Duisburg-Essen, Germany; fabian.roesken@uni-due.de

SMART, a digital formative assessment tool for mathematics education, aids teachers by offering short online diagnostic checks and detailed analyses of response patterns. It identifies student misconceptions and provides teachers with valuable insights on students' level of understanding. This should have the potential to not only influence teaching and student understanding but also the teachers' professionalisation. This paper explores the potential impact of using SMART on five mathematics teachers' PCK in elementary algebra. Initial findings indicate the tool's effectiveness in enhancing teachers' knowledge of student thinking and typical mistakes.

Keywords: Technology-supported formative assessment, PCK, teacher professionalisation

## Theoretical background

Formative assessment has been proven to be a valuable approach for students to benefit in their learning. Andrade & Heritage (2018) concretise that a successful implementation of formative assessment consists of a diagnosis of students' performance and progress, and an adjustment of teaching to the individual needs, for example by reacting to diagnosed misconceptions. Busch et al. (2015) have investigated teachers' formative assessment practices and found out, that content specific pedagogical content knowledge (PCK) has a high impact on diagnostic practices. They observed a shift from superficial diagnoses (e.g. just correcting mistakes) to a deeper analysis as soon as aspects of content specific PCK are shown. The PCK of teachers is often divided into different equally meaningful aspects, e.g. in the COACTIV study it was defined as *knowledge of explaining and representation*, *knowledge of student thinking and typical mistakes*, and *knowledge of the potential of mathematical tasks*. This framework has demonstrated predictive validity in enhancing instructional quality and fostering student learning gains across various studies (Krauss et al., 2020).

Technology has the potential to support teachers in formative assessment processes while also developing the needed PCK of teachers (Stacey & William, 2013). One example of technology-based formative assessment is the SMART system, which provides a fast and in-depth diagnosis by specifically designed diagnostic items and their deep analysis (Stacey et al., 2018). After students complete a 5- to 10-minute test, only the teachers receive an automated diagnosis for each student in the form of comprehension levels and misconceptions. In addition, further explanations and teaching suggestions are provided, which include tips for teaching, general advice on rituals, attitudes, methods and desirable concepts, as well as concrete tasks that can be used directly to support the individual students (Price et al., 2013).

In this study we used a test from the field of algebra. The test *Meaning of Letters* examines whether students consistently interpret variables in such a way that they stand for numerical values and not as an abbreviation for objects occurring in the context. The test includes six multiple-choice items whose incorrect answer options reflect typical misinterpretations. For example, when the question is “Biros are sold in packs of 3. Sam bought *p* packs and got *b* biros altogether. Choose the correct equation”,
the answer option "3b = p" can be understood as "There are 3 biros in one pack", where the variables are interpreted as abbreviations for biros and packs respectively. This is known as the letter-as-object (LO) misconception (Stacey et al., 2018). This misconception is especially important since it can linger with students as they navigate their way through further algebra. Students who struggle to accurately formulate equations and expressions to represent real-world scenarios may miss out on harnessing the problem-solving potential of algebra. Given the accessibility of digital technology capable of solving equations, it becomes increasingly crucial for students to master the skill of constructing equations and accurately interpreting equations created by others, rather than focusing solely on solving equations manually (Arcavi et al., 2017). The SMART test compares the answer to the Biro item with the responses to similar items and searches for specific answer patterns to reveal if a student has this misconception.

The automatic diagnosis is displayed to the teachers in the form of three comprehension levels, based on the frequency on which this misconception is revealed. One additional misconception that the students may hold could also be flagged. The solution-as-coefficient (SAC) misconception is a subtype of the LO misconception, where a possible solution for an equation is already found and placed in front of the variables as coefficients. It is particularly important that teachers are able to recognise these misconceptions and know how to address them. Teachers who recognise that existing misconceptions may impede algebra learning can support their students by openly addressing the distinctions between coding, other notation systems, and the realm of algebra. Therefore the PCK aspect *knowledge of student thinking and typical mistakes* is particularly important for successful algebra teaching.

## Research design

The nature of SMART tests suggests that teachers increase their *knowledge of student thinking and typical mistakes* while using SMART, because they engage with their own students' misconceptions and possible ways to overcome them. The aim of this study was to find out, to what extent a development of content specific PCK among teachers can be seen after using the SMART tests. To answer this question, five teachers of grade 7 participated in a pre-intervention-post-test design. The *knowledge of student thinking and typical mistakes* was surveyed in a pre-test with a competency test. As an intervention, they used the SMART test *Meaning of Letters* with their students and, based on the results, taught the topic of variables and their meaning for about 4 weeks. They then used a second version of this SMART test with their students to track student development. As a post-test, the teachers were surveyed with a second version of the competency test.

To test the PCK competency, an existing test by Busch et al. (2015) was adapted for algebra and understanding of variables. The analysis of the COACTIV study revealed that written tests were highly predictively valid for individual learning support (Krauss et al., 2020). In the competency test, four example student solutions were generated for teachers to assess. The examples represent typical student solutions to tasks on the understanding of variables and the concept of terms and equations and contain frequent misconceptions. Table 1 shows an example of one of the four examples in which the LO misconception is present. With the relevant *knowledge of student thinking and typical mistakes*, it can be recognised that this student does not interpret the variable as a placeholder for a numerical value, but rather as an abbreviation for an object (*m* stands for marshmallows).
Additionally, the students' solution includes the correct answer as the equation (SAC) and is likely to be read as a solution sentence ("*4 marshmallows and 5 caramels together cost 80 cents* ").

<table><caption>Table 1: Example student solution “Candy shop” from the pre-test</caption><tr><td>As a task to formulate linear equations, Anthony received the following task:</td></tr><tr><td>Catherine went into a candy shop:</td></tr><tr><td>“I have bought marshmallows and caramels and paid a total of 80 cents.</td></tr><tr><td>The marshmallows cost 10 cents each and the caramels 8 cents.”</td></tr><tr><td>Formulate an equation, that describes this situation.</td></tr><tr><td>Anthony wrote: $4m + 5c = 80$</td></tr></table>

The teachers were asked to formulate their own diagnosis as well as approaches for spontaneous and further supportive teaching practice. The teachers' answers were analysed in how far they could describe the possible student thinking and if they – implicitly or explicitly – refer to the underlying misconceptions the student may hold. The analyses of the pre- and post-tests were compared and examined for a development of PCK. The teachers' statements were qualitatively analysed by the first author and interated by at least one other person.

## Results

After examining the cases in how far they show *knowledge of student thinking and typical mistakes*, we were able to contrast three different ways of development: (1) two teachers showing implicit knowledge in the pre-test and becoming more explicit in the post-test, (2) two teachers showing no corresponding knowledge in the pre-test and showing implicit knowledge in the post-test, and (3) one teacher who constantly shows a lot of explicit knowledge.

As stated above, two teachers exhibited an implicit *knowledge of student thinking and typical mistakes*, primarily focusing on describing and evaluating the student's solution. For instance, they acknowledged the student's grasp of the situation but noted the inclusion of an unnecessary solution in the equation. Implicitly addressing the SAC misconception, the teachers lacked further explanation, failing to identify the underlying LO misconception. This gap was evident in the superficial and process-oriented supportive practices they expressed, in which they emphasised to highlight key information of the text and practice general strategies for text tasks. In the post-test, the teachers can identify slightly more precise that the student shows difficulties with the meaning of the variables. This becomes evident when they express possible supportive approaches in the post-test, because they focus on the meaning of the variable and suggest to "*write down what m and c mean*" and "*use simple number examples to check the solution*".

A similar development can be seen for two of the teachers who not only remained on a superficial focus in the pre-test ("*The equation does not make sense*"), but sometimes even diagnosed incorrectly and did not perceive the student's error at all. Accordingly, the supportive teaching practices remained predominantly at a general, motivational level ("*He should read the assignment again carefully*") or at a process-oriented level, vaguely suggesting that they would try to solve the equation together with the student. In the post-tests, these teachers showed a greater understanding of the student's difficulties, and the approaches show a more concrete connection to the task and the student's errors.
For example, they suggest that the student should write the equation again without using the initial
letters. This hints that these teachers show an implicit understanding of the LO misconception.

In one case, a teacher showed a consistently high *knowledge of student thinking and typical mistakes* in pre- and post-test ("*For Anthony, m seems to stand for the object marshmallows, and he is probably thinking of a spoken form of the equation: '4 marshmallows and 5 caramels cost 80 cents.' He apparently already has a solution in his head.*").

**Discussion**

The preliminary findings on SMART’s impact on teachers’ PCK are promising, as progress was seen for all teachers regarding the PCK aspect *knowledge of student thinking and typical mistakes*. While initial diagnoses had few variations, all teachers showed more explicit knowledge of underlying misconceptions to students’ errors in the post-test. Those with implicitly recognisable knowledge in the pre-test became more explicit, utilising content specific terms. Teachers that struggled to explain student thinking in the pre-test could show implicit knowledge of the LO misconception in the post-test. Teachers with a lot of corresponding knowledge maintained this precision. These results align with the expectation that SMART increases teachers’ *knowledge of student thinking and typical mistakes* and strengthens them to avoid an exclusive procedural knowledge focus. The presented cases are part of a broader study with over 40 teachers receiving various distinct interventions.

**References**

Andrade, H. L., & Heritage, M. (2018). *Using formative assessment to enhance learning, achievement, and academic self-regulation*. Routledge

Arcavi, A., Drijvers, P. & Stacey, K. (2017). *The learning and teaching of algebra*. Routledge.

Busch, J., Barzel, B. & Leuders, T. (2015). Die Entwicklung eines Instruments zur kategorialen Beurteilung der Entwicklung diagnostischer Kompetenzen von Lehrkräften im Bereich Funktionen. *Journal für Mathematik-Didaktik*, 36(2), 315–338.

Krauss, S., Bruckmaier, G., Lindl, A., Hilbert, A., Binder, K., Steib, N. & Blum, W. (2020). Competence as a continuum in the COACTIV study: the “cascade model”. *ZDM Mathematics Education*, 52, 311–327.

Price, B., Stacey, K., Steinle, V., & Gvozdenko, E. (2013). SMART online assessments for teaching mathematics. *Mathematics Teaching*, 235(4), 10–15.

Stacey, K., Steinle, V., Price, B., & Gvozdenko, E. (2018). Specific mathematics assessments that reveal thinking: An online tool to build teachers’ diagnostic competence and support teaching. In T. Leuders, J. Leuders, K. Philipp, & T. Dörfler (Eds.), *Diagnostic competence of mathematics teachers – Unpacking a complex construct in teacher education and teacher practice* (pp. 241–263). Springer

Stacey, K., & William, D. (2013). Technology and assessment in mathematics. In M. A. K. Clements, A. J. Bishop, C. Keitel, J. Kilpatrick & F. K. S. Leung (Eds.), *Third international handbook of mathematics education* (pp. 721–751). Springer.
# Formative feedback in problem-solving lessons in German primary schools

Yasmin Theile¹ and Benjamin Rott²

¹,² University of Cologne, Germany; yasmin.theile@uni-koeln.de

In this article, we present initial results of the research on formative feedback, which is used by teachers in problem-solving orientated lessons in German primary schools. The analyses of five lessons on problem solving from German primary schools revealed 15 different forms of formative feedback from different levels (Hattie & Timperley, 2007) which will be exemplified. Most of the feedback forms identified relate to the task, results or process. Formative Feedback regarding the level of self-regulation could not be identified.

Keywords: Problem solving, primary education, formative feedback.

## Introduction

*We all need people who will give us feedback. That's how we improve.*

Bill Gates

Problem solving (PS) is one of the main activities of mathematical work and therefore an important skill to be learned in school mathematics. It is uncontroversial that students should learn to solve problems from primary school onwards and to reflect on and, if necessary, adapt their approaches to PS. However, studies have shown that primary school students in particular have problems in performing these skills (Heinrich et al., 2014). As a result, there is an increased need for teacher support to improve students' PS skills. The opening quote supports the importance of feedback for improvement – feedback can therefore be seen as a starting point for promoting problem-based teaching. As the particular complexity of everyday teaching makes diagnosis and specific support for students difficult (Heinrichs & Kaiser, 2018), this study focuses on formative feedback from teachers.

## Theoretical Background

Feedback is one of the key strategies of formative assessment (Black & William, 2009), which can be defined as a *practice* that is

formative to the extent that evidence about student achievement is elicited, interpreted, an used by teachers, learner, or their peer, to make decisions about the next step in instruction that are likely to be better, or better founded, than the decision they would have taken in the absence of the evidence that was elicited (Black & William, 2009, p. 9).

Feedback can generally be “conceptualized as information provided by an agent (e.g., teacher, peer, book, parent, self, experience) regarding aspects of one’s performance or understanding” (Hattie & Timperley, 2007, p. 81). In the field of educational research, a distinction is also made between formative and summative feedback. Formative feedback aims to record the current learning status of the students and, based on this, to adapt the teaching and control the learning process of the students (Black & William, 2009; Polly et al., 2017). Hattie and Timperley (2007) also distinguish between four different levels of formative feedback. Feedback at the personal level is aimed directly at students and mainly includes praise or motivation and is often unrelated to the task or performance of the
student (Hattie, 2012; Hattie & Timperley, 2007). Feedback at the task and product level includes all feedback and information about how well a task has been mastered or performed (Hattie & Timperley, 2007). It is also often called corrective feedback and is the most common form of feedback given in a classroom setting (Hattie, 2012). Feedback at the process level refers to the processes on which the execution of a task is based. It "can lead to providing alternative processing, reducing cognitive load, [or] helping to develop learning strategies and error detection" (Hattie, 2012, p. 119). The highest form of feedback takes place at the level of self-regulation. Feedback at this level refers to the students' ability to monitor their own learning processes (Hattie, 2012). Although the influence of feedback on learning is undisputed, there are so far – especially in relation to problem-based teaching – only a few empirical findings on the implementation of formative feedback in mathematics education (e.g. Green, 2023). Since studies have shown that the positive effects of formative feedback on learning could depend on the subject and specific implementation (McLaughlin & Yan, 2017), studies are needed specifically on problem-based mathematics teaching.

## Research question and study design

To be able to make statements about the effectiveness of specific forms of feedback in problem-based teaching, it is useful to identify the forms of feedback that teachers use in everyday teaching. Amongst this background, the aim of the study is to investigate how teachers support their students' solution processes in problem-based lessons. This article will focus on the question which forms of formative feedback can be identified among primary school teachers in problem-based lessons. Tor this study, five lessons were videotaped in different primary schools in which the predefined aim of the lessons was to work on a problem-based task. For this purpose, the teachers were given a catalogue of problems to choose from. Three teachers taught a 2^nd^-grade class, the other two 4^th^-grade classes; more lessons are currently recorded and evaluated. Each lesson (45 min) was videotaped using two cameras to cover the classroom and a microphone or GoPro for the teacher. No further specifications were given to the teachers in advance.

The analysis is based on Mayring's (2000) qualitative-content-analysis methodology, allowing for an deductive-inductive category development. To develop a category system, the levels of feedback (Hattie & Timperley, 2007) were used as superordinate categories to provide a basic structure. The various forms of feedback that could be identified in the analysis were then described in detail, categorized by similar characteristics and subordinated to the levels of feedback. The previous analyses were performed by one coder.

## Results

We identified a total of 176 situations in which teachers gave feedback to students on their work. These could be summarized into 15 feedback categories, which in turn could be categorized into three of the four feedback levels. Only three, because feedback at the level of self-regulation has not been identified in the so-far recorded data. One category, the feedback form *R<sub>1</sub>: Reception signal*, could not be assigned to any of the four levels and was therefore coded as Unspecific reactions (Rx). Typical reception signals that could be observed were *hm*, *mhm*, *aha* or emotional expressions such as laughter as reactions. The category *T<sub>1</sub>: Correctness of Result* – categorized at the task and product level (*T*<sub>x</sub>) – is coded when the teacher explains to the students that a (partial) result is correct, incorrect or incomplete without providing further information. As this category could be identified in the lesson
of each teacher, it will be shown as an example below. Mrs. P's interaction with a student illustrates this. The students must find the number of squares that can be found on a chessboard (204), which is likely to be a problematic task for many primary school students, as they do not have a suitable solution scheme is not immediately obvious. The student approaches her teacher during the lesson and states:

157 Student: Mrs. P. ... I think I'm done, I have 118.

158 Mrs. P.: No its more.

159 Student: What, great.

After the last sentence, the student returns to her seat. The teacher does not provide any further information, such as what has not yet been correct or forgotten by the student. Interactions that could be assigned to this category were identified 24 times.

A complete list of all categories that could be identified is given in figure 1. It also shows how often each category occurred in the analyses. The letter in the abbreviation stands for the level to which the categories have been assigned – personal level ($S_x$), task and product level ($T_x$) and process level ($P_x$). The complete coding agenda will be gladly provided on request.

<figure><img src="image_80.png" /><figcaption>Figure 1: Categories of formative Feedback in problem-based lessons</figcaption></figure>

It is noticeable that some forms of feedback such as *T<sub>2</sub>: Reference to task conditions* and *T<sub>8</sub>: Providing a solution approach* were identified very frequently and with every teacher, while others such as *P<sub>5</sub>: Tip Cards* occurred only with individual teachers. However, it should be noted that these results may be due to the small amount of data in this study. The analysis of further lessons will provide more in-depth findings here. The results presented support the findings of Hattie (2012) that most feedback is given at the task and product level. Overall, it is noticeable that the teachers in this study most frequently chose forms of feedback that address the product level, although problem solving is a process-related skill and therefore the actual process should be the focus.

The study also revealed that certain forms of feedback were identified as recurring for a teacher. An exemplary case of this is Mrs. T. In the evaluation of her lesson, 40 feedback situations were identified in which she responded with one of three forms of feedback in around 64% of cases (*T*<sub>1</sub>, *T*<sub>2</sub> and *T*<sub>8</sub>). All other identified categories occurred only sporadically. While this phenomenon was recognizable among all teachers, the categories that occurred particularly frequently differed between the individual teachers. This will be analyzed in greater depth in further analyses.
Discussion

The aim of this study was to identify and analyze different forms of formative feedback which are
used by German primary school teachers in lessons regarding problem solving. It was possible to
identify 15 different categories, which can be assigned to three levels of feedback. The catalogue of
criteria resulting from this study is currently being further tested for its applicability by expanding
the data by analyzing additional lessons – including external analyzes by other coders. It should be
noted that there may be further changes to the list of categories in the subsequent analyzes. In
particular, categories at the level of self-regulation (Hattie & Timperley, 2007) that could not be
identified in the current analyzes are conceivable here. Based on the findings so far, further analyses
should also focus on exploring possible recurring preferred forms of feedback that occur across
lessons for individual teachers. In the long term, it seems particularly interesting to examine what
effect the forms of formative feedback used by the teachers have on students’ problem-solving
processes in order to allow specific conclusions for teaching practice.

References

Black, P., & William, D. (2009). Developing the theory of formative assessment. *Educational Assessment, Evaluation and Accountability, 21*(1), 5–31. https://doi.org/10.1007/s11092-008-9068-5

Green, J. (2023). Primary students’ experiences of formative feedback in mathematics. *Education Inquiry, 14*(3), 285–305. https://doi.org/10.1080/20004508.2021.1995140

Hattie, J. (2012). *Visible learning for teachers: Maximizing impact on learning*. Routledge.

Hattie, J., & Timperley, H. (2007). The Power of Feedback. *Review of Educational Research, 77*(1), 81–112. https://doi.org/10.3102/003465430298487

Heinrich, F., Jerke, A., & Schuck, L.-D. (2014). “Fehler” in Problembearbeitungsprozessen von Grundschulkindern. In J. Roth & J. Ames (Eds.), *Beiträge zum Mathematikunterricht 2014* (pp. 499–502). WTM.

Heinrichs, H., & Kaiser, G. (2018). Diagnostic Competence for Dealing with Students’ Errors: Fostering Diagnostic Competence in Error Situations. In T. Leuders, J. Leuders, & K. Philipp (Eds.), *Diagnostic Competence of Mathematics Teachers. Unpacking a Complex Construct in Teacher Education and Teacher Practice* (pp. 79–94). Springer International.

McLaughlin, T., & Yan, Z. (2017). Diverse delivery methods and strong psychological benefits: A review of online formative assessment. *Journal of Computer Assisted Learning, 33*(6), 562–574. https://doi.org/10.1111/jcal.12200

Polly, D., Wang, C., Martin, C., Lambert, R. G., Pugalee, D. K., & Middleton, C. W. (2017). The Influence of an Internet-Based Formative Assessment Tool on Primary Grades Students’ Number Sense Achievement. *School Science and Mathematics, 117*(3–4), 127–136. https://doi.org/10.1111/ssm.12214
# Students' feedback process using automated post-submission report in their modeling process

Muna Touma<fnref n="1" /> and Shai Olsher<fnref n="2" />

In this study, we explored a pair of students' interactions with the automated post-submission report to learn how they use this tool to deepen their own modeling process. The students worked on a modeling activity consisting of four example-eliciting tasks in which they were asked to construct position-over-time graphs. The findings show that by interacting with the post-submission report, new sequences of mathematical modeling competencies use were identified, which enables fostering the student's interpreting and validating the mathematical results.

Keywords: Feedback process, mathematical modeling, modeling process.

## Introduction and theoretical background

Feedback is an ongoing process in which learners make sense of information related to the task or process of learning, which is provided by an agent (e.g., teacher, peer, book) regarding aspects of one's performance or understanding (Hattie & Timperley, 2007), to fill a gap between what is understood and what is aimed to be understood (Sadler, 1989). Indeed, studies have shown that the feedback given to students has a greater impact on their achievement than any other teaching strategy (Hattie & Timperley, 2007). In their study, Yerushalmy et al. (2023) discussed how artifacts, particularly personal feedback that are designed as tools by others, might become instruments for learning. They referred to personal feedback as a mirror that describes an instance by offering a task designer's description of a learner's example. This description includes the characterization of students' examples in words.

Using technology in mathematics education affects the teaching and learning processes; specifically, it can be used to support the feedback process in many ways. Technology can be used to generate outputs such as numeric grades, written reports, and statistics for use by teachers (Sangwin & Köcher, 2016), students (Yerushalmy et al., 2023), or both (Abu Raya & Olsher, 2021; Sadler, 1989). Some technologies also have the feature of reporting to the teacher regarding the tools that students use when they solve the task (Abu Raya & Olsher, 2021), while others have the feature of analyzing the answers according to mathematical characteristics (Abu Raya & Olsher, 2021). In addition, the feedback process can be supported by the effective use of digital tools, such as simulations and interactive diagrams, which provide learners with immediate information that can help them solve problems and can positively affect various competences, such as understanding, validating, and interpreting, which are considered modeling sub-competences (Greefrath, 2011). Thus, technology can promote mathematical modeling, which is an important competency that students of all ages ought to acquire, owing to its role as a method for better understanding the world around us. Greefrath (2011) described two areas in which the modeling process occurs: the rest of the world and mathematics, and introduced a modeling cycle (MC) that represents key phases (in black font) and transitions between them (in red font) in the processing of reality-related problems. He suggests that

<footnote n="1">¹University of Haifa, Israel; munatouma89@gmail.com</footnote>

<footnote n="2">²University of Haifa, Israel; olshers@edu.haifa.ac.il</footnote>
digital tools may be useful at each step of this process (Figure 1). For example, for experimenting, by transforming a real situation into a geometrical model, with the help of dynamic geometry software.

<figure><img src="image_81.png" /><figcaption>Figure 1: Modeling cycle with the added influence of digital tools (Greefrath, 2011)</figcaption></figure>

Mathematical modeling literature highlights the need for more research to better understand how technology may assess or foster mathematical modeling competencies (MMC) (Cevikbas et al., 2022). Therefore, the purpose of this study is to gain insights into automated post-submission reports serving as a bridging tool between the literacy phenomenon and the given mathematical model. We asked: (1) How do students use post-submission reports while working on the modeling activity? (2) Do students' feedback processes using automated post-submission reports affect their MMC?

# Methodology

## Population

We present a case study involving a pair of 9th graders (14-year-old students), Ray and Lour (fictitious names), who completed a computer-based modeling activity with their teacher during mathematics lessons. The pair of students was chosen based on their educational level (medium-high), their good expressiveness, and their seriousness.

## Tools

The research tools included a modeling activity designed on the STEP platform and class observations. The STEP platform was designed to support the assessment of various patterns of open-ended example-eliciting tasks. The students' submitted examples were analyzed online according to specified characteristics, producing a post-submission report for the students (Harel et al., 2019). These characteristics can provide information that goes beyond whether students' submissions are correct or not, can give students a sense of the breadth of their personal example space, and may be resources to create a shift in students' understanding (Harel et al., 2019).

The modeling activity “Cycling” used in this study was designed on the STEP platform according to specific design principles (for further details, see Touma and Olsher, 2022). It includes four example-eliciting tasks that are designed so that the first task is an introductory task that helps students familiarize themselves with the content and tools, and the advanced tasks offer opportunities for higher cognitive demands. Some of the tasks include digital simulations and tools that allow interaction with the various components of the model, offloading part of the student's mathematical work to these tools (Touma and Olsher, 2022). This activity deals with constructing position-over-time graphs appropriate for real-life situations. In the first task, students were asked to use a given
diagram to drag blue points and submit three different examples of graphs that represent a girl named Nora's ride. The ride must meet two requirements: it ends at 13:00 and ends at home. The GeoGebra-based applet (like the left window in figure 2) included a 4-segment interactive graph that can be manipulated by dragging the blue points. The left blue point (a fixed point) represents Nora's starting location, and the right blue point represents Nora's finishing location, both relative to home, while the x-axis represents the time relative to 8 am.

In this study, we focused on the second task of the activity (described below), which, unlike the first task, includes a simulation. The uniqueness of the simulation, which adds value to the use of the graph, is that it presents features of the bicycle's motion, such as the change in position relative to home, the change in speed, and the change in direction. These animation features can enhance students' understanding of the underlying mathematics associated with the task. Beyond the given simulation, we reinforced these features through verbal descriptions (including non-critical characteristics in addition to the requirements of the task) that appear in the post-submission report produced automatically according to the data the students chose to submit.

## The task

The students were given the following instruction: *Yael rides a bicycle once a week. In each ride, she passes a total distance of 50 km in 4 hours. Use the diagram below, drag the blue points and submit three different examples of graphs representing Yael's ride.*

The task included a GeoGebra-based applet (Figure 2) divided into two windows. The left window includes a 3-segment interactive graph that can be constructed by dragging blue points. The left blue point represents Yael's starting location, and the right blue point represents Yael's finishing location, both relative to home, while the x-axis represents the time relative to 8 am. The right window includes a simulation of Yael's ride (overview of the track) corresponding to the graph on the left side of the applet, which is displayed by clicking the play button, and measurements of each simulation state (total distance, starting hour, and total time). Indeed, the applet includes multi-linked representations (MLR), which enable students to investigate various input data by reflecting their actions in a different representation and thus see their own ideas differently.

<figure><img src="image_82.png" /><figcaption>Figure 2: screenshot of the GeoGebra-based applet in task 2</figcaption></figure>
## Procedure

The pair of students worked together on the modeling activity within the mathematics class. They solved the tasks in the order in which they appeared in the activity. After completing the first task, the students did not refer to the post-submission report (despite the teacher's request to do so), but they did refer to it after completing the rest of the tasks.

## Data collection and analysis

The data collected in the study consisted of (a) students' submissions from the STEP platform, (b) video recordings of the pair's computer screen while working on a modeling activity, and (c) field notes of the researcher taken during the pair observations.

To answer the first research question, the first author attended and observed the whole class, focusing on the pair of students while using the post-submission report. The recordings and notes taken during the observations were then transcribed. We analyzed all the statements and actions that the students performed during the interaction with the post-submission report, while focusing on which parts of the report they referred to. The first author defined and coded the data by mapping each statement and action within the MC and linking it to MMC. She then generated categories for students' statements and actions (detailed below). To answer the second research question, the first author examined the interactions she found while answering the first question and associated them with the modeling process. She formulated the answer to the second question by describing the routes within the MC.

The research was by the Faculty of Education at the University of Haifa and the Israeli Ministry of Education's IRBs.

## Findings

Next, we present the submitted examples, the post-submission report of the second task (see Figures 3 and 4), and students' interactions with the post-submission report.

<figure><img src="image_83.png" /><figcaption>Figure 3: The three examples Ray and Lour submitted</figcaption></figure>

After submitting three examples in the second task (represented in figure 3), Ray and Lour referred to the post-submission report. The post-submission report presented to the students (Figure 4) consisted of two lists that were prepared as part of the task design: (1) Task requirements: a list of critical characteristics that provide information on whether the example is an example of Yael's ride, as described in the task. Each characteristic is marked as V or X according to its existence in the example. (2) Example characteristics: A list of non-critical characteristics that can give students a
sense of the breadth of their personal example space. This list includes the characteristics that the task designers would like to deepen the students' interaction with; therefore, the full list appears beneath each example. After analyzing the submitted examples, STEP highlights the characteristics that exist in each example in yellow. Among these characteristics are the various mathematical attributes of the different graphs and their implications on the real-life situation described in the task.

As shown in the report, the submitted examples met the task requirements of overall time and overall distance. In addition, the following eight characteristics were assessed in each example: Yael started the ride from home, Yael finished riding at home, Yael changed direction at least once, Yael stopped at least once, Yael rode at different speeds, Yael passed through home, Yael started the ride at 8 am, and Yael started the ride before 8 am. According to the automated analysis, three characteristics out of them existed in the three examples: Yael started the ride from home, Yael rode at different speeds, and Yael started the ride at 8 am.

<figure><img src="image_84.png" /><figcaption>Figure 4: the post-submission report Ray and Lour received</figcaption></figure>

Below are some of Ray and Lour's reactions to the post-submission report.

1 Lour: Yael started the ride from home, Yael stopped at least once, Yael rode at different speeds, Yael started the ride at 8 am. [Pointing at the characteristics highlighted in yellow in the list]

2 Ray: Yael finished riding at home. However, they did not tell us. [Pointing at the second characteristic on the list: "Yael finished riding at home," which appeared beneath all examples].

3 Ray: Yael changed direction at least once. [Pointing at the third characteristic in the list: "Yael changed direction at least once," which appeared beneath all examples].

4 Lour: Yael stopped at least once. [Pointing at the fourth characteristic in the list: "Yael stopped at least once," which appears beneath the left example].

5 Lour: Here she stopped. [Pointing at the graph of the left example]

6 Lour: Yael passed through home. Why was this not highlighted? [Pointing at the sixth characteristic in the list: "Yael passed through home," which appears beneath the three examples].

7 Ray: because we did not make her pass.

8 Ray: all answers are the same. We did a stop only in the left example. [Pointing at the fourth characteristic in the list: "Yael stopped at least once," which appears beneath the left example].
9 Ray: If she drove for 5 hours then this will be incorrect? [Asking the teacher while pointing at the first task requirement "overall time meet the task requirement"]

10 Teacher: right. You will get X beside the first task requirement

The following categories describe the students' use of the post-submission report:

## Deepening interaction with the example characteristics

The validation process in mathematical modeling involves a series of critical steps aimed at ensuring that the obtained mathematical results accurately reflect real-world situations (Ferri, 2018). In lines 1,4,5 and 8, the students used the verbal description within the report to validate the characteristics of their submitted examples, even though their answers were correct. The interaction with the report enabled them to go through a control process in which the designed characteristics helped them reflect on their answers within the given situation and think about the differences between the graphs submitted. This is not what we would expect in the case of a "correct" answer and indicates the deepening of the interaction with the transition between a realistic situation and a mathematical situation.

## The need of external validation

Lines 9-10 describe the interactions between Ray, the report, and the teacher. By asking the teacher about the effect of alternative input data on what is presented in the post-submission report, a "what if" situation, Ray ensures that she understood the first critical characteristic (overall time), and its mathematical representation. This interaction indicates that in some cases, external validation, in this case the teacher's mediation, is needed in order to understand the results of the report, which is also attributed to the control step within the validation process.

## Focusing on the task requirements

In line 2, Ray compared the results of the automated analysis with task requirements. She emphasized that the given situation did not require finishing the ride at home; therefore, they did not submit such an example of a graph. Thus, she validated the mathematical results in a given real situation. This might indicate that students prefer to focus on task requirements more than on other mathematical characteristics.

## Unattended characteristics

Ray and Lour did not attend to the non-critical characteristic "Yael started the ride before 8 am" that appears at the end of the list. This might have happened because it was a negation of the characteristic before it, "Yael started the ride at 8 am".

Next, we describe how students' use of the post-submission report affects their MMC by describing the new nonlinear modeling route within the MC.

## New non-linear modeling route

While associating the analyzed interactions (mentioned above) within the modeling process, we noticed that the interaction with the report, as part of the feedback process, enabled the students to perform a new route within the MC. The students, being at the "real results" phase, arrived to the "real situation" phase by proceeding within the MC in the following order: interpreting the real results presented as verbal descriptions in the post-submission report, validating the mathematical
interpretations of the real situation with their submitted examples while using the mathematical model (the graphs), gaining insights to the real situation. For example, in lines 4-5, Lour noticed that the left example represents a situation in which Yael stopped at least once, although the trip description did not require that. Here, she performed a modeling route that included ensuring that the mathematical results obtained accurately reflected the real-world situation by emphasizing what appeared in the mathematical model (by pointing at the graph). Another example is shown in lines 6-7, by pointing at and reading the characteristic "Yael passed through home" Lour attended the real results presented as verbal descriptions. Indeed, this conversation validates the mathematical interpretations of a real situation with the submitted examples while using the mathematical model. In this situation, unlike Lour, Ray recognized that the mentioned characteristic did not exist in their examples, which shows that she made a connection between the mathematical model and the given verbal description. In both examples, the fact that the students made a connection with the given situation can be interpreted as a return to the real situation.

## Discussion and conclusions

The aim of the present study was to examine how an automated post-submission report can serve as a bridging tool between the literacy phenomenon and the given mathematical model. To this end, we explored students' use of post-submission reports while working on a modeling activity and its effect on their modeling process. Contrary to traditional mathematical problem solving, in which students often conclude their process with the acquisition of mathematical results, the validation process in mathematical modeling goes beyond (Ferri, 2018). The competencies required for effective validation involve critical checking and reflection on the solutions found. Technological tools can assist in this process not only by providing interactive simulations but also by enhancing the feedback process in other ways. Our findings illustrate this idea and answer the second research question: the verbal descriptions in the post-submission report helped Ray and Lour validate their answers by reviewing various parts of their solution within the given situation and the given mathematical model (graphs) by critically checking the submitted examples and reflecting on them. By interacting with the post-submission report, a non-linear modeling route was identified: from the real result (Phase 6) to the mathematical model (Phase 4) to the real situation (Phase 1). This new route enables students to interpret and validate mathematical results obtained in the extra-mathematical world. These findings support the idea that artifacts such as personal reports, which are designed as tools by others, may become instruments for learning (Yerushalmy et al., 2023) and contribute to the hypothesis that the post-submission report serves as a bridging tool between the literacy phenomenon and the given mathematical model.

Usually, reports are the last step in the feedback process and do not prompt further meaningful interaction with the task, especially if the answer is correct. In this study, the findings, which answer the first research question, show that the post-submission report is an integrative part of the ongoing feedback process, which enables students to make sense of information relating to the task and regarding aspects of their understanding (Hattie & Timperley, 2007). Indeed, looking at the whole interaction of the students with the task as a holistic feedback process that includes using the simulation and interpreting the results of the post-submission report contributes to the validation process that occurs while engaging with the report. For this purpose, a post-submission report is a
pivotal part of the task. It should be noted that the findings for this study are preliminary, concerning one pair of students, but we will expand the sample to more pairs.

## References

Abu-Raya, K., & Olsher, S. (2021). Learning analytics based formative assessment: Gaining insights through interactive dashboard components in mathematics teaching [Paper presentation]. *AI for Blended-Learning: Empowering Teachers in Real Classrooms Online Workshop, EC-TEL '21*, Bozen -Bolzano, Italy.

Cevikbas, M., Kaiser, G., & Schukajlow, S. (2022). A systematic literature review of the current discussion on mathematical modelling competencies: State-of-the-art developments in conceptualizing, measuring, and fostering. *Educational Studies in Mathematics*, *109*, 205-236.
https://doi.org/10.1007/s10649-021-10104-6

Ferri, R. B. (2018). *Learning how to teach mathematical modeling in school and teacher education*. Springer. https://doi.org/10.1007/978-3-319-68072-9

Greefrath, G. (2011). Using technologies: New possibilities of teaching and learning modelling— Overview. In G. Kaiser, W. Blum, R. Borromeo Ferri, & G. Stillman (Eds.), *Trends in Teaching and Learning of Mathematical Modelling* (ICTMA 14), 301–304. Springer.
https://doi.org/10.1007/978-94-007-0910-2_30

Harel, R., Olsher, S., & Yerushalmy, M. (2022). Personal elaborated feedback design in support of students’ conjecturing processes. *Research in Mathematics Education, 26*(1), 70-89. https://doi.org/10.1080/14794802.2022.2137571

Hattie, J., & Timperley, H. (2007). The power of feedback. *Review of educational research, 77*(1), 81-112. https://doi.org/10.3102/003465430298487

Sadler, D. R. (1989). Formative assessment and the design of instructional systems. *Instructional science, 18*(2), 119-144

Sangwin, C. J., & Köcher, N. (2016). Automation of mathematics examinations. *Computers & Education, 94*, 215-227. https://doi.org/10.1016/j.compedu.2015.11.014

Touma, M., & Olsher, S. (2022). Designing computer-based modeling activities that support students’ modeling process. In J. Hodgen, E. Geraniou, G. Bolondi, & F. Ferretti (Eds.), *Proceedings of the Twelfth Congress of the European Society for Research in Mathematics Education (CERME12)*, 1167-1174.

Yerushalmy, M., Olsher, S., Harel, R., Chazan, D. (2023). Supporting inquiry learning: An intellectual mirror that describes what it “sees.” *Digital Experiences in Mathematics Education, 9*, 315–342. https://doi.org/10.1007/s40751-022-00120-3
# Combining self-assessment and automatic-assessment – a mixed methods study

Carina Tusche<fnref n="1" />, Daniel Thurm<fnref n="1" /> and Shai Olsher<fnref n="2" />

This paper introduces a digital learning environment featuring a novel assessment module that merges self-assessment and automatic-assessment. This integration is particularly notable given the limited research on combining these two assessment forms. The module aims to leverage the reflective nature of self-assessment with the efficiency and objectivity of automatic-assessment, potentially enhancing learning outcomes. The paper also outlines the initial phase of a mixed-methods study designed to evaluate this combined assessment module. This study will provide a comprehensive analysis of the module's effectiveness compared to other assessment methods. Through this research, we aim to offer valuable insights into the benefits and limitations of integrating self-assessment and automatic-assessment in educational settings.

Keywords: Automatic-assessment, self-assessment, technology, linear functions.

## Theoretical background

Previous studies have shown that digital formative assessment can support student learning, for example through automatic-assessment and feedback on students' solutions (Harel et al., 2022; Olsher & Thurm, 2021). Furthermore, self-assessment is considered important for the development of students' metacognitive skills and the promotion of personal responsibility for their own learning process (Andrade, 2019). However, there is a gap in research that addresses the **combination** of automatic-assessment and self-assessment to support mathematical learning processes (Olsher & Thurm, 2021).

Self-assessment can be conceptualized as a process in which students reflect on the quality of their work, evaluate how well it aligns with established goals or criteria and make modifications accordingly (Andrade et al., 2019). Self-assessment can promote students' metacognitive and self-regulatory processes by encouraging them to evaluate, reflect on and revise their work (Panadero et al., 2017). However, it is important to emphasize that self-assessment carries the risk of students drawing incorrect conclusions about their learning process.

Technology offers various ways to support formative assessment and formative self-assessment, for example with interactive tasks and adaptive real-time feedback (Harel et al., 2022; Olsher & Thurm, 2021, Olsher et al., 2016). As Harel et al. (2022) showed, students working on digital “example-eliciting tasks” (tasks in which students construct examples that illustrate/support their answers to a given problem) can be supported by automatic “attribute isolation elaborated feedback” (AIEF) by providing information on whether specific predefined mathematical characteristics are present in their constructed examples. In this context, Olsher and Thurm (2021) suggested that learner engagement can be further enhanced if they self-assess their work in terms of the predefined characteristics before receiving the AIEF.

<footnote n="1">¹University of Siegen, Germany; tusche@mathematik.uni-siegen.de, thurm@mathematik.uni-siegen.de</footnote>

<footnote n="2">²University of Haifa, Israel; olshers@edu.ac.il</footnote>
# Designing an EET addressing the positional relationships of two linear functions

Based on the concept of Olsher and Thurm (2021), we developed a digital learning setting using a GeoGebra applet. The learning setting aims to explore the relationship between the parameters, the number of intersection points, and the positional relationships of two linear functions.

The task requires students to construct three examples with different positional relationships between two linear functions (see Figure 1, left, for one example). To do this, students can move points on the graphs, use sliders or change the parameters on an algebraic level. In the task, students should also formulate a conjecture about how the parameters, the positional relationships and the number of intersection points of the two linear functions relate to each other (Figure 1, box on the right). An example of a students’ conjecture could be: “Both graphs always intersect if one function has a positive and one has a negative slope”.

<figure><img src="image_85.png" /><figcaption>Figure 1: Task and GeoGebra applet</figcaption></figure>

After the students have solved the task (Figure 1), they first evaluate each of their constructed examples and decide which of twelve predefined characteristics are present in their constructed examples (Figure 2, left). If they are unsure whether a characteristic is present, they can indicate this by selecting the question mark. Since the students are supposed to explore the relationship between the parameters, the number of intersection points and the positional relationships, the characteristics were constructed in such a way that each characteristic relates to one of the aspects (parameters, number of intersection points, positional relationships). In addition, we defined a characteristic which is not possible to generate with any example to inspire further reasoning processes. After submitting their work, students receive a report consisting of three parts:

a) an overview of their self-assessment (can no longer be changed),

b) an overview of the automatic-assessment (i.e., overview of which characteristics are present in their examples)

c) a combined overview showing conflicts between the self-assessment and the automatic-assessment (Figure 2, right).

Subsequently students work on the task again to improve their task solution.
<table><caption>Figure 2: Left: Self-assessment with predefined characteristics; right: the combined overview with highlighted conflicts</caption><thead><tr><th>Characteristics of your own example</th><th>Characteristics of your own example</th><th>ⓘ</th><th>ⓘ</th></tr></thead><tbody><tr><td>Both graphs have a positive slope</td><td>Both graphs have a positive slope</td><td>-</td><td>-</td></tr><tr><td>Both graphs have a negative slope</td><td>Both graphs have a negative slope</td><td>-</td><td>-</td></tr><tr><td>Both graphs have exactly the same slope</td><td>Both graphs have exactly the same slope</td><td>-</td><td>-</td></tr><tr><td>Both graphs have the same y-intercept</td><td>Both graphs have the same y-intercept</td><td>-</td><td>-</td></tr><tr><td>Both graphs have a different y-intercept</td><td>Both graphs have a different y-intercept</td><td>++</td><td>-</td></tr><tr><td>Both graphs have exactly one intersection point with each other</td><td>Both graphs have exactly one intersection point with each other</td><td>++</td><td>-</td></tr><tr><td>Both graphs have no intersection point with each other</td><td>Both graphs have no intersection point with each other</td><td>-</td><td>-</td></tr><tr><td>Both graphs have an infinite number of intersection points with each other</td><td>Both graphs have an infinite number of intersection points with each other</td><td>-</td><td>-</td></tr><tr><td>Both graphs have exactly two intersection points with each other</td><td>Both graphs have exactly two intersection points with each other</td><td>+</td><td>-</td></tr><tr><td>Both graphs are parallel</td><td>Both graphs are parallel</td><td>-</td><td>-</td></tr><tr><td>Both graphs intersect in exactly one point with each other</td><td>Both graphs intersect in exactly one point with each other</td><td>++</td><td>-</td></tr><tr><td>Both graphs are identical</td><td>Both graphs are identical</td><td>-</td><td>-</td></tr></tbody></table>

## Study design

The cluster-randomized mixed-methods study is conducted with approximately 300 ninth grade students divided into three different intervention groups. Within the 45-minute intervention each group engages in the following activities twice:

**A) Combination of self- and automatic-assessment:** This group works on the task, then performs a self-assessment with the characteristics (Figure 1, left) and then receives the combined overview highlighting the conflicts between self-assessment and automatic-assessment (Figure 2, right)

**B) Only automatic-assessment:** This group works on the task, does not carry out a self-assessment, but receives a report with the automatic-assessment (i.e. which characteristics are present in the examples)

**C) Only self-assessment:** This group completes the task and then only carries out the self-assessment with the characteristics, without receiving an automatic-assessment.

To ensure that all intervention groups have the same knowledge at the beginning of the intervention, all three groups receive the same 45-minute introductory phase in which technical terms (e.g. intersection) are learned and repeated. In the introductory phase, no explicit connections are made between the parameters of the linear functions, the number of intersection points and the positional relationships, as these are to be explored in the context of the developed task. At the end of the introductory phase, all students complete a short test to check their knowledge.

## Research goal and methods

This study will investigate the extent to which the different interventions affect

i) the metacognitive activities,

ii) the written conjectures,

iii) the variety of generated examples and

iv) the understanding of the relationship between parameters, the number of intersection points and positional relationships of two linear functions.

To reconstruct the metacognitive activities, 4-6 students in each intervention group are filmed, and the recordings are analyzed qualitatively. The written conjectures and the variety of generated examples are reconstructed from the work of the students in the digital learning setting and quantitatively evaluated following the data collection. The understanding of the relationship between
parameters, the number of intersection points and positional relationships is determined by a post-
test.

Outlook

It is not expected that one of the three interventions will show the best results for all outcome
measures i) - iv). By emphasizing the conflicts (between self-assessment and automatic-assessment,
see Figure 2, right), intervention A) could help students to use these conflicts for their own learning
process. However, the high cognitive load of the combination of self-assessment and automatic-
assessment could also be disadvantageous. For example, it may be that the students concentrate more
on the reduction of their conflicts and neglect the revision of their hypotheses. In summary, we expect
detailed insights into the mathematical learning processes in different assessment conditions, which
can help in the design of digital learning settings that integrate automatic-assessment and self-
assessment.

References

Andrade H. L. (2019). A critical review of research on student self-assessment. *Frontiers in Education, 4*, 1-13. https://doi.org/10.3389/feduc.2019.00087

Harel, R., Olsher, S., & Yerushalmy, M. (2022). Personal elaborated feedback design in support of students’ conjecturing processes. *Research in Mathematics Education, 1–20*. https://doi.org/10.1080/14794802.2022.2137571

Olsher, S., & Thurm, D. (2021). The interplay between digital automatic-assessment and self-assessment. In M. Inprasitha, N. Changsri, & N. Boonsena, (Eds.), *Proceedings of the 44th Conference of the International Group for the Psychology of Mathematics Education* (Vol. 3, p. 431–440). PME.

Olsher, S., Yerushalmy, M., & Chazan, D. (2016). How might the use of technology in formative assessment support changes in mathematics teaching?. *For the learning of mathematics*, 36(3), 11–18.

Panadero, E., Jonsson, A., & Botella, J. (2017). Effects of self-assessment on self-regulated learning and self-efficacy: Four meta-analyses. *Educational Research Review, 22*, 74–98.
# Facilitating teachers' in-the-moment feedback practices

Sumeyra Tutuncu

IOE, UCL's Faculty of Education and Society, London, United Kingdom; s.tutuncu@ucl.ac.uk

This theoretical paper draws attention to the need for further research on the resources that can facilitate mathematics teachers' in-the-moment formative assessment practices, with a focus on the feedback aspect. The paper elaborates on the argument that to effectively integrate formative assessment into their daily teaching practices, teachers need resources that clearly communicate its essence. Thus, I will present an overview of the effects of different types of feedback from literature complemented with a critique of a teacher guide.

Keywords: feedback, formative assessment, curriculum resources.

## Introduction

While the key role of formative assessment practices in effective learning has been recognised and acknowledged by policymakers and researchers, it remains a challenge to transfer the essence of formative assessment to teachers' actual in-the-moment practices. This challenge might result from teachers' beliefs regarding the essence of formative assessment as well as a lack of sufficient tools and knowledge (Amado & Morselli, 2023; Antoniou & James, 2014). In that sense, it is important to provide teachers with classroom resources that facilitate a shift in their beliefs as well as provide tools and knowledge.

The five strategies and one big idea of formative assessment proposed by William and Thompson (2007) can be a useful framework to guide teachers. Namely, they draw attention to the key function of formative assessment as teachers' making evidence-informed decisions and highlight the key strategies as: (1) Clarifying, sharing, and understanding learning intentions and success criteria; (2) Eliciting evidence of learning; (3) Providing feedback that moves learning forward; (4) Activating learners as instructional resources for another; (5) Activating learners as owners of their learning.

In this paper, the third strategy is the key focus. After presenting an overview of the various types of feedback and their potential influence on students' learning under various conditions, in order to complement the theoretical discussions, I will critique a teacher guide that provides explicit recommendations for feedback. This paper contributes new insights into the potential role of daily curriculum resources in challenging teachers' ineffective feedback practices.

## Varying impact of different types of feedback on learning in different situations

The term feedback was originally used to refer to the information that is provided to the learner to reduce the gap between the actual learning and intended learning (Ramaprasad, 1983). More recent conceptualisations considered feedback as a process in which students are expected to be engaged with the information to improve their work (e.g., Hattie & Timperley, 2007).

Intriguingly, the research evidence points to unfavourable potential consequences of teachers' feedback on students' learning (Kluger & DeNisi, 1996). Among the studies Kluger and DeNisi reviewed in their comprehensive review, more than one-third of the feedback interventions harmed students' learning mainly due to the attention on student self instead of the task-related feedback.
More recent reviews provided further insights into the conditions that result in positive or negative effects of feedback (Hattie & Timperley, 2007; Shute, 2008; Van der Kleij et al., 2015).

These three reviews provided insights into the impact of different types of feedback on learning. Hattie and Timperley suggested four levels of feedback: whether the task was accomplished (task-level); how to accomplish task or how to improve the intended product (process-level); highlighting students' skills as learner (self-regulation level) and personal comments on students' personality (self-level). Similar to Kluger and DeNisi's findings, Hattie and Timperley's review has taken attention to the harmful effect of feedback that focuses on the self. Further, feedback types can be categorised according to the extent to which the teacher provides information about students' response: telling whether the answer is correct or incorrect; providing the correct answer; and providing additional instructions with elaborated feedback (Shute, 2008). Another common categorisation of feedback types is on timing: feedback can be either provided immediately or it can be delayed (Hattie & Timperley, 2007; Kluger & DeNisi, 1996; Shute, 2008).

Such variety within the feedback practices can suggest a challenge for teachers to decide which type can be helpful in which situation in the classroom. Existing research provides insights into the impact of various feedback types according to students' prior learning or the level of skill the task requires. More explicitly, while immediate feedback can be useful for enabling students' engagement with difficult tasks in the early steps and consolidating procedural or conceptual knowledge, delaying feedback can prevent students' potential distraction when they are working on easy tasks and can be beneficial for long-term learning of high-level skills (Fyfe & Rittle-Johnson, 2016; Shute, 2008).

It should be noted that most of the studies that provide insights into the effect of feedback were conducted in an artificial experimental design environment beyond reflecting on teachers' real classroom feedback practices. Antoniou and James (2014) explored primary school teachers' existing tendencies related to formative assessment practices. The findings of their study suggest that teachers tend to provide immediate corrective feedback and judgment rather than providing feedback that can enhance students' learning. Additionally, they found short-term rewards and the use of grades as common feedback strategies, which may undermine students' intrinsic motivation and hinder their long-term development.

To conclude, the existing literature provides insights into the various impacts of different types of feedback on students' learning. Also, there is evidence that teachers may tend to not pay attention to these different types of feedback but they may provide rewards that can be an example of the feedback for self and immediate corrective feedback, which may not ultimately support long-term and high-level learning. One way of facilitating teachers' change can be the development of teachers' daily resources in which effective feedback practices are integrated.

## A critique of a teacher guide that involves explicit suggestions for feedback

In the previous section, I provided an overview of the types of feedback and their potential impact on learning as well as highlighting teachers' need for daily use resources that can facilitate their feedback practices. In this section, employing the insights gained from this literature, and drawing on the analytical strategies developed earlier (Tutuncu et al., 2023), I will critique a teacher guide for its integrating of specific guidance for teachers' practices. This teacher guide is from a set of materials on the Mathematics Assessment Project (MAP). This teacher guide was chosen as it provides rich
insights and tools for teachers for formative assessment practices in general and also involves specific guidance for feedback practices. However, despite its strong focus on providing guidance and tools for teachers, the provision of guidance and tools for feedback is limited to only two aspects. In the following paragraphs, these examples are elaborated.

First, in the teacher guide, teachers were provided with guidance for process-level and immediate feedback when students struggled to start working on an open answer and open-method task, which could enable students to understand what was required in the task. Teachers were provided with detailed guidance in terms of the way of providing this process-level and immediate feedback. Importantly, they were advised to use questioning rather than directly telling them what to do, such as the following example questions.

> What useful information are you given? Underline this. What do you need to find out? How can you use the information you know to do this? (MAP, Drawing to Scale: A garden, page T-4)

This immediate feedback that was provided in the initial steps of students' work could facilitate students' engagement with the rest of the task as Shute (2008) suggested. In the literature, one limitation of immediate feedback is suggested as hindering high-level learning. The suggested immediate feedback in this teacher guide could mitigate this risk. That is to say, using prompts that assist students to realise what procedures they should apply and connect this understanding with their existing knowledge might support students' strategic competence and adaptive reasoning beyond procedural fluency and conceptual understanding (Kilpatrick et al., 2001).

Second, teachers were recommended against providing evaluative feedback. The rationale for this advice was explained with students' potential distraction by comparing their scores with their peers' scores rather than focusing on how to improve their learning. As an alternative to scoring, teachers were encouraged to use questioning to enable students to reflect on their work. As an example, when teachers observed that students have difficulty in making calculations with decimals, they were advised to use the following questions as feedback.

> How do you convert 1m in real life to a measurement on the plan? What about 3m? Now apply the same method to figure out what the length 3.25 m is on the plan. (MAP, Drawing to Scale: A garden, page T-4)

These questions can be more beneficial than traditional immediate-corrective or immediate-elaborated feedback. That is to say, rather than correcting students or re-teaching the topic, students are encouraged to reflect on their work by using numbers with which the calculations can be relatively straightforward.

## Conclusion

This paper draws attention to the need for teachers' daily resources that have the potential to facilitate teachers' effective in-the-moment feedback practices. The existing literature offers insights into the impacts of different types of feedback on students' learning. Although studies exclusively examining teachers' daily feedback practices without any form of intervention are constrained in their scope and potential insights, there is evidence that teachers tend to provide immediate feedback which is corrective or judgmental rather than constructive forms of feedback. In this paper, I present an
example teacher guide that has the potential to facilitate mathematics teachers' more effective feedback practices, drawing attention to the limited existing resources that can have this potential.

## References

Amado, N., & Morselli, F. (2023). Teachers' beliefs about assessment: A study in Italy and Portugal. In Drijvers, P., Csapodi, C., Palmér, H., Gosztonyi, K., & Kónya, E. (Eds.). *Proceedings of the Thirteenth Congress of the European Society for Research in Mathematics Education (CERME13)* (pp. 3916–3923). Alfréd Rényi Institute of Mathematics and ERME, Budapest, Hungary.

Antoniou, P., & James, M. (2014). Exploring formative assessment in primary school classrooms: Developing a framework of actions and strategies. *Educational Assessment, Evaluation and Accountability, 26*(2), 153–176. https://doi.org/10.1007/s11092-013-9188-4

Fyfe, E. R., & Rittle-Johnson, B. (2016). Feedback both helps and hinders learning: The causal role of prior knowledge. *Journal of Educational Psychology, 108*(1), 82–97. https://doi.org/10.1037/edu0000053

Hattie, J., & Timperley, H. (2007). The power of feedback. *Review of Educational Research, 77*(1), 81–112. https://doi.org/10.3102/003465430298487

Kilpatrick, J., Swafford, J., & Findell, B. (2001). *Adding It Up: Helping Children Learn Mathematics*. Washington, DC, USA: National Academies Press.

Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. *Psychological Bulletin, 119*(2), 254–284. https://doi.org/10.1037/0033-2909.119.2.254

Ramaprasad, A. (1983). On the definition of feedback. *Behavioral Science, 28*(1), 4–13. https://doi.org/10.1002/bs.3830280103

Shute, V. J. (2008). Focus on formative feedback. *Review of Educational Research, 78*(1), 153–189. https://doi.org/10.3102/0034654307313795

Tutuncu, S., Hodgen, J. & Golding, J. (2023). Proposing educative features of curriculum materials that can enhance teachers’ noticing. In Drijvers, P., Csapodi, C., Palmér, H., Gosztonyi, K., & Kónya, E. (Eds.). *Proceedings of the Thirteenth Congress of the European Society for Research in Mathematics Education (CERME13)* (pp. 4205–4212). Alfréd Rényi Institute of Mathematics and ERME, Budapest, Hungary

Van der Kleij, F., Feskens, R. C. W., & Eggen, T. J. H. M. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes: A Meta-analysis. *Review of Educational Research, 85*(4), 475–511. https://doi.org/10.3102/0034654314564881

Wiliam, D., & Thompson, M. (2007). Integrating assessment with learning: What will it take to make it work? In C. A. Dwyer (Ed.), *The Future of Assessment* (pp. 53–82). Mahwah, NJ, USA: Lawrence Erlbaum Associates.
# Students' experiences with automated final answer diagnoses for mathematics tasks

Gerben van der Hoek¹, Bastiaan Heeren², Rogier Bos¹, Paul Drijvers¹ and Johan Jeuring¹

¹Utrecht University, the Netherlands; g.vanderhoek@uu.nl

²Open University, the Netherlands

Model Backtracking (MBT) is a novel technique for automated detailed diagnoses based on final answers. In this small-scale pilot study, we answer research questions about nine 15-to-17-year-old senior general secondary students' experiences with MBT diagnoses. The students practiced linear extrapolation in a learning environment that provides error-specific feedback and selects appropriate subtasks using MBT. Data included screen captures of students navigating the environment and interviews on students' experiences with the environment. Results showed approaches ranging from correcting an error after receiving feedback to trial-and-error behavior while repeatedly consulting the worked-out solution. Furthermore, students preferred error-specific feedback over worked-out solutions. They found that worked-out solutions provide an insightful overview; yet, errors are not pinpointed, and worked-out solutions reduced motivation for further practice.

Keywords: Automated assessment, feedback, linear extrapolation, model backtracking.

## Introduction

The use of automated formative assessment in mathematics education is rapidly increasing. Various software tools are available to give students feedback on mathematical calculations and underlying strategies. Many of these tools either provide feedback that is non-specific to the student error or require all calculation steps as input. Inputting every step of a calculation can be cumbersome for a learner (Drijvers, 2019). Moreover, software should allow a student to authentically do mathematics, without unintended effects caused by the environment's interface (Kieran and Drijvers, 2006). To provide error-specific feedback, the interface for calculations that consist of several nonequivalent steps (e.g., linear extrapolation) often contains an input box for each step. However, in this way, a calculation is pre-structured in the interface and a student's reasoning might be unintentionally scaffolded by this structure. This unintended effect can be avoided by using final answer diagnoses through Model Backtracking.

Model Backtracking (MBT) (van der Hoek, 2020; van der Hoek et al., 2023) is a technique that uses the strategy language (Heeren et al., 2010) to provide a detailed diagnosis of an entire student calculation based on a final answer. MBT itself is comprised of several techniques that serve two purposes. The first purpose is to mitigate the combinatorial explosion associated with calculating all possible final answers, given a set of possible student errors. The second purpose is to increase the accuracy of final answer diagnoses by selecting specific tasks. In MBT, the starting parameters for a task are selected such that the number of different ways to reach a final answer is minimized.

MBT allows for providing error-specific feedback based on a final answer. This might help students in postponing viewing worked-out solutions to tasks they should learn from by solving them. Students tend to alleviate their feelings of uncertainty (Shute, 2008) by viewing worked examples. However, viewing these work examples might prevent students from developing problem-solving skills
(Goodman and Wood, 2004). This paper reports on a small-scale qualitative pilot study on how senior general secondary 15 to 17-year-old students experience working with error-specific feedback in our online environment. To investigate these experiences, we formulated two research questions:

1. How did the students use the error-specific feedback and the worked-out solutions provided by the environment?

2. How did the students describe experiences their with error-specific feedback as opposed to worked-out solutions?

In the next section, we identify a theoretical framework suitable for answering these questions.

## Theoretical framework

Online learning environments offer functions that are grounded in the following pedagogical approaches: learning from feedback, learning from worked examples, and learning from tasks. Here, we explore these approaches further. We start by listing various feedback types relevant to our research. Shute (2008) presents several feedback types, such as verification, try-again, and elaborated feedback. Below we further elaborate on these types. Verification feedback provides learners with knowledge about the correctness of a response, which is often referred to as knowledge of results (KR). Try-again feedback (TA) allows learners to provide a new response after some other type of feedback is provided. As for elaborated feedback, Shute distinguishes several variants, two of which are of interest here: Topic-contingent feedback and feedback on bugs. The former is feedback about the topic that is being studied, which could be a worked example (WE) of a task; the latter is error-specific feedback (ES), which is based on a diagnosis of the learner's response.

Feedback can benefit learners in two ways: it can resolve uncertainty and it can alleviate cognitive load. We discuss uncertainty and cognitive load in more detail since these are important aspects of providing feedback. Uncertainty is an unpleasant state that may distract learners from task performance; hence, they wish to avoid or resolve it (Bordia et al., 2004). Therefore, providing feedback can increase performance by alleviating this uncertainty. Cognitive load is introduced when task execution floods learners' working memory. Cognitive load can be decreased by using worked examples. For instance, Sweller et al. (1998) showed that worked examples reduced the cognitive load for low-ability students in problem-solving tasks.

Learning from problem-solving activities as opposed to learning from worked examples has been studied in the late eighties and nineties (Chi et al., 1989, Renkl 1997). These studies generally favored learning from worked examples; because the cognitive load that problem-solving activities introduce can cloud the actual learning process. However, Chi et al. also reported that positive learning outcomes using worked examples strongly depend on a student's ability to self-explain the steps in the worked example. Furthermore, Goodman and Wood (2004) found that very specific feedback can be detrimental to long-term performance, as it may prevent learners from developing problem-solving skills.

In conclusion: when learners perform a task, they can experience an adverse state of cognitive load and uncertainty. As such, they wish to alleviate this state by seeking feedback. Therefore, feedback is a useful tool to help the learning process. Nonetheless, caution is warranted for its use; because
worked examples only benefit students who possess self-explaining skills, and feedback that is too specific can hinder self-development.

## Methods

The methods section consists of two parts. We first elaborate on the design of our online environment. After that, we proceed to explain the methods used to conduct the experiment involving the students.

### Environment design

The environment consists of a website together with an MBT script that calculates feedback or suggests a subtask based on a student's input. The environment offers a main task and three subtasks. All tasks have random starting values based on 50 pre-calculated integer task parameters that ensure high diagnosis accuracy. This allows a student to retry a task with different starting values after KR, ES, or WE feedback.

#### Main task format:

Given the table with values for $x_1, x_2, y_1, y_2$ and $x_v$

<table><tr><td>x</td><td>x<sub>1</sub></td><td>x<sub>2</sub></td><td>x<sub>v</sub></td></tr><tr><td>y</td><td>y<sub>1</sub></td><td>y<sub>2</sub></td><td>y<sub>v</sub></td></tr></table>

Q: Use linear extrapolation to compute $y_v$.

#### Subtask A: Simpler numbers.

Given the table with values for $x_1, y_1, y_2$ and $x_v$

<table><tr><td>x</td><td>x<sub>1</sub></td><td>x<sub>1</sub> + 1</td><td>x<sub>v</sub></td></tr><tr><td>y</td><td>y<sub>1</sub></td><td>y<sub>2</sub></td><td>y<sub>v</sub></td></tr></table>

Q: Use linear extrapolation to compute $y_v$, first determine the change in y when x increases with 1.

#### Subtask B: Known slope.

Given the table with values for $x_1, y_1$ and $x_v$ and known slope a

<table><tr><td>x</td><td>x<sub>1</sub></td><td>x<sub>v</sub></td></tr><tr><td>y</td><td>y<sub>1</sub></td><td>y<sub>v</sub></td></tr></table>

Q: Suppose the slope is a, use linear extrapolation to compute $y_v$.

#### Subtask C: Calculate slope.

Given the table with values for $x_1, x_2, y_1$ and $y_2$

<table><tr><td>x</td><td>x<sub>1</sub></td><td>x<sub>2</sub></td></tr><tr><td>y</td><td>y<sub>1</sub></td><td>y<sub>2</sub></td></tr></table>

Q: Calculate the slope.
The feedback in the environment is designed following findings in the literature. The system provides KR feedback, and when a diagnosis of a student error is possible it provides ES feedback. The student then has the option to try again to obtain TA feedback. The ES feedback in the main tasks has low specificity (i.e., verbally formulated suggestions), whereas the ES feedback in the subtasks has higher specificity (i.e., suggestions that may contain calculations). WE feedback, a worked-out solution, is available; but we postpone the worked-out solution until after a student has selected a subtask. A student, however, is at liberty to immediately select a subtask and view the worked-out solution. Once a student returns to the main task WE feedback will be available for the main task.

<figure><img src="image_86.png" /><figcaption>Figure 1: Example of ES feedback during a task</figcaption></figure>

Figure 1 shows ES feedback a student receives when inversely computing the slope. To detect the various (combinations of) student errors, so-called buggy rules are implemented in the system. These buggy rules represent erroneous steps in a student's calculation. Over 24 rules are implemented. The rules are based on empirical findings by Van der Hoek et al. (2023) and work by Van Dooren et al. (2005) on the unwarranted use of proportional models in missing value problems. The rules include, for instance, using a proportional model (i.e., calculating: $y_v = y_1/x_1 \cdot x_v$), inversion of the slope (i.e., using: $\Delta x/\Delta y$), and (in)correct intermediate rounding of the slope. Subsets of the buggy rules for the main task were used for the subtasks.

In the environment, a student has several options presented by buttons: (1) go to a subtask that is selected by the system based on a diagnosis, (2) retry the current task with different starting values, and (3) view a worked-out solution (when available). The subtask is selected by the MBT system, through the following arrangement: Subtask A is selected in case no error could be detected, a student
has provided no input yet, or a student calculates: $y_v = y_2 + (y_2 - y_1)$. Subtask B is selected when a student correctly calculated the slope (rounding errors are allowed) but made a detectable error elsewhere. Subtask C is selected when a student detectably calculates the slope incorrectly.

To view how a student navigated the environment, see https://youtu.be/YYRkew5-EEI for an example replayed at 10 times the normal speed. In the next subsection, we further explain how the experiment in which students use the environment was set up.

## Data and data analysis

For this qualitative experiment, a convenient sample of eight senior general secondary students from 10th grade and one student from 11th grade were recruited from four different classes in the school in the Netherlands where the first author is employed. Participation was based on availability and consent to partake. The students had received prior education on linear extrapolation as part of their standard curriculum, but not within four weeks before the experiment.

The students were invited to complete the main task in the environment in a session ranging between 10 and 30 minutes depending on the student. Screen captures along with audio recordings were used to document the students using the environment. Pen, paper, and an onscreen graphic calculator were available to the students. A researcher supported the students in case of confusion on how to operate the system, but not in case of confusion on the task. After the session with the environment ended, the researcher conducted a semi-scripted interview to determine the experiences of the students with the environment.

The screen captures of the sessions with the environment were transcribed into chronological accounts of the events. These accounts were then ordered according to the amount of help the student required from the system. From this ordering five different approaches to using the error-specific feedback and the worked-out solutions emerged. General descriptions of the approaches are formulated and summarized in Table 1. The students' utterances in the interviews that showed the experiences of the students with worked-out solutions and error-specific feedback were transcribed. These utterances were grouped by similarity and coded with a common theme.

## Results

In Table 1 we find descriptions of students' approaches to using the error-specific feedback and the worked-out solutions. The *help level* ranges from requiring no help at all from the system (0) to requiring much help (4). KR feedback provides knowledge of results, ES feedback is error-specific feedback, WE feedback consists of a worked example and TA feedback allows for retries after KR, WE, or ES feedback.
<table><caption>Table 1: Various usages of help in the environment</caption><thead><tr><td>Help level</td><td>Description of help usage</td><td>Frequency</td></tr></thead><tbody><tr><td>0</td><td>The student correctly completed the main task and received KR feedback</td><td>1</td></tr><tr><td>1</td><td>The student corrected an error after ES feedback</td><td>2</td></tr><tr><td>2</td><td>The student solved the tasks by using ES feedback, only using WE feedback when ES feedback was unclear</td><td>2</td></tr><tr><td>3</td><td>The student solved the tasks by studying WE feedback and correcting errors with ES feedback</td><td>3</td></tr><tr><td>4</td><td>The student alternated between TA feedback and WE feedback</td><td>1</td></tr></tbody></table>

Using Table 1 we can divide students into two groups, Group A with a help level of 2 or less, and Group B with a help level of 3 or more. The difference between these groups is the use of worked-out solutions. Students in Group A seldom used a worked-out solution and if they did it was because feedback from the system was unclear to them. In contrast, students in Group B used a worked-out solution as a worked example in the sense of Chi et al. (1989). Of these students, three students immediately viewed the worked-out solution of a subtask without trying the subtask. Furthermore, in Group B we also found the student with help level 4, this student exhibited a trial-and-error-like behavior almost frantically switching between retrying a task and viewing the worked-out solution. The difference between group A and group B can perhaps be explained by the level of uncertainty (Bordia et al., 2004) the students experienced that could have been caused by a lack of sufficient prior knowledge.

Aside from students' approaches to working in the environment, we also investigated students' experiences with worked-out solutions and error-specific feedback through interviews. Table 2 summarizes the utterances during the interviews that were conducted after interacting with the environment.

<table><caption>Table 2: Utterances in the interviews</caption><thead><tr><td>Utterance</td><td>Frequency</td></tr></thead><tbody><tr><td>Error-specific feedback helps to identify the error</td><td>7</td></tr><tr><td>An error is not pinpointed in a worked-out solution</td><td>4</td></tr><tr><td>A worked-out solution reduces motivation for further practice</td><td>3</td></tr><tr><td>A worked-out solution provides an insightful overview of the task</td><td>3</td></tr></tbody></table>

For worked-out solutions, we have two negative utterances with a total frequency of 7 (the second and the third from the top) and we have one positive utterance with a frequency of 3 (the fourth). For error-specific feedback, we have only a positive utterance with frequency 7 (the first). If one were to summarize these results, one could conclude that in this group error-specific feedback is preferred over worked-out solutions.
## Conclusion

How did our sample of senior general secondary students experience practicing linear extrapolation in our MBT-driven learning environment? First, we consider the research question on the use of error-specific feedback and worked-out solutions. We found that some students studied the worked-out solutions instead of only using worked-out solutions to check their answers. Studying worked-out solutions can lead to memorizing them without proper self-explanation (Chi et al., 1989). This effect can be reduced by offering error-specific feedback and postponing worked-out solutions.

Three of the nine students in our sample immediately viewed the worked-out solution of a subtask without trying the subtask. Perhaps this can be explained by the students' uncertainty, caused by a lack of sufficient prior knowledge of the tasks. If so, this can be remedied by providing the opportunity for direct instruction, possibly by incorporating an instruction video in the environment. A combination of direct instruction and inquiry is an effective way of learning (De Jong et al., 2023).

Next, we consider the research question on students' experiences with ES (error-specific) feedback as opposed to WE feedback (worked-out solutions). Overall, students had a positive experience with the ES feedback provided by the environment. They found it helped them pinpoint errors in their calculations whereas WE feedback did not. Furthermore, they found that WE feedback provided a clear overview of the task. However, WE feedback reduced motivation for further practice with similar tasks, since there is not much left to explore after studying the worked-out solution. This provides an argument for offering ES feedback and postponing WE feedback.

Now we reflect on the validity of any claims we made. We have a very small sample from a specific group of students, which means that we cannot generalize beyond statements about the existence of certain behaviors or opinions of students. Even the explanations for the various phenomena offered in this section and the previous sections are at best hypotheses. Since these explanations are given *after* the phenomenon was observed, they hold little to no bearing on any claims of cause and effect. Then, what have we gained by this endeavor? We have gained two things, firstly we have gained *leads* for further study and secondly, we have gained *leads* for improvement of the environment.

To summarize, the experiences of students with the MBT environment were overall positive and eight out of the nine students completed the main task using the information provided by the system. Further development of such MBT-driven systems might contribute to improving learning processes. This future research could show that senior general secondary students, with low self-explanation skills, can benefit from practicing procedures such as extrapolation with the aid of error-specific feedback provided by MBT.

## References

* Bordia, P., Hobman, E., Jones, E., Gallois, C., & Callan, V. J. (2004). Uncertainty during organizational change: Types, consequences, and management strategies. *Journal of Business and Psychology, 18*, 507–532.

* Chi, M. T., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. *Cognitive science, 13*(2), 145–182.
De Jong, T., Lazonder, A. W., Chinn, C. A., Fischer, F., Gobert, J., Hmelo-Silver, C. E., ... & Zacharia, Z. C. (2023). Let's talk evidence-The case for combining inquiry-based and direct instruction. *Educational Research Review*, 100536. https://doi.org/10.1016/j.edurev.2023.100536

Drijvers, P. (2019). Digital assessment of mathematics: Opportunities, issues and criteria. *Mesure et Évaluation en Éducation, 41*(1), 41–66. https://doi.org/10.7202/1055896ar

Fong, C. J., Patall, E. A., Vasquez, A. C., & Stautberg, S. (2019). A meta-analysis of negative feedback on intrinsic motivation. *Educational Psychology Review, 31*, 121–162.

Goodman, J. S., & Wood, R. E. (2004). Feedback specificity, learning opportunities, and learning. *Journal of Applied Psychology, 89*(5), 809.

Heeren, B., Jeuring, J., & Gerdes, A. (2010). Specifying rewrite strategies for interactive exercises. *Mathematics in Computer Science, 3*(3), 349–370. https://doi.org/10.1007/s11786-010-0027-4

Kieran, C., & Drijvers, P. (2006). The co-emergence of machine techniques, paper-and-pencil techniques, and theoretical reflection: A study of cas use in secondary school algebra. *International Journal of Computers for Mathematical Learning, 11*(2), 205–251. https://doi.org/10.1007/s10758-006-0006-7

Renkl, A. (1997). Learning from worked-out examples: A study on individual differences. *Cognitive science, 21*(1), 1–29.

Sweller, J., Van Merriënboer, J., & Paas, F. (1998). Cognitive architecture and instructional design. *Educational Psychology Review, 10*, 251–296.

Tall, D. (2013). *How Humans Learn to Think Mathematically: Exploring the Three Worlds of Mathematics* (Learning in Doing: Social, Cognitive and Computational Perspectives). Cambridge University Press. https://doi.org/10.1017/CBO9781139565202

Van Dooren, W., De Bock, D., Hessels, A., Janssens, D., & Verschaffel, L. (2005). Not everything is proportional: Effects of age and problem type on propensities for overgeneralization. *Cognition and Instruction, 23*(1), 57–86. https://doi.org/10.1207/s1532690xci2301_3

Van der Hoek, G., (2022). Evaluating digital student work through model backtracking. In J. Hodgen, E. Geraniou, G. Bolondi, & F. Ferretti (Eds.), *Proceedings of the Twelfth Congress of the European Society for Research in Mathematics Education (CERME12)* (pp. 2873–2880). Free University of Bozen-Bolzano and ERME.

Van der Hoek, G., Heeren, B., Bos, R., Drijvers, P., & Jeuring, J. (2023). *Model backtracking: Designing and testing an approach to automated diagnosis of students’ work through final answers* [Manuscript submitted for publication].
# GeoGebra pop-up notifications as formative feedback for learning computational thinking in mathematics lessons

Wahid Yunianto<fnref n="1" />, Theodosia Prodromou<fnref n="2" /> and Zsolt Lavicza<fnref n="3" />

Computational thinking (CT) is a fundamental skill for everyone that is relevant to 21st-century skills. Initiatives have been carried out to enhance CT outside of computer science (CS) courses. In this study, we integrated CT into mathematics lessons utilizing GeoGebra. Our lessons' development was guided by the educational design research approach (EDR). When we piloted our lessons with junior high school students, we found that GeoGebra could provide pop-up notifications as formative feedback to support students' debugging skills. Pop-up notifications could act as a negotiated-style interruption or an immediate-style interruption. The first interruption is when the students get a pending message about the errors so that they become aware of them. Meanwhile, the latter informs students to revise the errors immediately. In this paper, we will discuss how these interruptions as a formative assessment could be a means of support for students while learning CT+Maths lessons.

Keywords: Computational thinking, mathematics, formative feedback, assessment, GeoGebra.

## Introduction

The purpose of this study is to explore formative feedback that can be supported by GeoGebra while learning computational thinking in mathematics lessons. Papert (1980) introduced computational thinking (CT) through computer programming within mathematics contents. His idea was to build students' CT skills and mathematical knowledge when interacting with the LOGO turtle. A program was developed by him and his colleagues. Students could input the commands and observe what happened to them. For instance, to make a triangle, students could type in: forward 50, right 120, forward 50, right 120, forward 50, and right 120. It would produce an isosceles triangle with a length of 50 for each vertex. The program was intended to be intuitive, and students could develop CT skills as well as mathematics concepts, which were considered too formal for young learners.

Since LOGO Turtle and its similar programs were introduced, schools in the US have started to initiate programming activities. However, Resnick (2009) found that this initiative was not sustainable due to some challenges, such as students finding it difficult with the textual commands (syntaxes). Later, he and his colleagues developed what we now call block programming, 'Scratch' to smoothen how students learn programming. Nowadays, Scratch is used by users around the world. Additionally, in order to enhance CT in mathematics learning, visual programming (Scratch) and mathematics software (spreadsheets, GeoGebra, and MATLAB) have been utilized (Ye et al., 2023). However, visual programming tends to provide little space for debugging as it was not designed to do so (Liu et al., 2017), and block programming prevents syntax errors from happening (Resnick, 2012). Therefore, our lessons utilized GeoGebra, which has a 'pop-up' feature to facilitate debugging when command errors are inputted.

<footnote n="1">¹Johannes Kepler University Linz, School of Education, Linz, Austria; yunianto.wah@gmail.com</footnote>

<footnote n="2">²University of New England, Australia; theodosia.prodromou@une.edu.au</footnote>

<footnote n="3">³Johannes Kepler University Linz, School of Education, Linz, Austria; zsolt.lavicza@jku.at</footnote>
Ukkonen (2023) asserted that integrating CT in mathematics education is new, and to ensure successful integration, we need to consider formative assessment. Instead of the formative feedback provided by the teachers, we consider formative feedback from the tool students use (GeoGebra) that aligns with the idea of constructionism by Papert (1980) in which students build up their knowledge through their interaction with computer programs. We would like to present our experience when utilizing GeoGebra for formative feedback while learning CT in mathematics lessons. Therefore, we would like to find out how GeoGebra's pop-up could support students in accomplishing debugging tasks, which is a form of formative assessment.

## Conceptual Framework

We commonly have two types of assessment, namely formative (assessment for learning) and summative (assessment of learning) (Tan, 2011). The first aims to support students to reach their learning goals through feedback during the learning process and enhance students' learning (Black & William, 1998). For instance, when a student had a misconception about adding a decimal number with an integer by neglecting the decimal separator, the teacher could assist students with questions and/or feedback to deal with the misconception. Meanwhile, the second is conducted at the end of the topic, lesson, term, or semester to know what students have gained from their learning. In this paper, we are interested in formative assessment, which mainly focuses on formative feedback.

## Formative feedback

Cizek et al. (2019) defined formative feedback as teachers' and students' support to infer strengths, weaknesses, and opportunities for improvements in learning. They stated that with formative assessment, students could deepen their understanding, improve their achievement, take responsibility for their learning, and self-regulate their learning. However, Burkhardt and Schoenfeld (2019) argued that it is unlikely that teachers could provide formative assessments for each student during a class teaching period because of time limitations.

As formative assessment could be in the form of feedback, Hattie and Timperley (2007) explained how the power of feedback could benefit learning. Feedback is information that leads students to confirm, add to, overwrite, tune, or restructure it in memory (Winnie & Butler, 1994 as cited in Hattie & Timperley, 2007). Hattie and Timperley (2007) described three directions and four levels of feedback. For the directions, they classified them into (1) feedback about where a student is (FeedBack), (2) where a student is going (FeedUp) and (3) where a student will go next (FeedForward). The following model (Figure 1) by (Hattie & Timperley, 2007, p. 87) is used to help us consider the feedback provided by GeoGebra pop-up.
<figure><img src="image_87.png" /><figcaption>Figure 1: A model of feedback to enhance learning (Hattie & Timperley, 2007, p. 87)</figcaption></figure>

Ukkonen (2023) used this model for analyzing a case study with a teacher and two students when learning computational thinking in a mathematics lesson. In this paper, we will utilise this model to specify to which parts the GeoGebra pop-ups belong to the model, discuss, and analyse how the GeoGebra pop-ups have helped students in our study.

## Technology and Formative Assessment

Digital technology can gather and process data on a large scale and speed, making it a powerful resource to support teachers in formative assessment (Looney, 2010). Dalby and Swan (2019) argued that digital technology could support and empower students in formative processes by providing feedback. Additionally, the feedback could act as a replacement for or an addition to teacher-led processes (Dalby & Swan, 2019). From their study, we could learn that technology could help process students’ responses quickly and provide feedback quickly so that students can modify their responses.

<figure><img src="image_88.png" /><figcaption>Figure 2: A model of feedback to enhance learning (Dalby & Swan, 2019, p. 841).</figcaption></figure>

The AuthoMath project (https://www.authomath.org/) utilizing STACK and GeoGebra is relevant to our study. Using STACK and GeoGebra for formative assessment has helped students to be more successful with accomplishing the tasks (Sanz-Ruiz et al., 2023). With this digital technology, students would receive feedback if they made mistakes by pressing a check button. The difference from our study is that GeoGebra will automatically notify students when incorrect commands are inputted without pressing the check button. Our previous study, Yunianto et al. (2023) utilized this GeoGebra feature (pop-up notifications), and it benefited students. GeoGebra's pop-up is related to
two styles of interruptions, namely, the immediate style of interruption and the negotiated style of interruption. Robertson et al. (2004) investigated these interruptions when students were debugging and found that negotiated-style interruption is better than immediate-style interruption when supporting debugging activities.

## Method

This study explored the potential of the GeoGebra pop-up as formative feedback when learning computational thinking (CT) in mathematics lessons. It investigated how GeoGebra pop-up could act as formative feedback to benefit students when accomplishing Math+CT tasks. The analysis of our data about how this feature has helped us in our study is qualitative descriptive research as it provides a comprehensive summary of data describing the primary properties of the participants' actions, words, and experiences during data collection and then interpreting it (Ayton, 2023).

## Context

We developed GeoGebra-based Math+CT lessons on the topic of the area of a circle (Yunianto et al., 2023) guided by an educational design research (EDR) approach (McKenney & Reeves, 2018). More details of the development of GeoGebra-based Math+CT lessons can be found in our previous study. In the lessons, students learned how to construct an inscribed regular polygon on a circle with GeoGebra commands. Students also had to debug a program to create a circle and an inscribed polygon. The participants of this study were seventeen junior high school students (aged 12-15) from five different schools in Indonesia.

## Data collection and analysis

We collected data by recording students' screens while they were engaging in the tasks. Students were well informed by their teachers about this research and consented to participate in this study. Not all videos recorded full lessons due to the limited storage capacity of the devices and technical issues. Some videos captured the voices of students and a teacher. For this paper, we could not provide the analysis of all videos but rather a few videos that depict the potential of formative feedback. We transcribed students' discussions with themselves, their peers, and the teachers and then provided the interpretation of the transcripts. The analysis was one of progressive focussing (Robson, 1993). At the first stage, the recordings were simply transcribed, and screenshots were incorporated as necessary to make sense of the transcription. Subsequently, the first author turned the transcript into a plain account and an interpretative account was written and the validity of those interpretation was discussed with other authors.

## Findings

The following paragraphs present the work of a student who worked on constructing a regular polygon with a slider. The polygon vertices could change as the slider was moved (from 3 to 10). Therefore, this construction would produce an equilateral triangle, a square, a pentagon, a hexagon, a heptagon, an octagon, a nonagon, and a decagon respectively to the slider. This student inputted an incorrect command for the slider 'n-Slider(3,10,1)' and GeoGebra notified her with a pop-up. It said, "Create slider(s) for n". She then edited the command to be n=Slider(3,10,1) and continued to input the next command. Finally, she accomplished this task by creating a regular polygon with a slider.
<figure><img src="image_89.png" /><figcaption>Figure 3: A pop-up notification for an incorrect slider command</figcaption></figure>

While doing this task, this student talked to herself immediately after the pop-up notification appeared. The self-talk of this student is as follows:

Student A: Why is it incorrect? (the student was thinking for a while)

Student A: (self reply) Oh this one, the equal sign.

From the transcript, we can see that this student reflected on her command after seeing the pop-up notification and thought for a while what was wrong. Her self-reflection led her to the correct answer.

On another task, she also benefitted from the notification. She was working on a task and inputted all correct commands but missing a (the angle). The pop-up (Figure 4) notified the error (*Please check your input, undefined variable B*'). The student misinterpreted the notification first and she edited the polygon, thinking it was the source of error. Afterwards, she discussed with her peer, and they reflected to the previous experiences of similar tasks. Based on their prior knowledge and peer discussion, she realized that she missed the a. Hence, she inserted a (a=360deg/n). In the end, she had 3 runs and 0 clear and accomplished the task.

<figure><img src="image_90.png" /><figcaption>Figure 4: A pop-up notifying an error</figcaption></figure>

The following excerpts present another case when a student discussed with his teacher when he encountered the pop-ups. He inputted commands for point A and point B correctly. He typed in n=Slinder(3,10,1), with the additional letter n. A pop-up appeared notifying the "Unknown command: Slinder" (Figure 5). He edited it and pressed enter, then he deleted it.

Student B: Why is it incorrect?

Teacher: That is there to make a slider.

Student B: I saw it from the hint!

Teacher: Yes, you can see it from the hint

Student B: I made it sir!

Teacher: Are you sure? Where is the slider?

Student B: Look at this first!(pointing to the slider on screen)
<figure><img src="image_91.png" /><figcaption>Figure 5: A pop-up notification as an immediate-style interruption</figcaption></figure>

The above excerpts show how the GeoGebra pop-up notification provided immediate feedback to the student, so he looked at the hint to receive guidance about how to create a slider.

In another case, we witnessed a negotiated style of interruption where an incorrect point B command $B=(2:2)$ was inputted but did not provide a pop-up because it could be run as a slider. This notification was delayed until it was used for the polygon's command. It would notify it as incorrect when it is used for the polygon command as B was not a point (illegal argument Number B) (Figure 6). It helped students to locate the error so that they could go directly and fix the command that caused the error. Besides it directed students to the error, it suggested them how to use correct commands and variables when programming geometrical objects. For instance, in this case they only needed to use points to create a polygon.

<figure><img src="image_92.png" /><figcaption>Figure 6: A pop-up notification as a negotiated-style interruption</figcaption></figure>

We counted the number of pop-up cases on our 17 videos. In total, there are 35 cases where 27 of them led students to successfully accomplish the tasks because both styles of interruption informed them about the errors and how they could fix them. Nevertheless, we did not investigate if the students struggle with the pop-ups that were written in English language.

## Discussion

Students benefited from the GeoGebra pop-up notifications that helped them to complete the tasks successfully. This result is in line with the study by Sanz-Ruiz et al. (2023). We observed that in some cases after our students received a pop-up, they made fewer errors when they engaged later with the remaining tasks. This might be called effective error detection skills as per Hattie and Timperley (2007). Students experienced errors and would notice similar pop-ups if they made the same mistakes. They could avoid the same mistakes in later tasks. In this paper, we studied the impact of GeoGebra pop-up notifications on students' actions at the task level of the model of Figure 1. In future studies, we aspire to examine in detail the impact of feedback on students' actions and interactions at the process level, self-regulation level, and self-level.
We showed that the formative feedback from the GeoGebra pop-ups provided students with immediate feedback and notification about their errors so they could reflect on their actions, revise, and make relevant corrections. This type of feedback belongs to the Feed Back direction at first level, feedback of the task (FT) and second level, feedback of the process (FP) by Hattie and Timperley (2007). The provided information about pop-ups and the task is similar to the study by Simmons and Cope (1993) in which LOGO feedback was attributed to FT leading students to more correct answers. At the FP level of the model (Figure 1), the GeoGebra pop-up notification (Figure 6) could act as a cueing mechanism (Hattie & Timperley, 2007), which helps students improve their strategies of programming and coding. It could be in the form of immediate-style of interruption and negotiated-style of interruption to support students reflect on their mistakes. Our data shows that students spent more time to identify their errors when they used the negotiated style of interruption. On the contrary, the immediate-style interruption informed them directly about their errors. However, our data do not support that immediate style of interruption is better than negotiated style of interruption as suggested by the literature (Robertson et al., 2004). Our data show that both styles of interruptions are beneficial to students.

Our GeoGebra-based Math+CT lessons seem to follow the model in Figure 2 by Dalby and Swan (2019). We provided students with GeoGebra tasks. If students input incorrect commands, a pop-up will notify them. Students could proceed with their actions or discuss the pop-up with their peers and teacher. Next, students could delete or edit their incorrect commands and proceed with the next task.

## References

Ayton, D. (2023). Qualitative Descriptive Research. In D. Ayton, T. Tsindos, & D. Berkovic (Eds.), *Qualitative Research – a practical guide for health and social care researchers and practitioners* (pp. 49–54). Monash University.

Black, P., & William, D. (1998). Assessment and classroom learning. *International Journal of Phyto remediation, 21*(1). https://doi.org/10.1080/0969595980050102

Burkhardt, H., & Schoenfeld, A. (2019). Formative Assessment in Mathematics. In H. L. Andrade, R. E. Bennett, & G. J. Cizek (Eds.), *Handbook of Formative Assessment in the Disciplines* (pp. 35–67). Taylor and Francis. https://doi.org/10.4324/9781315166933-3

Cizek, G. J., Andrade, H. L., & Bennett, R. E. (2019). Formative Assessment: History, Definition, and Progress. In *Handbook of Formative Assessment in the Disciplines*.

Dalby, D., & Swan, M. (2019). Using digital technology to enhance formative assessment in mathematics classrooms. *British Journal of Educational Technology, 50*(2). https://doi.org/10.1111/bjet.12606

Hattie, J., & Timperley, H. (2007). The power of feedback. In *Review of Educational Research* (Vol. 77, Issue 1). https://doi.org/10.3102/003465430298487

Liu, Z., Zhi, R., Hicks, A., & Barnes, T. (2017). Understanding problem solving behavior of 6–8 graders in a debugging game. *Computer Science Education, 27*(1). https://doi.org/10.1080/08993408.2017.1308651
Looney, J. (2010). Making it Happen: Formative Assessment and Educational Technologies. *Promethean Thinking Deeper Research Papers*.

McKenney, S., & Reeves, T. C. (2018). Conducting Educational Design Research. In *Conducting Educational Design Research*. https://doi.org/10.4324/9781315105642

Papert, S. (1980). *Mindstorms: Children, Computers, and Powerful Ideas* (1st ed.). Basic Books.

Resnick, M. (2012). Reviving Papert’s Dream. *Educational Technology, 52*(4).

Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., & Kafai, Y. (2009). Scratch: Programming for all. *Communications of the ACM, 52*(11). https://doi.org/10.1145/1592761.1592779

Robertson, T. J., Prabhakararao, S., Burnett, M., Cook, C., Ruthruff, J. R., Beckwith, L., & Phalgune, A. (2004). Impact of interruption style on end-user debugging. *Conference on Human Factors in Computing Systems - Proceedings*. https://doi.org/10.1145/985692.985729

Robson, C. (1993). *Real World Research: A Resource for Social Scientists and Practitioner-Researchers* (1st ed.). Blackwell.

Sanz-Ruiz, M., Diego-Mantecón, Manuel, J., & Ortiz-Laso, Z. (2023). Formative feedback on linear equations. *CADGME 2023: Digital Tools in Mathematics Education*.

Simmons, M., & Cope, P. (1993). Angle and rotation: Effects of different types of feedback on the quality of response. *Educational Studies in Mathematics, 24*(2). https://doi.org/10.1007/BF01273690

Tan, K. (2011). Assessment for Learning Reform in Singapore – Quality, Sustainable or Threshold? In *Assessment Reform in Education*. https://doi.org/10.1007/978-94-007-0729-0_6

Ukkonen, A. (2023). Formative assessment of computational thinking in mathematics – a case study. *Proceedings of the Thirtieth Congress of the European Society for Research in Mathematics Education, 1–8*.

Ye, H., Liang, B., Ng, O.-L., & Chai, C. S. (2023). Integration of computational thinking in K-12 mathematics education: a systematic review on CT-based mathematics instruction and student learning. *International Journal of STEM Education, 10*(3), 1–26. https://doi.org/10.1186/s40594-023-00396-w

Yunianto, W., Prodromou, T., & Lavicza, Z. (2023). Debugging on GeoGebra-based Mathematics+Computational Thinking lessons. *28th Asian Technology Conference in Mathematics, 322–331*.
# Mathematics teacher educators' microteaching implementations: selection of settings and approaches to assessment and feedback for prospective primary teachers

Bilge Yılmaz Aslan<fnref n="1" /> Mehmet Fatih Özmantar<fnref n="2" /> Bilge Kuşdemir Kayıran<fnref n="3" />

This study investigates microteaching in mathematics method courses, focusing on how mathematics teacher educators (MTEs) prepare primary teachers for teaching mathematics. We surveyed 65 MTEs across 71 Turkish universities, examining their approaches to topic selection, session settings, and assessment and feedback in microteaching. We found a preference for MTE-led topic selection and peer-based teaching sessions. Most MTEs tailor evaluation rubrics to individual needs, emphasizing a personalized assessment and feedback process. Furthermore, revealing evaluation criteria before sessions was noted to improve feedback transparency. The study underscores the impact of these practices on formative assessment and feedback.

Keywords: Microteaching, Mathematics methods courses, Mathematics teacher educators.

## Introduction

Primary teachers are essential in developing foundational skills in learners during critical learning periods. Unlike specialized subject teachers, primary teachers cover a range of disciplines, including literacy, social studies, and mathematics. To prepare them, teacher training programs include method courses that focus not only on subject knowledge but also on effective teaching strategies and creating supportive learning environments (Strawhecker, 2005). Recognizing the importance of these courses, we conducted an in-depth study on the structure of mathematics method courses managed by MTEs. This paper presents our preliminary analysis and insights into microteaching within these courses from the MTEs' perspectives. We explore how these practices influence formative assessment and feedback, which are crucial for nurturing teacher candidates and achieving broad educational objectives of enhancing mathematical teaching skills (Bosica et al., 2021). This study addresses two main research questions: (1) Why do/ don't MTEs employ microteaching in method courses? and (2) How do MTEs structure the implementation of microteaching?

## Literature review: microteaching in teacher preparation

Microteaching is a key component in teacher education, renowned for its effectiveness in improving teaching skills and easing the shift from theory to practice. It creates a simulated environment that enhances practical teaching abilities—like concept explanation, material preparation, and performance analysis—in small groups, as highlighted by Brown (1976). It also builds essential competencies for effective mathematics instruction (Dayal & Alpana, 2020; Cheng, 2017) and boosts self-efficacy and confidence in teacher trainees. Peker (2009) notes that microteaching significantly reduces teaching anxiety, improving confidence in lesson delivery. Batten (1979) and Majoni (2017) describe it as a method that focuses on specific skills, shortens teaching time, and reduces class size, effectively linking theoretical knowledge with actual teaching. Additionally, Elias (2018) stresses

<footnote n="1">¹Gaziantep University, Faculty of Education, Gaziantep, Türkiye; bilqe.yilmaz@gmail.com</footnote>

<footnote n="2">²Gaziantep University, Faculty of Education, Gaziantep, Türkiye; ozmantar@gantep.edu.tr</footnote>

<footnote n="3">³Gaziantep University, Faculty of Education, Gaziantep, Türkiye; kbilge01@gmail.com</footnote>
that microteaching facilitates feedback for behavioral change in candidates, adapting to individual
needs and promoting a focused environment for skill development.

Microteaching's effectiveness largely stems from the tailored feedback it provides, which enhances teaching competencies. Brodsky and Doherty (2010) highlight feedback's crucial role in fostering learning and self-assessment, helping trainees identify their strengths and areas for improvement. Zhongji (2006) also notes that incorporating student feedback into microteaching significantly boosts teaching skills, making the training more efficient. Beyond skill development, feedback enhances motivation and interest among trainees, as noted by Özcan and Gerçek (2018). Peer feedback in microteaching can influence perceptions of 'good teaching', focusing on presentation and style, which affects evaluation processes (Vander Kloet & Chugh, 2012). Additionally, Mathew (2018) shows that feedback in microteaching creates a psychological environment conducive to improvement.

Building on these insights, Saraçoğlu and colleagues (2018) highlight that microteaching enables pre-
service teachers to assess their own teaching proficiency in mathematics, providing a critical platform
for self-critique and peer feedback. This reciprocal feedback process, as Semerci (2000) points out,
not only enhances prospective teachers' self-evaluation skills but also improves their teaching
performance through insights from peer evaluations. Learning from peers helps prospective teachers
refine their teaching methods. These collective findings underscore microteaching's vital role in
evaluating and enhancing teaching competencies, especially in mathematics, preparing teacher
candidates for their future roles. Thus, microteaching is essential both as a feedback mechanism and
a preparatory tool, contributing significantly to the candidates' professional readiness.

Methods

In this study, we share a portion of our broader research that explores six key areas in the design and
structure of mathematics method courses within primary teacher education programs: the objectives,
content delivery approaches, the design of teaching-learning situations, assessment and evaluation,
collaborative endeavors, and research activities. To address each dimension, a mixed-methods survey
comprising both open and closed-ended questions was crafted.

This study focuses on responses related to assessment and evaluation in microteaching, inspired by a
literature review on diverse microteaching approaches (Mukuka & Alex, 2024). Three main
characteristics of microteaching—planning, teaching in varied settings, and the evaluation-feedback
cycle—were identified and reflected in the survey questions tailored to each aspect. Initially, the
survey investigates whether MTEs include microteaching in their programs and their reasons for its
inclusion or exclusion. Regarding planning, it queries how MTEs assign microteaching topics, from
predetermined topics to allowing candidates to choose, including other possible methods. For the
teaching settings, it asks about the intended audience, whether it is a real classroom, peers, a
combination with video analysis, or other settings. Concerning evaluation, the survey explores if
MTEs use a rubric for assessing microteaching and how these rubrics are sourced. It also examines
the use of the evaluation tool, such as whether criteria are shared with candidates beforehand, if
assessments are conducted solely by the MTE, or if peer evaluation is involved.

The participants of our study consisted of whole groups of MTEs who were responsible for teaching
the mathematics method course within the primary teacher undergraduate programs at education
faculties across Turkey. The target population consisted of 90 academics at 71 universities to whom
the survey was sent via email. Responses were received from 73 MTEs, and out of these, 65 provided complete answers to all the questions in the survey. The remaining 8 indicated that they could not participate due to various reasons, such as being new to the course. Overall, the sample of 65 participants out of the population of 90 indicates a 72.22% representativeness.

The responses provided by MTEs in the survey underwent both qualitative and quantitative analyses. For the qualitative part, we focused on the open-ended responses, where reasons cited by participants were grouped under common themes to identify the underlying motivations for either implementing or refraining from microteaching in their courses. Quantitatively, the answers to the remaining survey items were scrutinized using descriptive statistics, with a particular emphasis on frequency counts. This analysis helped pinpoint prevailing trends regarding the selection of topics, the settings chosen for teaching, and the employed strategies for evaluation and feedback, providing a comprehensive overview of the current practices among MTEs.

# Findings

## Findings regarding the first research question

Findings related to the first research question revealed that out of 63 MTEs who responded to the open-ended question about the inclusion of microteaching practices in mathematics teaching courses, 2 left the questions unanswered. For those MTEs (n=50) who reported implementing microteaching, their objectives for providing feedback were categorized into six distinct themes. While several MTEs' responses spanned multiple themes, others were associated with a single theme exclusively.

<table><caption>Table 1: The reasons for implementing microteaching</caption><thead><tr><td>Theme</td><td>n</td><td>Sample quotations</td></tr></thead><tbody><tr><td>Putting theory into practices</td><td>27</td><td>I have them do microteaching so that the theoretical knowledge they have learned can be implemented</td></tr><tr><td>Improving trainee self-assessment</td><td>15</td><td>It reduces their anxiety about giving a presentation. It provides instant feedback. Contributes to the development of evaluation and self-assessment skills</td></tr><tr><td>Planning and preparations</td><td>10</td><td>I think the best way to find answers to questions such as how to plan a lesson, how well they comply with this plan.</td></tr><tr><td>Developing instructional skills</td><td>9</td><td>I use it to provide an opportunity for teacher candidates to try their teaching skills, to see the mistakes and deficiencies they make during this time, and to correct them and try again.</td></tr><tr><td>Developing subject matter knowledge</td><td>5</td><td>Teacher candidates develop their own missing or additional subject knowledge by teaching.</td></tr><tr><td>Developing classroom management skills</td><td>4</td><td>To gain awareness and experience in classroom management, to get to know students more closely and to have real classroom environment experiences.</td></tr></tbody></table>

Examining Table 1 reveals that "putting theory into practice" is the predominant theme among the feedback topics MTEs address through microteaching for teacher candidates. This indicates a significant focus on the application of theoretical knowledge within practical teaching scenarios. Conversely, for the teacher educators who do not implement microteaching practices (n=13), the
analysis yielded three distinct codes. These codes, which capture the reasons behind the absence of microteaching in their programs, are detailed in Table 2.

<table><caption>Table 2: The reasons for refraining from Microteaching</caption><thead><tr><td>Codes</td><td>n</td><td>Sample quotations</td></tr></thead><tbody><tr><td>Time constraints</td><td>6</td><td>Incorporating practical applications into class time requires serious time.</td></tr><tr><td>Excessive course load</td><td>3</td><td>The most important reason is my high course load.</td></tr><tr><td>Class size</td><td>4</td><td>I think the classroom environment and class size are not suitable for a microteaching application</td></tr></tbody></table>

## Findings regarding the second research question

In this section, we present quantitative analysis regarding the second research question, more specifically, the methods MTEs use to deliver microteaching topics, the environments they prefer to conduct these sessions, the rubrics they use for assessment, and the evaluation and feedback approaches. The findings are presented in Table 3.

<table><caption>Table 3: MTEs' preferences to structure the microteaching</caption><thead><tr><td>Category</td><td>Choices</td><td>n</td></tr></thead><tbody><tr><td rowspan="3">MTEs' preferences of topic selection for microteaching</td><td>I determine the topics myself and distribute them to prospective teachers</td><td>28</td></tr><tr><td>Prospective teachers decide on the topic of their choice</td><td>16</td></tr><tr><td>Other</td><td>6</td></tr><tr><td rowspan="3">Preferred settings for microteaching implementations</td><td>Teaching in front of prospective teachers</td><td>34</td></tr><tr><td>Teaching in real classroom environment and video recording and then showing it to prospective teachers</td><td>9</td></tr><tr><td>Teaching in front of students in real classroom environment</td><td>7</td></tr><tr><td rowspan="3">Methods employed to create evaluation rubrics</td><td>I prepare it myself</td><td>26</td></tr><tr><td>I adapt the existing</td><td>11</td></tr><tr><td>I use ready-made</td><td>3</td></tr><tr><td rowspan="3">Preferences of MTEs in sharing evaluation criteria</td><td>I show the criteria to the prospective teachers in advance and evaluate them myself</td><td>27</td></tr><tr><td>The prospective teacher is evaluated by his/her peers as well as myself.</td><td>16</td></tr><tr><td>I evaluate only myself during the teaching without informing the criteria to the teacher candidates</td><td>7</td></tr></tbody></table>

With regard to choosing topics for microteaching sessions, a significant portion of MTEs (28 out of 50 respondents) opt to select the topics themselves to distribute to teacher candidates. In contrast, a smaller subset of 16 MTEs allows the candidates to choose their own topics.
As for the preferred settings for microteaching, a majority of the MTEs (34 out of 50) favor having teacher candidates conduct sessions in front of their peers. This method is more popular than teaching in a real classroom environment, which only garnered 7 responses, while 9 MTEs showed a preference for a combined approach where candidates teach in a real classroom and then review a video recording of their performance.

In terms of the assessment rubrics used for microteaching, most MTEs (26 out of 40) craft their own evaluation tools, indicating a trend towards tailor-made assessment strategies. Meanwhile, 11 MTEs adapt existing rubrics, and only 3 utilize pre-made rubrics without any alterations, underscoring the inclination towards personalized evaluation methods in microteaching.

With respect to the use of rubrics for evaluating microteaching sessions, the prevalent method among MTEs, with 27 indications, involves sharing the evaluation criteria with teacher candidates in advance, followed by the MTE conducting the assessment. A minority of 7 MTEs, however, choose not to reveal the criteria before the session and proceed with an independent evaluation. Additionally, 16 respondents incorporate peer feedback into the assessment process, combining it with their own evaluations.

# Discussion

## Discussion of research question one

Our study highlights the objectives behind MTEs' use of microteaching, primarily to transform theoretical knowledge into practical teaching application, bridging an important educational gap. This practice echoes Brodsky and Doherty's (2010) emphasis on the importance of effective feedback for the development of teacher candidates, as it helps them identify strengths and improvement areas. MTEs value "Improving self-assessment," in line with Zhongji's (2006) findings on the benefits of student feedback. Microteaching serves the dual purpose of honing teaching skills and fostering reflective practices, crucial for professional growth. MTEs also employ microteaching for comprehensive teacher preparation, including "Planning and preparations," "Developing instructional skills," "Development of subject matter knowledge," and "Classroom Management," recognizing the multifaceted nature of teaching and the diverse competencies required (Mathew, 2018). Drawing on insights from Saraçoğlu et al. (2018) and Semerci (2000), our findings argue that microteaching transcends a mere training approach, standing out as a critical, feedback-centric process that substantially contributes to teacher readiness. MTEs intentionally create a collaborative environment that bolsters peer learning and growth. Through microteaching, MTEs not only teach but also build a dynamic space for pre-service teachers to practice, reflect, and evolve, with feedback as a central element of this transformative experience.

## Discussion of research question two

In this section, we discuss three aspects of microteaching—topic selection, preferred settings, and evaluation and feedback approaches.

## Topic Selection

Our findings illuminate a pronounced inclination among MTEs to dictate microteaching topics, thereby exerting considerable influence over the trajectory of learning experiences and the nature of feedback provided to students. By selecting the topics, as we see it, MTEs are implicitly endorsing
certain instructional priorities and competencies that they deem essential for prospective teachers to develop. This practice may, however, inadvertently narrow the scope of prospective teachers' potential growth. There is a potential risk that the feedback becomes tailored to a specific set of topics and attributes, potentially at the expense of a more holistic instructional approach that includes adaptability, responsiveness and pedagogical creativity. Moreover, Benton-Kupper (2001) underscores the value of microteaching as a scaffolded platform that enhances pre-service teachers' skills by offering them a space to engage in teaching practices and receive targeted feedback. The MTEs' involvement in topic selection is critical in this context, as it directs the areas of teaching that are emphasized and scrutinized. This deliberate guidance can have profound implications for the development of teaching skills, as it can ensure that feedback is specific, actionable, and aligned with the MTEs' vision.

## Setting for microteaching

Our study indicates a significant inclination for conducting microteaching sessions in peer-based settings within a controlled environment, rather than directly within the real-world primary classroom context. Educators prefer these settings to facilitate a stable and secure environment where trainee teachers can develop their skills away from the complexities and unpredictability that come with an actual classroom (Benton-Kupper, 2001). This approach prioritizes an atmosphere conducive to formative feedback that is both immediate and specific, without the distractions and challenges that a typical primary classroom might impose. The benefit of such targeted and immediate feedback has been noted for its positive effects on the professional development of teacher candidates (Hidayah & Indriani, 2021). Nonetheless, it is recognized that feedback derived from a peer-based setting may inherently differ from that garnered in a real-classroom scenario. In a true classroom setting, feedback is not only prompt but also enriched by the real-life dynamics of classroom interaction, which is vital for fostering a teacher's ability to adapt and develop responsive teaching techniques (Sen, 2009). Traditional microteaching sessions in a university setting, with the oversight of MTEs, attempt to bridge the gap between educational theory and practice (Cheng, 2017), but still, they fall short of emulating the full spectrum of challenges in a primary classroom. Reflecting on this limitation, some researchers advocate for integrating microteaching sessions into actual primary classrooms to provide teacher candidates with a comprehensive and authentic teaching experience (Peker, 2009).

## Evaluation and feedback approaches

Our findings show that most MTEs prefer to design or tailor their evaluation rubrics, highlighting the importance of context-specific, learner-centered feedback for improving student understanding and informing teacher insights on teaching effectiveness (Haug & Ødegaard, 2015). MTEs commonly share evaluation criteria with candidates beforehand, promoting transparency and structure in the feedback process. This approach supports Megawati's (2018) findings that peer assessments in microteaching enhance teaching skills by boosting confidence within a supportive learning atmosphere. Moreover, the use of feedback forms significantly influences teacher candidates' notions of effective teaching and may affect their self-concept and teaching approaches, as noted by Vander Kloet and Chugh (2012). In contrast, a smaller group of MTEs who choose not to disclose criteria prior to teaching might aim to encourage an independent assessment of teaching abilities, potentially leading to the development of more spontaneous teaching skills. This feedback strategy may lead
candidates to display a wider array of teaching behaviors, contributing to a more comprehensive
development, suggesting a need to refine evaluation methods to better support educational objectives.

Acknowledgement

This presentation stemmed from the first author’s PhD studies.

References

Batten, H.D. (1979). *Factors Influencing the Effectiveness of Microteaching in a Teacher Education Programme*. Unpublished PhD thesis (Stirling, Department of Education, University of Stirling).

Benton-Kupper, J. (2001). The Microteaching Experience: Student Perspectives. Education 3-13, 121(4), 830-835.

Bosica, J., Pyper, J. S., & MacGregor, S. (2021). Incorporating problem-based learning in a secondary school mathematics preservice teacher education course. *Teaching and Teacher Education, 102*, 103335. http://dx.doi.org/10.1016/j.tate.2021.103335

Brodsky, D., & Doherty, E. G. (2010). Providing effective feedback. *NeoReviews*, 11(3), 117-122.

Brown, G. (1976). Using Microteaching to Train New Lecturers. *Educational Media International, 13*(3), 12-16.

Cheng, J. (2017). Learning to attend to precision: The impact of micro-teaching guided by expert secondary mathematics teachers on pre-service teachers' teaching practice. *ZDM, 49*(1), 279-289. http://dx.doi.org/10.1007/s11858-017-0839-7

Dayal, H., & Alpana, R. (2020). Secondary pre-service teachers’ reflections on their micro-teaching: Feedback and self-evaluation. *Waikato Journal of Education, 25*(1), 73-83.
http://dx.doi.org/10.15663/wje.v25i0.686

Elias, S. K. (2018). Pre-service teachers’ approaches to the effectiveness of micro-teaching in teaching practice programs. *Open Journal of Social Sciences, 6*(5), 205-224.
https://doi.org/10.4236/jss.2018.65016

Haug, B. S., & Ødegaard, M. (2015). Formative Assessment and Teachers' Sensitivity to Student Responses. *International Journal of Science Education, 37*(4), 629-654.
https://doi.org/10.1080/02607476.2015.1080424

Hidayah, N., & Indriani, L. (2021). Real time feedback in English microteaching practice: A case study on online learning. *Metathesis: Journal of English Language, Literature, and Teaching, 5*(2), 155-167. http://dx.doi.org/10.31002/metathesis.v5i2.4004

Majoni, C. (2017). Assessing the effectiveness of microteaching during teacher preparation. *European Journal of Research and Reflection in Educational Sciences, 5*(2).

Mathew, L. K. (2018). Comparative Study of Responses from Different Groups in Microteaching. *Journal of Medical Science and Clinical Research, 6*(2), 806-809.
https://dx.doi.org/10.18535/jmscr/v6i2.123

Megawati, F. (2018). Peer observation of teaching: Pre-Service Teachers’ Perspectives for Better Performance. *Advances in Social Science, Education and Humanities Research, 125*(1), 124-127.
http://dx.doi.org/10.2991/icigr-17.2018.30
Mukuka, A., & Alex, J. K. (2024). Review of research on microteaching in mathematics teacher education: Promises and challenges. *Eurasia Journal of Mathematics, Science and Technology Education, 20*(1), 1-15. http://dx.doi.org/10.29333/ejmste/13941

Özcan, Ö., & Gerçek, C. (2018). Multidimensional analyzing of the microteaching applications in teacher education via videograph. *European Journal of Teacher Education, 42*(1), 82-97. https://doi.org/10.1080/02619768.2018.1546285

Peker, M. (2009). The use of expanded microteaching for reducing pre-service teachers' teaching anxiety about mathematics. *Scientific Research and Essays, 4*(9), 872-880. https://doi.org/10.12973/ejmste/75284

Saraçoğlu, G., Gürışık, A., & Furat, D. (2018). Opinions of english teacher candidates regarding the criticism made after micro-teaching applications. *Turkish Journal of Educational Sciences, 16*(1), 58-76.

Semerci, N. (2000). The Effect of Critical Thinking on Developing Criticism Skills in Micro Teaching Course (FÜ Technical Education Faculty Sample). *Education and Science, 25*(117).

Sen, A. I. (2009). A study on the effectiveness of peer microteaching in a teacher education program. *Education and Science, 34*(151), 165-174.

Strawhecker, J. (2005). Preparing elementary teachers to teach mathematics: How field experiences impact pedagogical content knowledge. *Issues in the Undergraduate Mathematics Preparation of School Teachers: The Journal, 4*(Curriculum), 1-12.

Vander Kloet, M. A., & Chugh, B. P. (2012). An interdisciplinary analysis of microteaching evaluation forms: How peer feedback forms shape what constitutes “good teaching”. *Educational Research and Evaluation, 18*(6), 597-612.

Zhongji, H. (2006). Microteaching-Researching: an effective approach to improving the teaching skills of college teachers. *Journal of Lanzhou Institute of Education, 3*, 40-42.
# Certainty-based marking as feedback in the context of formative assessment in mathematics lessons at school

Joerg Zender<fnref n="1" /> and Martina Geisen<fnref n="2" />

Self-reflection is an essential skill needed to adjust one's learning process. However, training is necessary to develop this skill. Certainty-based marking is an easy-to-integrate method of formative assessment that strengthens predictive accuracy about one's answers to mathematics problems. Knowing better about one's knowledge might lead to better self-reflection. An explorative study with German 5th graders is presented here, in which the influence of certainty-based marking in formative assessment on pupils' mathematical self-reflection during mathematics lessons is examined. Initial findings show that the certainty and the accuracy with which the pupils state their knowledge rise over time.

Keywords: Formative assessment, certainty-based marking, self-reflection, middle school, feedback.

## Introduction

Self-reflection is a cognitive process involving conscious observation of one's thoughts, feelings, and actions and is essential for understanding individual learning, e.g. for mathematics lessons at school. In dealing with heterogeneity, various forms of differentiation are used in mathematics lessons in which, among other methods, learners choose from the options provided (e. g. Hußmann & Prediger, 2007). However, this can be a challenge for students. They must first learn to assess themselves correctly and then make a selection based on this to control their learning and take responsibility for it in the long term. So, promoting students' self-reflection can improve teaching and learning outcomes. One possibility for training students' self-reflection could be the method of certainty-based marking (CBM) in the context of formative assessment. CBM is a testing format that requires students to express a degree of certainty in their responses, which will be considered for grading.

This article presents the initial findings of an ongoing exploratory study that investigates how CBM in formative assessment influences the self-reflection of 5th-grade learners regarding mathematics at school. In the following sections, CBM is first explained to address its potential in the context of formative assessment and the need for research in this area. Then, the methodological approach is described, initial results are presented, and finally, the results are reflected upon.

## Certainty-based marking as feedback in the context of formative assessment

Studies on feedback in the context of formative assessment were carried out by Brensing et al. (2021). The results show that combined formative assessment and feedback can be a powerful tool for fostering mathematical learning. In their study on a first-semester course of mathematics for engineering, counselling after a formative assessment leads to better results, and even weak students with low mathematical knowledge could come close to the mathematical performance of the middle group. The feedback was given in individual counselling sessions, which was quite work-intensive and time-consuming, but it helped the students to adjust their learning and perform better. Following Kruger and Dunning (1999), in these counselling sessions it was revealed that the weak students did

<footnote n="1">¹Private School, Frankfurt am Main, Germany; joerg@zender.xyz</footnote>

<footnote n="2">²University of Potsdam, Germany; martina.geisen@uni-potsdam.de</footnote>