# Assessment through mathematical problem-posing

Rogier Bos and Rebecca Kuijpers

Utrecht University, the Netherlands; r.d.bos@uu.nl

This study concerns problem-posing as a means of assessment in upper-secondary mathematics education. The open character of problem-posing as a task, allows students to show their creativity but makes it difficult to control the focus on the learning goals. Problem-posing can be structured by adding an initial problem to the prompt. We aim to investigate how this form of structuring affects the resulting problems and the extent to which they reveal students' thinking and knowledge with respect to learning goals. In line with previous research on assessment through problem-posing by Kwek and by Mishra and Iyer, we classify the complexity of the problems. Additionally, we analyze whether the problems address the learning goals and are solvable. The main outcome is that structuring the problem-posing prompt is more suitable for assessment since the resulting problems align better with the learning goals and reveal more of the qualities and misunderstandings of the students.

Keywords: Assessment, Mathematics education, Problem-posing.

## Introduction

Problem-posing is a teaching approach where students are invited to create or reformulate a problem rather than solve a given one. It has been implemented in some national curriculums and is part of the U.S. Principles and Standards for School Mathematics of the NCTM. Research on problem-posing has been ongoing since at least the '90s (Silver, 1994; Stoyanova, 1997); see, e.g., the two recent reviews by Baumanns & Rot (2020; 2021). Reasons to teach problem-posing are, among others, that it invites students to analyze situations and that it fosters their creativity (Baumanns & Rot, 2020).

In analogy to the well-known distinction between teaching problem-solving and teaching through problem-solving, one can distinguish between assessing problem-posing and assessing through problem-posing. The former has been studied extensively (e.g., Silver & Cai, 2005). However, research on assessing through problem-posing is limited to our knowledge (Kwek, 2015; Mishra & Iyer, 2015). If students are taught through problem-posing, as advocated Zhang and Cai (2021), assessing through problem-posing would improve constructive alignment (Biggs & Tang, 2011). The question is: how can we assess a student's learning, knowledge, and skills by considering the problems they pose? The open character of problem-posing tasks seems to prevent teachers from focusing the outcome on the learning goals. Hence the handed-in problems may not allow teachers to assess what they intend to assess. Moreover, the problems students pose may not display the level at which the teacher intended to assess. However, the same open character also allows teachers to assess students' creativity (Baumanns & Rot, 2021). It may allow students to show what they can do beyond what the teacher might envision, and that could be an attractive property for assessment.

In this study, we investigated how structuring the problem-posing prompt, i.e., providing an initial problem as part of the prompt, might help teachers to nudge the problem-posing in the direction of the desired learning goals and the desired level. It is a trade-off: by structuring the problem the task will be less open, but more focused. Does that hamper creativity? Does it lead to a more informative
assessment? By comparing two groups of 10th graders that pose problems, with or without an initial problem, we aim to study how problem-posing can be used for assessment, and whether adding an initial problem improves assessment.

**Theoretical background**

Stoyanova and Ellerton (1996) differentiate problem-posing situations as free, semi-structured, or structured. The problem-posing starts from a provided real-life or artificial situation. If there are no further restrictions, the situation is called *free*. In a semi-structured situation, the problem should require prescribed mathematical concepts or skills. In a structured situation, students are provided with an initial problem, after which they are invited to pose more problems about the same situation. In the latter case, mathematical concepts and skills are suggested by the initial problem. Baumanns and Rott (2020) often take the two types free and semi-structured together, and so shall we in this article, using the label *unstructured situation*.

We found two papers that explicitly address problem-posing as a means of assessing using semi-structured prompts: Kwek (2015) and Mishra & Iyer (2015). Kwek (2015) introduced rubrics to classify the complexity of the problems that students pose. Table 1 shows a shortened version of these rubrics. Kwek analyzes a set of problems posed by 7th and 9th graders. For grade 9, 78% of the problems are solvable, 67% are of low complexity, 30% are of moderate complexity, and 3% are of high complexity. Students were invited to discuss each other's problems and decide whether these were interesting and challenging. The grade 9 students found 58% of the problems interesting and 50% of the problems challenging. The grade 9 students showed appreciation for problems with strong mathematical content. Kwek concludes that both cognitive factors, like thinking processes and understanding, and affective factors were revealed through classroom problem-posing, making it a suitable assessment activity. However, we believe it could be of additional interest to focus on how the problems reveal students' misunderstanding and to see whether problems cover learning goals.

Mishra and Iyer analyze problems posed as part of a computer science course. Similarly to Kwek, they classify the complexity of the problems based on rubrics: 39% low, 51% medium, and 10% high. 85% of the advanced students who scored high on a classical assessment still produced a problem of medium to low complexity. This indicates that students are not necessarily challenged to perform to their highest ability by a problem-posing assignment. Mishra and Iyer also track which learning goals concerning computational thinking are covered by the problems. They find that some learning goals are better addressed than others, ranging from some goals only covered by 8% of the problems to others by 96%. Missing out on certain learning goals in an assessment can be problematic. In this paper, we study whether providing structured prompts improves such coverage of the learning goals.

In line with these papers, we propose, when assessing through problem-solving, to take into account the solvability of the problem, the complexity of the problem, and the extent to which it covers the learning goals. Solvability is a measure of correctness: if students produce a problem that cannot be solved, this influences the assessment in the same way an incorrect answer influences a traditional assessment. Moreover, it makes sense to take the complexity into account. A problem-posing task has a degree of freedom that could best be compared to the difference between a simple correct answer and an impressive correct answer: complexity is a quantity that allows one to capture this dimension.
As part of the study that we report on in this paper, we compare structured with unstructured problem-
posing as a form of assessment. Our research question is: how do these types of situations and prompts
for problem-posing contribute to assessment? Is one type more suitable than the other? We restrict
ourselves to high-achieving 10th-graders and the subject of probability, but later discuss whether the
results might extend beyond these specifics.

<table><caption>Table 1. Complexity of a posed problem; adapted from Kwek (2015)</caption><thead><tr><th></th><th>Low complexity</th><th>Moderate complexity</th><th>High complexity</th></tr></thead><tbody><tr><th>Description</th><td>The problem typically specifies what the solver is to do, which is often to carry out some procedure that can be performed mechanically.</td><td>Solving the problem involves more flexible thinking and choice among alternatives. It requires going beyond routine approaches or using multiple steps.</td><td>High-complexity problems make heavy demands on solvers, who must engage in more abstract reasoning, planning, analysis, judgment, and creative thought.</td></tr><tr><th>Cognitive demand</th><td><ul><li>Recall or recognize a fact, term, or property</li><li>Perform a specified routine of steps</li><li>Retrieve information from a graph, table, or figure</li></ul></td><td><ul><li>Represent a situation mathematically in more than one way</li><li>Justify steps in a solution process</li><li>Interpret a visual representation</li><li>Extend a pattern</li><li>Interpret a simple argument</li></ul></td><td><ul><li>Perform a non-routine procedure having multiple steps and multiple decision points</li><li>Generalize a pattern</li><li>Explain and justify a solution to a problem</li><li>Provide a mathematical justification</li></ul></td></tr></tbody></table>

Method

The study was performed in the context of Mathematics D Online, a Dutch nationwide hybrid (mixed online and onsite) course on advanced mathematics for high-achieving secondary school students. As part of this program, students were invited to hand in answers to a weekly set of tasks. 275 students aged 15 to 16 were enrolled in the course in 2022/2023. We had a sample of 20 students, which we found sufficient, based on previous studies on problem posing with similar data collection (Kwek, 2015; Stoyanova, 1997). The sample was not random, but based on student's positive replies to a request to participate: a convenience sample. We replaced four of the hand-in tasks on probability with problem-posing tasks. Each task consisted of a context and a prompt. For both structured and unstructured tasks, the context was identical, but the prompt differed (see Table 2).

The handed-in problems and accompanying answer models were analyzed and coded for coverage of the learning goals, complexity, and solvability by the second author. The first author performed a second coding. The coverage of learning goals was determined by comparing it to a list of learning
goals, representing the material the students were working on. The complexity was coded using an extended version of Table 1. The solvability was determined by carefully examining the problem and the answer model, taking into account that the problem needs to be clear about what needs to be solved and provide enough information to do so, and that it needs to be mathematically correct and consistent. Next, these results were statistically analyzed with suitable tests to allow comparison of structured and unstructured prompts.

<table><caption>Table 2. Examples of problem posing tasks</caption><thead><tr><th rowspan="2">Context</th><th colspan="4">Prompt</th></tr><tr><th colspan="2">Structured</th><th colspan="2">Unstructured</th></tr></thead><tbody><tr><td>A random variable X has the following distribution:</td><td></td><td></td><td></td><td></td></tr><tr><td>X</td><td>0</td><td>1</td><td>2</td><td></td></tr><tr><td>P(X = x)</td><td><sup>1</sup>&frasl;<sub>2</sub> - <sup>p</sup>&frasl;<sub>2</sub></td><td>p</td><td><sup>1</sup>&frasl;<sub>2</sub> - <sup>p</sup>&frasl;<sub>2</sub></td><td></td></tr><tr><td>with 0 &le; p &le; 1.</td><td colspan="4"></td></tr><tr><td>A factory produces blue, green, and red soaps. The weight of a soap is normally distributed, with &mu; = 100g and &sigma; = 3g for blue soaps, &mu; = 120g and &sigma; = 4g for red soaps and &mu; = 80g and &sigma; = 3g for green soaps. The volume of a soap is normally distributed, where &mu; = 0,2l and &sigma; = 0,002l for blue soaps, &mu; = 0,25l and &sigma; = 0,003l for red soaps and &mu; = 0,18l and &sigma; = 0,003l for green soaps. The factory sells blue soaps for €1,-, red soaps for €1,50, and green soaps for €0,85. The number of soaps sold per day is normally distributed, with &mu; = 40 and &sigma; = 3 for blue soaps, &mu; = 35 and &sigma; = 2,5 for red soaps and &mu; = 40 and &sigma; = 2,5 for green soaps.</td><td colspan="2">a. Compute the standard deviation in terms of p.<br/>b. Pose two more problems on this distribution. Also make the answer model.</td><td>Pose three problems on this distribution.<br/>Also make the answer model.</td></tr><tr><td colspan="5">a. Compute the probability that the volume of a blue soap is less than 0,24l or more than 0,26l.<br/>b. Pose two more problems on this distribution. Also, make the answer model.</td></tr></tbody></table>

## Results

Nine students handed in 33 problems from structured prompts, and 11 students handed in 53 from unstructured prompts. All problems were coded by the second author. For assessing interrater agreement, 1/3 of the problems were coded by the first author. The codes on solvability and learning goals were identical. Before discussion, there was a Cohen's kappa of 0.66 on the coding of complexity. This was mainly due to confusion about whether a multistep problem should be coded as low or moderate complexity. The raters decided to add the distinction "routine versus non-routine" to the complexity matrix. This decision gave clarity on most differences, leading to a Cohen's kappa of 0.96.
## Sample problems

We discuss three problems from our sample here. The first problem was constructed as a response to the first context. The posed problem was: Compute the standard deviation when *p* = 1/2. The problem was coded as having a low complexity because computing the standard deviation is a routine procedure for these students. The problem is solvable and covers a learning goal, namely computing the standard deviation from a probability distribution. Hence the problem posed and the answer model allow us to assess the student's progress for this learning goal, but only on a reproductive level.

The second problem was constructed as a response to the second context. The posed problem was: Compute the probability that the gain of the factory is more than €36, just from the green soap. It was coded as having a moderate complexity because it is not a routine problem. While the computation is a standard one, the solver first has to realize that the answer lies in the number of soaps sold, not the price of the soap. The covered learning goal is: to compute probabilities using the normal distribution, where the average value, standard deviation, and the boundaries are given. The problem was formulated with clarity and is solvable. Hence, it allows us to assess the students on this goal and also conclude that a level of flexibility and creativity was achieved.

The third problem was also formulated in the second context and consists of two parts. Part 1: Which color of soap has a higher probability of being sold less than 39 times per day? Part 2: what are the differences in those probabilities per color? This problem, too, was coded as having a moderate complexity, because the problem, apart from computing probabilities, involves comparing these probabilities. The problem covers the same learning goal as the second problem. The problem is not properly solvable, because of an accidentally too open formulation: the student did not specify how the difference between probabilities should be expressed. While it is unusual to rate the complexity of unsolvable problems (see for example Silver & Cai, 1996), we chose to do so, because with all these problems the intention of the author of the problem was clear, also from their answer model. The formulation of the problem reveals a gap in knowledge about how probabilities should be compared.

## Problems covering the learning goals

For each problem posed we analyzed whether it covered at least one learning goal. Of the structured tasks, 97.0% addressed at least one learning goal, whereas for the unstructured versions, this was 90.7%. If a problem did not address a learning goal, then it was in most cases also of low complexity. The learning goals we wanted to be addressed within a context, were addressed in at least some of the problems posed for all learning goals, except one; this exception was due to our fault of not providing a good context for it.

A chi-square test was applied to analyze the differences in coverage of learning goals between structured and unstructured problem-posing exercises. From this, we conclude that there was no significant difference between the number of posed problems that covered the learning goals in structured and unstructured problem-posing exercises (*p* = 0.315).
## Complexity of problems posed

To analyze the differences in complexity between structured and unstructured problem-posing exercises, we applied a Mann-Whitney U test to the coded problems. This gave $M - W = 641$, $p = 0.048$, which means that there is a significant difference in complexity between problems posed as a result of structured and unstructured prompts. The results in Table 3 support that problems posed in response to structured prompts are generally of higher complexity than those posed in response to unstructured prompts.

Problems posed in response to an unstructured prompt tend to be solvable by routine. For example, in the first context a student posed the problems: a. compute the expectation value; b. compute the quadratic deviations; c. compute the variance. The formulation of the structured prompt prevents posing such routine questions because the routine steps are already included as initial problems in the prompt (see Table 2).

<table><caption>Table 3. Relative frequencies of complexity for structured and unstructured exercises</caption><thead><tr><th rowspan="2"></th><th colspan="3">Complexity</th><th rowspan="2">Total</th></tr><tr><th>Low</th><th>Moderate</th><th>High</th></tr></thead><tbody><tr><th>Structured</th><td>48,5%</td><td>36,4%</td><td>15,2%</td><td>100%</td></tr><tr><th>Unstructured</th><td>68,0%</td><td>30,0%</td><td>4,0%</td><td>100%</td></tr></tbody></table>

When students pose complex problems, they do this by combining the context with new elements. For example, one student posed the following problem within the second context: A box contains 20 soaps, of which 6 are blue, 7 are red and 7 are green. What is the probability of grabbing a red soap that weighs more than 125 grams two times in a row (not putting soaps back in the box)? So the student combined discrete and continuous probability.

## The solvability of posed problems

We found that 69.7% of the problems posed in response to a structured prompt were solvable, whereas 94.3% of the ones from unstructured prompts were. To analyze the differences in solvability between structured and unstructured problem-posing exercises, we applied a chi-square test. There was a significant difference in solvability between problems posed resulting from structured and unstructured prompts ($p = 0.002$).

The unsolvable problems could be categorized into two categories, namely a category of problems that revealed a misconception or misunderstanding of concepts and a category of problems that were poorly formulated, but otherwise sound. An example of the first category: a student introduces the following discrete probability distribution:
<table><thead><tr><th>X</th><th>0</th><th>1</th><th>2</th><th>3</th></tr></thead><tbody><tr><td>H(x = x)</td><td>1 - h</td><td>h</td><td>h + 1</td><td>h + 2</td></tr></tbody></table>

Firstly, we see the notational issues in the first column, which should contain “x” and “P(X = x)”. This indicates that the student does not understand the role of the random variable X and the variable x. Moreover, adding up the probabilities, one finds 2h + 4, which should equal 1 (hence $h = -\frac{3}{2}$). However, the student means h not to be determined, and may not realize that the probabilities should add up to 1.

An example of the second category: Compute the revenue of green soaps with a probability of 0,115069670222. This is poorly formulated, and hence unsolvable, since it is not clear what the probability applies to. It could be the probability that a soap is sold to a customer, or the probability that the green soap has a certain weight, etcetera. The answer model revealed a consistent interpretation showing the student's intention, hence this problem allowed us to assess the student's progress on both learning content and mathematical problem formulation.

## Conclusion and discussion

In conclusion, we state that structured prompts seem more suitable for assessing, for three main reasons. Firstly, structured prompts invite more complex problems, which in turn show more of the students' capabilities. Secondly, Structured prompts lead to significantly more unsolvable problems than unstructured prompts. This may seem bad, but it is good from an assessment point of view: those problems are usually not fundamentally unsolvable, and how the problems are unsolvable reveals misunderstandings and misconceptions of students. This may be caused by students challenging themselves more with structured prompts. This is in contrast with Mishra and Iyer (2015), who observed that, with semi-structured prompts, students would produce problems below their capabilities, as mentioned in the theoretical background. Thirdly, structured prompts lead to more problems that cover learning goals—though not significantly more. Either way, for both types of prompts the context of the task, combined with the context of the presentation of the task, namely as part of a work on a chapter on statistics and probability, ensured the problems posed revealed students' progress with respect to the learning goals of the chapter. In most cases, the problems addressed the learning goals we had envisaged, though in some cases this potential was not realized.

Our results were obtained specifically with high-achieving students enrolled in a hybrid national course. Also, the topic was specific: probability. However, we believe the conclusion on the impact of structuring the prompts holds beyond these specifics. Yet, the effect of structuring might wane after students get used to problem-posing, and know what is expected of them. From a new study, we have indications that this may happen within two or three iterations.

Since this study did not involve classroom situations, for future research we would be interested in the impact of exchange between students with respect to the problems they pose, in the line of Kwek's work (2015). Such discussion might reveal more of students' thinking and invite them to improve their problems.
# Acknowledgments

The authors would like to thank Benjamin Rott for suggestions and discussions concerning this study.

# References

Baumanns, L., & Rott, B. (2020). Rethinking Problem-Posing Situations: A Review. *Investigations in Mathematics Learning, 13*(2), 59–76. https://doi.org/10.1080/19477503.2020.1841501

Baumanns, L., & Rott, B. (2021). Developing a framework for characterizing problem-posing activities: a review. *Research in Mathematics Education, 24*(1), 28–50. https://doi.org/10.1080/14794802.2021.1897036

Biggs, J & Tang, C. (2011). *Teaching for Quality Learning at University*, McGraw-Hill and Open University Press, Maidenhead.

Kwek, M.L. (2015). Using Problem Posing as a Formative Assessment Tool. In: Singer, F., F. Ellerton, N., Cai, J. (Eds.) *Mathematical Problem Posing*. Research in Mathematics Education. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-6258-3_13

Mishra, S., Iyer, S. (2015). An exploration of problem posing-based activities as an assessment tool and as an instructional strategy. *Research and Practice in Technology Enhanced Learning, 10*(5). https://doi-org/10.1007/s41039-015-0006-0

Silver, E. A (1994). On mathematical problem posing. For the Learning of Mathematics, 14(1), 19–28.

Silver, E. A., & Cai, J. (1996). An Analysis of Arithmetic Problem Posing by Middle School Students. *Journal for Research in Mathematics Education, 27*(5), 521–539. https://doi.org/10.2307/749846

Silver, E. A., & Cai, J. (2005). Assessing students’ mathematical problem posing. *Teaching Children Mathematics, 12*(3), 129–135. https://doi.org/10.5951/TCM.12.3.0129

Stoyanova, E., & Ellerton, N. F. (1996). A framework for research into students’ problem posing in school mathematics. In P. C. Clarkson (Ed.), *Technology in mathematics education* (pp. 518–525). Melbourne, Mathematics Education Research Group of Australasia

Stoyanova, E. (1997). *Extending and exploring students' problem-solving via problem-posing*. [Doctoral dissertation, Edith Cowen University]. Research Online Institutional Repository. https://ro.ecu.edu.au/theses/885/

Zhang, H., & Cai, J. (2021). Teaching mathematics through problem posing: insights from an analysis of teaching cases. *ZDM Mathematics Education* 53, 961–973. https://doi.org/10.1007/s11858-021-01260-3
# Designing with the Teaching for Robust Understanding framework: indicators for the activation and realization of formative assessment strategies

Alessandra Boscolo¹, Francesca Morselli², Simone Quartara³ and Elisabetta Robotti⁴

¹DIMA University of Genova, Italy; boscolo@dima.unige.it

²DIMA University of Genova, Italy; morselli@dima.unige.it

³IIS Italo Calvino, Genova, Italy; simone.quartara@calvino.edu.it

⁴DIMA University of Genova, Italy; robotti@dima.unige.it

The study explores the implementation of formative assessment strategies in the context of algebraic thinking and argumentation within a teaching experiment. The Teaching for Robust Understanding framework and theoretical references guide the task design. Specific indicators for two formative assessment strategies are developed, tailored to the learning goals, and examples of their instances (activation and realization by teachers and students) are provided. Future work will extend this analysis to other strategies and assess its applicability in other learning sequences.

Keywords: Formative assessment, TRU framework, community of inquiry

## Introduction and background

Our contribution addresses task design and the assessment of algebraic thinking and argumentation as key learning objectives. We rely on Black and William's (2009) characterization of formative assessment as a method of teaching where "evidence about student achievement is elicited, interpreted, and used by teachers, learners, or their peers, to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions they would have taken in the absence of the evidence that was elicited". (p. 7). Schildkamp and colleagues (2020) underscore the challenges teachers face in implementing formative assessment effectively within their classrooms. In their literature review, they point out that it is imperative to integrate formative assessment seamlessly into the teaching and learning process, surpassing the mere addition of formative assessment activities. Moreover, teachers should be inclined to share the responsibility of instruction with students, thereby renegotiating the role and authority of the teacher in the teaching and learning process. Teacher prerequisites supporting this shift in perspective and practice encompass pedagogical content knowledge (essential for identifying student difficulties and offering feedback), the ability to articulate and share learning goals with students, and the capacity to facilitate class discussions. Additionally, social factors, such as collaboration with colleagues and cultivating positive relationships with students, play pivotal roles. All these factors underscore the importance of making teachers able to effectively implement formative assessment in their daily classroom practices, recognizing the complexity inherent in the teaching and learning process. We contend that creating an appropriate context for teachers to reflect on their practice, share and compare experiences, and providing them with theoretical tools supporting the design and implementation of teaching sequences, including formative assessment activities as integral components, is paramount.
Our study was conducted within the community of inquiry DIVA (Didattica, Inclusione, Valutazione formativa, Argomentazione – Didactics, Inclusion, Formative Assessment, Argumentation), established at the Mathematics Department of the University of Genoa in February 2023. This community of teacher-researchers has been collaborating to identify theoretical tools for reflection, address specific needs and areas of interest, and design teaching and learning sequences. The initial theoretical tool shared and utilized was Schoenfeld's TRU framework (Teaching for Robust Understanding) (2016) that identifies five dimensions for learning: mathematics, cognitive demand, equal access to content, agency, ownership and identity and formative assessment. The mathematical dimension is at the core of the model and the other dimensions are shaped around it. The dimensions do not contain prescriptive "recipes" for teachers but rather guidelines for creating powerful learning environments, that result in students becoming resourceful thinkers and learners. The dimensions provide an analytical tool for the observation and reflection on one's own teaching practice and can be used for designing and evaluating the effectiveness of the intervention and thinking about the next steps in teaching action.

In this contribution, we present a teaching and learning sequence conceived and implemented within the TRU framework and we study to what extent formative assessment strategies were implemented. William and Thompson (2007) discuss five key strategies that may help promoting formative assessment in the classroom: FA1) clarifying and sharing learning intentions and criteria for success; FA2) engineering effective classroom discussions and other learning tasks that elicit evidence of student understanding; FA3) providing feedback that moves learners forward; FA4) activating students as instructional resources for one another; FA5) activating students as the owners of their own learning. Not only the teacher, but also the peers and the student himself/herself may act as agents of formative assessment.

## Research Design

Our exploratory study is based on a teaching experiment, involving a teacher (the author SQ) who is a member of the DIVA community of inquiry. Despite being an experienced teacher, who already took part in teaching experiments concerning formative assessment (Morselli & Quartara, 2023), this represents his first attempt to design a learning activity through the framework TRU. The activity took place in a grade 9 class (18 students) of an upper secondary school with a scientific orientation.

With reference to the mathematical content dimension, which is at the core of the TRU framework, the activity at issue was aimed at the development of algebraic thinking and of argumentative competence, with a strong focus on the interaction between them. As mentioned earlier, what sets DIVA apart is its approach to design, guided by the theoretical tool TRU, and the practice of sharing explicit reflections with teachers in the community. These reflections are often guided by additional theoretical tools that cater to the various dimensions of TRU and the specific content that their proposals intend to cover. In this case, additional theoretical tools refer to algebraic thinking and argumentation.

Algebraic thinking is explicitly linked to Arcavi's conceptualization of symbol sense (1994). The development of symbol sense involves: understanding how and when to use symbols to represent relationships, generalizations, and proofs; being aware that in some cases it is more convenient to
abandon symbols in favor of other approaches; dealing with the dialectic between manipulation and interpretation of symbols; being aware of the possibility of creating symbolic expressions, and being able to create them; being able to select, but also to abandon or change a symbolic representation; being aware of the need to constantly check symbol meanings during problem-solving; being aware of the fact that symbols may play different roles. Concerning the second objective, that is, the development of argumentative competence, we refer to Habermas's characterization of "rational behavior" (Morselli & Boero, 2010), thus identifying three components: epistemic (inherent in the correctness of the argumentative process); teleological (inherent in the problem-solving character of the process, and in the related strategic choices); communicative (related to the comprehensibility and communicative choices of the argumentation). All the specific theoretical references were shared with the teacher before starting the design and implementation.

Once the mathematics dimension was fixed, the design of the activity was structured to take into account the other dimensions of the TRU framework. Due to space constraints, we summarize in Figure 1 the structure of the activity, involving 4 stages, specifying the TRU dimensions motivating the introduction of each stage. We will defer this description to future work. The activity is based on the resolution of an algebraic item selected from the INVALSI national assessment repository GESTINV3 (D14 G10 year 2010). In the first individual stage (10 minutes), students were asked to explore and conjecture around the following open-ended question: "If n is any natural number, what do you get by adding the three numbers 2n+1, 2n+3, 2n+5?". Consequently, they were asked to conjecture the truth value of the following statements: Mario's ("You always get the triple of one of the three numbers"); Luisa's ("You always get an odd number"); Giovanni's ("You always get a multiple of 3"). In the second stage, students were asked to discuss in small homogeneous groups and compare their own conjectures, formulated in the previous stage, with group members, answering the following multiple-choice question: "Who is right? a. All of them, b. Only Mario, c. Only Luisa, d. Only Giovanni." Students were required to come to a consensus on a solution and produce a written argument in which their solution is accompanied by a justification regarding the truth value of each of Mario, Luisa and Giovanni's statements. In the third stage, involving a whole class discussion, the teacher displayed the students' responses on the whiteboard and asked each group to narrate the solutions and arguments they had previously developed, involving the students from other groups as well to ask, comment, and compare strategies. During the discussion, aspects related to the algebraic correctness of the solutions were examined, along with the formulation of arguments, and the role that algebra had played in the diverse solutions and argumentations. Finally, the students were asked to complete a self-assessment questionnaire, with a Google module, containing questions aimed at monitoring the aspects of the five TRU framework dimensions on which the design focused. All the discussions were video-taped and transcribed. All the students' productions were collected.

<footnote n="3">³ https://www.gestinv.it/Index.aspx</footnote>
<figure><img src="image_23.png" /><figcaption>Figure 1: Stages of the activity</figcaption></figure>

## Analytical tool: the indicators

To study the actual implementation of formative assessment strategies (Wiliam and Thompson, 2007), we developed specific indicators to detect the *activation* and *realization* of each of them, declined in the specific case of algebraic thinking and argumentation, and indicators for the effective activation of the strategy. The indicators were theoretically set up by tailoring formative assessment strategies to the specific learning goals of the activity, guided by the theoretical frameworks used as references: symbol sense and rational behavior.

Due to space constraints, here we present indicators for FA1 (*clarifying and sharing learning intentions and criteria for success*) and FA2 (*engineering effective classroom discussions and other learning tasks that elicit evidence of student understanding*). These strategies were selected for being the most related to the design of the sequence and to the theoretical frameworks used to frame the learning goals.

The strategies FA1 and FA2 were categorized based on the specific learning goals they addressed: FA1.1 and FA2.1 pertain to algebraic thinking, while FA1.2 and FA2.2 are associated with rational behavior.
<table><caption>Table 1: Indicators for FA1 and FA2</caption><thead><tr><th></th><th>Indicators for activation</th><th>Indicators for realization</th></tr></thead><tbody><tr><td>FA1.1</td><td>a. <u>the teacher</u> underlines/makes explicit the importance of using symbols to represent relationships, generalizations and proofs<br>b. <u>the teacher</u> points out that in some cases it is more convenient to abandon symbols in favor of other approaches<br>c. <u>the teacher</u> makes explicit the importance of dealing with the dialectic between manipulation and interpretation of symbols<br>d. <u>the teacher</u> underlines the possibility of creating symbolic expressions, and being able to create them;<br>e. <u>the teacher</u> underlines the importance of selecting but also abandoning or changing a symbolic representation<br>f. <u>the teacher</u> underlines the importance of constantly checking symbol meanings during problem solving<br>g. <u>the teacher</u> underlines that symbols may play different roles</td><td>The <u>student</u> shows to be aware of the learning goals and criteria for success concerning algebraic thinking (e.g. mentioning the importance of using algebra to generalize)</td></tr><tr><td>FA1.2</td><td>a. <u>the teacher</u> underlines the importance of providing explanations<br>b. <u>the teacher</u> clarifies the criteria for a good argumentation<br>c. <u>the teacher</u> promotes a reflection on the epistemic component (correctness)<br>d. <u>the teacher</u> promotes a reflection on the teleologic component (strategy to solve the problem, goal-oriented actions...)<br>e. <u>the teacher</u> promotes a reflection on the communicative component (comprehensibility of the solution, ...)<br>f. <u>the teacher</u> promotes a reflection on the role of examples in argumentation</td><td>The <u>student</u> shows to be aware of the learning goals and criteria for success concerning argumentation (e.g. recognizing the need to move beyond numeric examples in proving)</td></tr><tr><td>FA2.1</td><td>The design encompasses activities such as small group work/class discussion/self assessment, aimed at:<br>a. comparing solving strategies and solutions<br>b. reflecting on strengths and weaknesses of the solving strategies /e.g. choice of the formalization)<br>c. making students explicit their solving process</td><td>The <u>teacher</u> poses questions aimed at eliciting evidence of student understanding, with reference to algebraic thinking<br><br>The <u>student</u> provides evidence of his/her understanding, with reference to algebraic thinking.</td></tr><tr><td>FA2.2</td><td>The design encompasses activities such as small group work/class discussion/self assessment, aimed at:<br>a. comparing argumentations<br>b. reflecting on strengths and weaknesses of the proposed argumentations (e.g. role for the numerical examples in proving)<br>c. making students explicit their choices related to argumentation (e.g. use of specific terms)</td><td>The <u>teacher</u> poses questions aimed at eliciting evidence of student understanding and realization of argumentation.<br><br>The <u>student</u> provides evidence of his/her understanding, with reference to argumentation.</td></tr></tbody></table>
Discussion of results

We employ the aforementioned indicators to detect activation and realization of formative
assessment strategies in the teaching and learning sequence. Due to space limitations, we present only
two examples of this analysis.

For strategy FA1 (*clarifying and sharing learning intentions and criteria for success*), we can identify instances of *activation* in the teacher's efforts to clarify the desired learning objectives. The actual *realization* of FA1 can be found in the students' protocols, their contributions during discussions, and answers in the self-assessment questionnaires, reflecting their grasp of the learning objectives. The example we present specifically pertains to FA1.1.

An instance of the *activation* of FA1.1, particularly with reference to indicator *a*, can be seen in the
following excerpt (discussion, min. 26.28):

Teacher (T): Well, it's the power of algebra: the ability to generalize and condense infinite numbers into a single symbolic representation. Indeed, that's how it is. 2n: we've written down all those infinite even numbers in a single expression.

Examples of the *realization* of FA1.1, related to indicator *a*, were observed both during the class discussion and in students' responses to self-assessment questionnaires. In the class discussion, students justified their solutions by highlighting the role played by algebra, prompted by the teacher, as shown in the following excerpt (discussion, min. 02.49 - 03.30):

T: How did we go from 6n+9 to 3(2n+3)?

Mary: We performed factorization

T: What was its purpose?

Mary: To find a number, which is 3k. This factorization became k.

T: Matt, would you like to help Mary?

Matt: We modeled what was inside the parentheses by naming it k, and then we got 3k, which means - well, 3k represents all the multiples of 3.

Further, in the self-assessment questionnaires, answering the question “Where does algebra come into play, and how has it helped you?”, students provided responses like the following: "It helped me generalize parts that would have otherwise required infinite examples." (Elia).

Looking at strategy FA2 (*engineering effective classroom discussions and other learning tasks that elicit evidence of student understanding*), we provide an example of its *activation* for FA2.2, involving all three indicators (*a*, *b*, *c*): the strategy focused on comparing protocols with group responses during collective discussions. It allowed the teacher to address a critical aspect during discussions, aiding students in transitioning from arithmetic to algebra: the role of numerical examples in justifying their solutions to the task.

An example of the teacher's realization of FA2.2, specifically indicator *b*, can be seen in the teacher's interventions during the whole class discussion (min. 24.12), seeking to understand the challenges arising from students' arguments, especially by providing feedback on where to focus attention, particularly regarding the role of numerical examples:

T: Alright. Why? Florin, if you remember, Florin, why did you provide an example??
*Realization* of FA2.2 (indicator *b*) concerns both the teacher, highlighting the goal indicated in FA1.2 *f*, and student intervention related to understanding, as referenced in FA1.2 *f*, can be found in the following excerpts of the discussion (min. 07.35 - 08.02):

T: Do you agree that just one example, for instance, a specific example like they took n=0, which is particular as the smallest number, is enough to prove that Luisa is correct, that is, you always get an odd number?

Phil: Since you can insert infinite numbers, well, it's not enough. But one counterexample is enough to refute the theory.

While showcasing these examples, it's important to note that they represent only exemplar instances of the *activation* and *realization* of the formative assessment strategies, concerning exclusively some specific indicators. In our analysis, we observed that these strategies were activated on several occasions and, further than by the teacher, by a multiplicity of students in the class. This provides a measure of the effective implementation of formative assessment concerning FA1 and FA2 in the teaching and learning sequence, in terms of pervasiveness.

## Conclusions, limitations and further directions

In the present contribution, we have shown how designing a teaching and learning sequence within the TRU framework, where all the dimensions are shaped around the mathematical learning goals, complemented by specific theoretical references (symbol sense and rational behavior), allows us to formulate a tool for analysis and, consequently, to evaluate the effective implementation of formative assessment strategies (Wiliam & Thompson, 2007) in the classroom. Indeed, the study leads us to conclude that the modes of designing the sequence allowed us, firstly, to identify specific indicators to monitor the activation and realization of formative assessment strategies.

Secondly, from an initial analysis using the formulated indicators, as shown in the examples illustrated, we could assess that the strategies FA1 (*clarifying and sharing learning intentions and criteria for success*) and FA2 (*engineering effective classroom discussions and other learning tasks that elicit evidence of student understanding*) were realized in the classroom. Although, due to space constraints, we were able to show traces of the activation regarding only some indicators and with few examples of realization taken from the discussion and self-assessment questionnaire, the comprehensive analysis highlighted the pervasiveness of formative assessment strategies (strategies were activated on several occasions and different students were involved). In further work, we will illustrate our analysis in relation to the other strategies.

A further direction involves generalizing the analytical tool, thus studying whether the development of contextual indicators for evaluating the effective implementation of formative assessment strategies can be transferred to the analysis of other teaching activities with specific learning goals.

## References

* Arcavi, A. (1994) Symbol sense: informal sense-making in formal mathematics. For the Learning of Mathematics, 14(3), 24–35.

* Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1), 5–31.
Morselli, F., & Boero, P. (2010). Proving as a rational behaviour: Habermas' construct of rationality as a comprehensive frame or the teaching and learning of proof. In V. Durand-Guerrier, S. Soury-Lavergne, & F. Arzarello (Eds.), *Proceedings of CERME 6, 6th Congress of European Research in Mathematics Education*, Lyon (France), 211–220.

Morselli, F., & Quartara, S. (2023). “My mind is getting used to always find a better solution process”: Formative assessment and self-regulation in secondary school algebra. In P. Drijvers, C. Csapodi, H. Palmér, K. Gosztonyi, E. & Kónya (Eds.), *Proceedings of CERME 12, the 12th Congress of the European Society for Research in Mathematics Education*, Alfréd Rényi Institute of Mathematics and ERME, 4020–4028.

Schoenfeld, A. H., & the Teaching for Robust Understanding Project. (2016). *An Introduction to the Teaching for Robust Understanding (TRU) Framework*. Berkeley, CA: Graduate School of Education. Retrieved from [http://map.mathshell.org/trumath.php](http://map.mathshell.org/trumath.php).

Schildkamp, K., van der Kleij, F.M., Heitink, M.C., Kippers, W.B., & Veldkamp, B. P. (2020). Formative assessment: A systematic review of critical teacher prerequisites for classroom practice. *International Journal of Educational Research*, 103, 101602. [https://doi.org/10.1016/j.ijer.2020.101602](https://doi.org/10.1016/j.ijer.2020.101602)

Wiliam, D., & Thompson, M. (2007). Integrating assessment with instruction: What will it take to make it work? In C. A. Dwyer (Ed.), *The future of assessment: Shaping teaching and learning*, 53–82. Erlbaum. [https://doi.org/10.4324/9781315086545](https://doi.org/10.4324/9781315086545)
# Evaluating the reliability of a framework for mathematical activities using generalizability theory

Ali Bozkurt<fnref n="1" />, Mehmet Fatih Özmantar <fnref n="2" /> and Sibel Tutan Teskin<fnref n="3" />

**Abstract**

This study investigates the measurement reliability of the Framework for Mathematical Activities (FfMA), developed to assess the quality of activity scripts and their implementations. Utilizing Generalizability Theory, the measurement reliability of scores obtained from the FfMA tool was determined. Data were collected based on a descriptive survey model. In this context, four activity scripts and classroom implementation videos of these texts were requested from each of 20 middle school mathematics teachers. The data were scored independently by three raters using the FfMA tool. The scores obtained from the raters were analyzed using the EDUG 6.1e program. Findings indicate that the measurement reliability of the FfMA tool is considered reliable with a value of 0.78, and it does not fall below 0.70 even in scenarios with the minimum number of raters (2) in D studies. These coefficients suggest that the FfMA tool consistently measures the quality of mathematical activities.

Key words: Mathematical activity script, mathematical activity implementation, generalizability theory, reliability.

**Introduction**

In mathematics education, activity-based teaching holds the potential to influence students' mathematical success and prepare them to be independent and democratic thinkers (Noreen & Rana, 2019). However, to realize this potential, there is a need for high-quality activity scripts and implementations. A review of the literature reveals various frameworks for assessing the quality of mathematics instruction, such as the Mathematical Quality of Instruction [MQI] (LMTP, 2011), Teaching for Robust Understanding [TRU] (Schoenfeld, 2013), Classroom Assessment Scoring System [CLASS] (Pianta & Hamre, 2009), and Framework for Teaching [FfT] (Danielson, 2013). These frameworks are generally designed to determine the overall quality of a lesson and do not provide a detailed evaluation specifically focused on the quality of the instructional activities. On the other hand, studies specifically concerning the mathematical dimensions of activities often limit their focus to particular aspects like cognitive demand, purpose, and materials (Stein & Smith, 1998). However, considering only limited, and often isolated, aspect such as cognitive demand or purpose is insufficient for a comprehensive evaluation, as the quality of a mathematical activity scripts and implementation is influenced by numerous components. In order to fill this gap, Framework for Mathematical Activities (FfMA) (Bozkurt et al., 2023) was developed to be used as a feedback tool and to evaluate the quality of activity text and activity applications.

**Framework for Mathematical Activities (FfMA)**

<footnote n="1">¹Gaziantep University, Faculty of Education, Gaziantep, Turkey; alibozkurt@gantep.edu.tr</footnote>

<footnote n="2">²Gaziantep University, Faculty of Education, Gaziantep, Turkey; ozmantar@gantep.edu.tr</footnote>

<footnote n="3">³Ministry of Education, İstanbul, Turkey; sibell27@outlook.com</footnote>
F/MA is an assessment tool that can be used to determine the quality of activity scripts and implementations by considering them separately. Based on these assessments, it is intended to be used in such a way as to provide users with feedback on the strengths and improvement areas of the mathematical activity text and the performance of the activity implementation. In order for F/MA to have a functional use as an assessment tool, concrete indicators and observable criteria were taken as a basis. In this way, F/MA is intended to serve for reliable scoring as well as valid results. The products that F/MA evaluates are the activity text and its implementation. The activity script is a concrete tool produced as a document found in various sources or prepared by the teacher himself/herself and has observable qualities. Implementation, on the other hand, occurs in the real classroom environment based on the interaction between the student-teacher-content triad and has observable characteristics. The dimensions and components of F/MA are illustrated in Figure 1.

<figure><img src="image_24.png" /><figcaption>Figure 1. Dimensions and components of F/MA (Bozkurt et al, 2023)</figcaption></figure>

As illustrated in Figure 5.1.1, the activity script has a total of 8 components and the implementation has a total of 11 components. Both dimensions, the activity text and the implementation, include an evaluation in terms of mathematical potential. Mathematical potential includes components related to determining the mathematical quality of the activity script and the implementation.

This study examines the measurement reliability of the Framework for Mathematical Activities (F/MA) (Bozkurt et al., 2023), which offers a broader perspective for evaluating activity texts and implementations. Utilizing the approach of Generalizability Theory, the study aims to determine the measurement reliability and generalizability of scores derived from the F/MA tool.

## Method

This study stems from a project which was initiated, funded by TUBITAK (Project Number: #119K773). Spanning two years from 2020 to 2022 (Bozkurt et al., 2022). Within the scope of this study, the reliability study of the F/MA developed by the expert researchers in this project was
conducted. In the literature, different theories have been developed to test the reliability of
measurements obtained from a measurement tool: Generalizability Theory, Classical Test Theory,
Multivariate Rash Model, Item Response Theory, etc. These theories differ according to the purpose
of use, limitations, and how the measurement results are used (Brennan, 2001). Generalizability
studies are organized to determine the source of variability from which measurement errors occur
with a single analysis. Generalizability Theory, the chosen method for testing the reliability of F/MA,
is a statistical theory based on the analysis of variance (ANOVA). It is particularly useful in
measurements involving different sources of error, allowing for the estimation of errors stemming
from these sources and their interactions. Generalizability studies are divided into two types: G
(generalization) studies and D (decision) studies. In G studies, the aim is to determine the variance
components that affect reliability, as well as to generalize the measurement taken from the sample to
the larger population (Brennan, 2001). D studies provide data for researchers to identify candidates
for selection-placement, compare experimental groups, and investigate the relationship between at
least two variables (Arterberry et al., 2012).

Data were collected from different schools and different grade levels during one semester in the 2022-
2023 academic year. Four activity scripts and video records of their classroom implementations were
collected from each of 20 middle school mathematics teachers. Participants were selected through
purposive sampling method (Rai & Thapa, 2015). The teachers were asked to design activities or
adapt an existing activity and implement it in their classrooms. They were also instructed to video
record their implementations in actual settings. The decision regarding which lesson to record and
when was left to the discretion of the teachers. The sample size was within the range of similar G-
theory models and observational measurements (e.g., minimum 8) (Hill et al., 2012). This approach
aligns with the methodological emphasis on the quality and applicability of the data rather than just
the quantity, ensuring a more targeted and relevant evaluation of the FfMA tool's reliability.

The activity scripts and in-class video recordings of the implementation of these scripts were analyzed
by the researchers. At the end of the analyzes, it was seen that in some videos, it was not possible to
obtain healthy data on the student-teacher-content triad. These video recordings and the activity
scripts used in these implementations were excluded from the evaluation. After this preselection
process, 65 activity scripts and corresponding implementation videos were selected for further
analysis. The data were independently scored by three raters using the F/MA. The components in the
F/MA are graded on 4 score types (0: Very low; 1: Low; 2: Medium; 3: High). The scores that can
be produced in an evaluation with F/MA are in the range of 0-24 points as minimum-maximum for
the 8-component activity script dimension. For the 11-component activity implementation dimension,
the minimum-maximum score range is 0-33 points. The scores obtained from the raters were then
analyzed using the EDUG 6.1e program. In this framework, a G (generalizability) study was
conducted following the pattern of "*activity script (a) x component interaction effect (c) x rater (r)*"
and "*implementation (u) x component interaction effect (c) x rater (r)*". This study involved analyzing
variance values for main and interaction effects. Subsequently, a D (decision) study was conducted
to calculate the G coefficients for the reliability of the scores.

Findings

Generalizability of FfMA's Activity Script Dimension
The G (generalizability) study conducted for evaluating the activity scripts in FfMA revealed that the variance component attributed to the activity text source explained 9.7% of the total variance. This indicates that the components in the script tool are capable of distinguishing between different components of the activity script. The variance component estimated for the activity script-component-rater (*a x c x r*) or residual (unobserved or unintended) effect was 0.01736 and this effect explained 47.8% of the total variance. The second-largest source of variance was the interaction between component and rater (*c x r*), accounting for 37.1%. This variance indicates variability in the ratings given by different raters to different components. The interaction between activity script and component (*a x c*), contributing 5.4% to the variance, suggests that the difficulty levels and qualities of the components do not vary significantly from one script to another. Other sources of variance (*c*, *r*, *a x r*) were found to contribute zero or near-zero to the total variance, indicating their minimal impact on the overall variability in this context.

The D (decision) studies, which varied the number of raters and components in different scenarios, demonstrated that an increase in the number of raters leads to an improvement in reliability parameters. Based on the relative error variance, the G coefficient was found to be 0.78. In scenarios where the number of raters was three, and the number of components was eight or more, the G coefficients exceeded 0.80. It was also determined that even when the number of raters was reduced to two, the reliability parameters did not fall below 0.70. These findings highlight the robustness of the FfMA tool's reliability across different scenarios, emphasizing that even with a reduced number of raters, the tool maintains a satisfactory level of measurement reliability. This consistency in reliability, regardless of the number of raters, underscores the effectiveness of the FfMA framework in providing dependable evaluations of mathematical activity scripts.

## Generalizability of FfMA's Activity Implementation Dimension

The G (generalizability) study focused on the evaluation of activity implementations within the FfMA framework revealed that the variance component attributed to the implementation of activities accounted for 5.7% of the total variance. This indicates that the components of the measurement tool are effective in distinguishing between the components of activity implementations. The most significant source of variance was the interaction between implementation, component and rater (*i x c x r*), accounting for 41%. This high level of variance suggests that the scores for the components of the activity implementations varied significantly due to the interaction effect and/or random errors, more than by common effects, from one rater to another and from one component to another. The second highest source of variance was the interaction between component and rater (*c x r*), accounting for 32%. This indicates that there is variability and inconsistency in the ratings given by different raters to different components of activity implementations. Interestingly, the variance component attributable to the raters alone (*r*) explained 0% of the total variance. This can be interpreted as an indication of the raters providing consistent scores, highlighting their reliability and uniformity in evaluating the activity implementations.

D studies were conducted by creating scenarios with varying numbers of raters and components. According to the results derived from the relative error variance, the G coefficient was found to be 0.78. It was observed that when the number of components increased while the number of raters remained constant, the G coefficient also increased. This suggests that the reliability parameters are
affected by the variance in errors, which fluctuate depending on the number of raters and components involved. Even in scenarios where the number of components was reduced, the reliability parameters did not fall below 0.70, even when the number of raters was as low as two. This finding underscores the robustness of the FfMA tool's reliability, demonstrating that it maintains a significant level of accuracy and consistency across various testing conditions. It highlights that the FfMA is a reliable tool for evaluating mathematical activities, providing dependable results even with variations in the number of raters and components used in the assessment.

Discussion

According to the findings of the study, the generalizability (G) coefficient of the reliability of the measurements obtained with the FfMA tool was found to be 0.78. Even in decision studies where the minimum number of raters (two) was used, the reliability coefficient did not fall below 0.70. In the study where the number of raters was 3, G coefficients were above 0.80 in scenarios where the number of components was 8 or more. These coefficients indicate that the FfMA tool can produce consistent assessments in different scenarios (Brennan, 2001; Shavelson & Webb, 1991). This demonstrates that FfMA measures the quality of mathematical activities in a consistent and reliable way.

Another finding of the study is the residual variance effect, which has the highest source of variance. The raters' contribution to the shared variance is zero, indicating their consistency in scoring. This finding shows that the quality of the activities varies from component to component and from rater to rater, and is more affected by the common effect and/or random errors. A similar finding was found in the study conducted by Solano-Flores (2006). In this study, G theory was used to estimate the amount of measurement error in the sources of variance of the instrument dealing with psychometric approaches to testing English language learners. Two groups were measured by giving questions in different dialects. The largest measurement error observed was due to the interaction of student, item and code (residual variance/random variance). The high residual variance may be due to various sources of variance not included in the design as well as the interaction effect (Uzun et al., 2018). Moreover, the decision studies reveal that reducing the number of raters does not significantly compromise the reliability of the tool. In fact, it was found that as the number of raters and components increases, so does the reliability coefficients. This finding is significant as it demonstrates the robustness of the FfMA tool in different evaluation contexts, ensuring that it remains a reliable measure for assessing the quality of mathematical activities, even with variations in the number of raters and components used.

Some of the coefficients obtained from the research were found to be low, which could be attributed to the closeness of the scores derived from the activities included in the study. This is because G coefficients are negatively affected by group homogeneity. According to Generalizability Theory, G and phi coefficients are calculated by relating universe variance to observed variance (Brennan, 2001). In other words, as the similarity between groups increases, the variance value decreases, which in turn could explain why the calculated G coefficients are not found to be high. A similar situation was observed in the study by Ozbası and Arcagok (2021), where they examined students' projects in a fully crossed design (j x p x i) within the framework of generalizability theory, considering jurors, projects, and items. In their research, the homogeneity among the projects led to a decrease in variance, thus affecting the generalizability coefficients in a similar manner. This underlines the
impact of group homogeneity on the reliability measures in generalizability studies, suggesting that while the FfMA tool is reliable, the nature of the activities and the scoring patterns can influence the overall variance and, consequently, the generalizability coefficients.

As a result, considering the studies conducted, the reliability of this measure obtained from FfMA is acceptable. As a result of the decision studies, it was observed that the G coefficient increased more in scenarios where the number of raters was increased while the number of components was kept constant. This study contributes to the literature in terms of determining the factors affecting the reliability of the scores obtained from the evaluation of the quality of activity design and implementation in mathematics education through G theory. Based on our findings, it is suggested to provide teachers with guidance on how to evaluate and score mathematical activities. This can include training or resources that clarify the assessment process within the framework of the FfMA tool. Furthermore, to enhance the reliability of assessments, it could be beneficial to involve peer and expert evaluations of activity scripts and implementations across different student groups. By allowing for a diverse range of assessments, it would be possible to analyze the measurement results within the framework of Generalizability Theory more comprehensively. Such initiatives could not only improve the accuracy of evaluations but also provide valuable insights into the effectiveness of mathematical activities in diverse educational settings.

## Acknowledgments

This research project was supported by Scientific and Technological Research Council of Turkey (TÜBİTAK 1001 no: 119K773)

## References

Arterberry, B. J., Martens, M. P., Cadigan, J. M. & Smith, A. E. (2012). Assessing the dependability of drinking motives via generalizability theory. *Measurement and Evaluation in Counseling and Development, 45*(4), 292-302.

Baykul, Y. (2000). *Eğitimde ve psikolojide ölçme: Klasik test teorisi ve uygulaması*. Ankara: ÖSYM yayınları.

Bozkurt, A., Özmantar M.F., Agaç, G., & Güzel, M. (2022). Developing an Evaluation and Feedback Tool for Determining the Quality of Mathematical Task and Implementation. *Project Report, TÜBİTAK 1001 119K773*.

Bozkurt, A., Özmantar M. F., Agaç, G. & Güzel, M. (2023). *A Framework for Evaluating Design and Implementation of Activities for Mathematics Instruction*. Ankara: Pegem Academy.

Brennan, R. L. (2001). *Generalizability theory*. USA: Springer-Verlag New York Inc.

Danielson, C. (2013). *The Framework for teaching evaluation instrument, 2013* edition. The Danielson Group.
Hill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. *Educational Researcher, 41*(2), 56-64.

Karasar, N. (2008). *Scientific Research Method*. Ankara: Nobel Publishing House.

LMTP (Learning Mathematics for Teaching Project), (2011). Measuring the mathematical quality of instruction. *Journal of Mathematics Teacher Education*, 14, 25-47

Noreen, R., & Rana, A.M.K. (2019). Activity-Based teaching versus traditional method of teaching in mathematics at elementary level. *Bulletin of Education and Research, 41*(2), 145-159.

Ozbası, D. & Arcagok, S. (2021). Examining student projects with Generalizability Theory, *Journal of Theory and Practice in Education, 17*(2), 69-78. doi: 10.17244/eku.1024532

Pianta, R.C. & Hamre, B. K. (2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. *Educational Researcher, 38*(2), 109-119.

Rai, N., & Thapa, B. (2015). A study on purposive sampling method in research. *Kathmandu: Kathmandu School of Law, 5*(1), 8-15.

Solano-Flores, G. (2006). Language, dialect, and register: Sociolinguistics and the estimation of measurement error in the testing of English language learners. *Teachers College Record, 108*(11), 2354-2379.

Schoenfeld, A. H. (2013). Classroom observations in theory and practice. ZDM,45(4), 607–621.

Shavelson, R. J., & Webb, N. M. (1991). *Generalizability theory a primer*. California: Sage Publications.

Stein, M. K., & Smith, M. S. (1998). Mathematical tasks as a framework for reflection: From research to practice. *Mathematics teaching in the middle school, 3*(4), 268-275.

Uzun, N. B., Alıcı, D., & Aktas, M. (2018). Reliability of the analytic rubric and checklist for the assessment of story writing skills: G and decision study in generalizability theory. *European Journal of Educational Research, 8*(1), 169-180.
# Expert design and implementation of effective classroom discussions for formative assessment

Annalisa Cusi¹, Francesca Morselli² and Cristina Sabena³

¹Sapienza Università di Roma, Italy; annalisa.cusi@uniroma1.it

²University of Genova, Italy; morselli@dima.unige.it

³University of Torino, Italy; cristina sabena@unito.it

We present the first results of a long-term study aimed at characterizing an expert design and implementation of effective classroom discussions for formative assessment. For the analysis of the data collected in this study we combine the use of three different theoretical constructs concerning: the expert teacher's roles during classroom discussions; shared attention; formative assessment key-strategies. The presented results concern, on one side, the expert's use of a specific digital technology (an interactive whiteboard) to empower specific teacher's roles to promote shared attention and, on the other side, the effects of the empowered teacher's roles in the activation of specific formative assessment strategies.

Keywords: formative assessment, classroom discussion, teacher's roles, shared attention.

## Introduction and theoretical background

In this contribution we present the first results of a long-term study aimed at characterizing an expert design and implementation of effective classroom discussions for formative assessment (FA).

We conceptualize teacher's expertise by referring to Mason and Spence (1999). Specifically, in our perspective, an expert design and implementation is realized by a teacher who knows-to as well as knows-how "to create suitable conditions and then to direct student attention effectively" (p. 147). This is linked to the teacher's awareness of the fact that "it is so vital for students to have the opportunity to be in the presence of someone who is aware of the awarenesses that constitute their mathematical 'seeing'

In our perspective, classroom discussions are *effective* for FA if they support the activation of *FA key-strategies* (William & Thompson, 2007): (A) clarifying and sharing learning intentions and success criteria; (B) engineering effective classroom discussions and other learning tasks that elicit evidence of student understanding; (C) providing feedback that moves learners forward; (D) activating students as instructional resources for one another; and (E) activating students as the owners of their own learning.

In line with Mason and Spence's (1999) ideas, our hypothesis is that promoting *shared attention* may foster fruitful FA processes. This is in tune with recent studies, developed in the field of mathematics education, on the role of moments of joint attention in fostering students' acquisition of a culturally appropriate meaning of mathematical objects (Shvarts, 2018; Salminen-Saari et al., 2021). To define shared attention, scholars (Shteynberg, 2015, Siposova and Carpenter, 2019, Fredriksson, 2022) stress on the crucial difference between the social-cognitive processes that take place when people act as detached observers of each other (third-person perspective), and the processes in which individuals interact by adopting an engaged attitude towards each other (second-person perspective).
Fredriksson (2022) emphasises that, when shared attention is realised, “a first-person perspective may develop into a we-perspective in which it is not an I, but “a we”, that is attuned to the world” (p. 115). In shared attention, two different beings “find the same attunement with the world” (p. 114) and acknowledge the commonality of their world.

To characterize the expert design and implementation of effective classroom discussions, we have analysed a large amount of data collected during the FaSMEd Project (Cusi, Morselli & Sabena, 2017), during which we carried out teaching experiments focused on the use of connected classroom technologies and Interactive Whiteboards (IWB) to support teachers’ FA practices.

The data analysis has been developed by referring to a theoretical construct useful for interpreting and analysing teachers’ roles, namely the *Model of Aware and Effective Attitudes and Behaviours, MAEAB* (Cusi & Malara, 2013, 2016). The MAEAB construct identifies two main groups of roles that an expert teacher intentionally plays, during a classroom discussion, with the main aim of “making thinking visible” and of stimulating metacognitive reflections. The first group of roles are those that the teacher plays in order to pose him/herself as a model by making visible the hidden thinking, the aims, the meaning of the strategies, and the interpretation of results when facing problems: (1) investigating subject and constituent part of the class; (2) practical-strategic guide; (3) activator of interpretative processes; (4) activator of anticipating thoughts. The second group includes the roles that the teacher plays when he/she stimulates metacognitive reflections to help students become aware of the meaning of the realized activities and of the learning processes themselves: (5) guide in fostering a harmonized balance between the syntactical and the semantic level; (6) reflective guide in the identification of effective practical-strategic models; (c) activator of reflective attitudes and meta-cognitive acts.

## Research method

Within the FaSMEd project, the teaching experiments took place in 36 classes encompassing students from 4th to 7th grade, across two consecutive school years (2014–15 and 2015–16), in three school clusters in north-western Italy. We collaborated with 20 teachers to collect approximately 450 hours of classroom sessions. During the teaching experiments, the role of the expert was played by a researcher, one of the authors, in line with the Italian paradigm of research for innovation (Arzarello & Bartolini Bussi, 1998), which theorizes the elimination of the classical distinction between observer and observed (on one side, the class, including the teacher, and, on the other side, the researcher).

We collected lesson’s video recordings, observers’ field notes and students’ written answers. Video recordings and their transcripts form the data corpus for the part of the study documented in this paper. The transcripts were analysed separately by the three authors. Non-converging elements of the analysis were discussed further so as to reach an agreement. We combined the use of the aforementioned theoretical constructs to study how the expert teacher designs and implements classroom discussions through the support of an IWB to empower the MAEAB’s roles by promoting shared attention that fosters fruitful FA key-strategies. More specifically, the expert’s interventions:

* were analysed according to the MAEAB construct (Cusi & Malara, 2013, 2016);

* were related to the foci of shared attention (Fredriksson, 2022) that they aimed to promote;
* were linked to their effects in terms of the FA key-strategies (Wiliam & Thompson, 2007) activated by means of these interventions.

In this 4-pages presentation we confine ourselves to outline the main results for the analysis. The analysis of a paradigmatic example will be added in the oral communication.

## Results and discussion

The first set of results concerns the expert's use of the IWB to empower specific MAEAB roles to promote shared attention. We found that specific uses of the IWB and other specific expert's actions empowered most of the MAEAB roles: zooming-out and/or scrolling from top to bottom to provide an overall view of the groups' answers; zooming-in to focus on particular answers; scrolling up to focus on elements of the given task; inviting one student to come to the IWB to comment on his answer focusing on both the answer and the text of the task; standing in front of the IWB, reformulating a student's discourse and repeating her/his gestures. These uses and actions promote shared attention on different foci: the task and its elements; the overall distribution of students' answers (collective product); a specific written answer (single product) and its characteristics; the approach taken by a student to solve the task (past thinking process); in current approach and reflection on the task (present thinking process).

This analysis highlighted elements of synergy between the shared attention construct and the activation of the roles introduced by the MAEAB construct:

* the shift from the “I-perspective” to the “we-perspective” (which is an indicator of the role of *investigating subject and constituent part of the class*),

* the intentional communication about a common object of attention (i.e. a representation in the case of the role of activator of interpreting processes, a strategy or an argument in the case of the role of *activator of reflective attitudes and metacognitive acts*, the thinking processes of a student or of the teacher in the case of the roles of *reflective guide* and *practical-strategic guide*),

* the focus on metacognitive processes.

The second set of results concerns the effects of the empowered MAEAB roles in the activation of specific FA strategies. The roles of activator of reflective attitudes and metacognitive acts and of reflective guide, contribute to the promotion of specific FA strategies. For instance, teachers can encourage shared attention on a subset of responses, fostering peer assessment among students and thereby promoting FA strategy D. Additionally, students are encouraged to offer feedback to one another, thus realizing FA strategy C. Teachers may also prompt meta-level reflections on provided answers or the reasoning behind them, encouraging students' self-assessment and thus promoting FA strategy E.

In the same way, we found examples of links between the role of guide in fostering a harmonized balance between the syntactical and the semantic level and FA strategy A, and the roles of operative-strategic guide and activator of interpretative processes and FA strategy E.

This study has two implications: (1) at the theoretical level, the study shows the effectiveness of combining the MAEAB construct and the construct of shared attention to gain insights into the ways
in which the expert teacher may promote FA during classroom discussions; (2) at the pragmatic level, this combination could provide a tool for teacher professional development aimed at promoting teachers' autonomous design and implementation of effective classroom discussions for FA.

## References

Arzarello, F., & Bartolini Bussi, M. (1998). Italian trends of research in mathematics education: A national case study in the international perspective. In J. Kilpatrick & A. Sierpinska (Eds.), *Mathematics education as a research domain: The search for identity* (vol. 2, pp. 243–262). Kluwer.

Cusi, A., & Malara, N.A. (2013). A theoretical construct to analyze the teacher's role during introductory activities to algebraic modelling. In B. Ubuz, C. Haser & M.A. Mariotti (Eds.), *Proceedings of Cerme 8* (pp. 3015–3024). Middle East Technical University and ERME.

Cusi, A., & Malara, N.A. (2016). The Intertwining of Theory and Practice: Influences on Ways of Teaching and Teachers’ Education. In L. English, & D. Kirshner (Eds.), *Handbook of International Research in Mathematics Education 3rd Edition* (pp. 504–522). Taylor & Francis.

Cusi, A., Morselli, F., & Sabena, C. (2017A). Promoting Formative Assessment in a Connected Classroom Environment: Design and Implementation of Digital Resources. ZDM - Mathematics Education, 49(5), 755–767.

Fredriksson, A. (2022). *A Phenomenology of Attention and the Unfamiliar Encounters with the Unknown*. Palgrave Macmillan.

Mason, J. & Spence, M. (1999). Beyond mere knowledge of mathematics: the importance of knowing-to act in the moment. *Educational Studies in Mathematics*, 38, 135–161.

Salminen-Saari, J. F. A., Garcia Moreno-Esteva, E., Haataja, E., Toivanen, M., Hannula, M. S., & Laine, A. (2021). Phases of collaborative mathematical problem solving and joint attention: a case study utilizing mobile gaze tracking. *ZDM - Mathematics Education*, 53, 771–784.

Shteynberg, G. (2015). Shared attention. *Perspectives on Psychological Science*, 10(5) 579–590.

Shvarts, A. (2018). Joint attention in resolving the ambiguity of different presentations: a dual eye-tracking study of the teaching-learning process. In N. Presmeg, L. Radford, W.-M. Roth & G. Kadunz (Eds.), *Signs of Signification. Semiotics in mathematics education research* (pp. 73-102). ICME-13 Monographs. Springer.

Siposova, B., & Carpenter, M. (2019). A New Look at Joint Attention and Common Knowledge. *Cognition*, 189, 260–274.

Wiliam, D., & Thompson, M. (2007). Integrating assessment with instruction: What will it take to make it work? In C. A. Dwyer (Ed.), *The future of assessment: Shaping teaching and learning* (pp. 53–82). Lawrence Erlbaum Associates.
# Enacting assessment accommodations in an inclusive formative classroom practice: The case of color-coding

Angeliki Dikarou<fnref n="1" /> and Chrissavgi Triantafillou<fnref n="2" />

This study explores how a Special Education Teacher in Mathematics (SETM) implements the assessment accommodation of color-coding in a Parallel Support setting. SETM's goal is to support a student with learning disabilities in a grade 8 mathematics classroom. We view formative assessment as a unified classroom practice that involves teachers' actions and students' responses to these actions. The results indicate that the main SETM's actions while enacting color-coding accommodation are a) repeating and extending student's short answers; b) asking student to justify his responses; c) evaluating and validating student's correct responses; d) honoring student's contribution by maintaining his mathematical idea and e) creating a positive and engaging learning environment by frequently rewarding student's responses.

Keywords: Inclusive formative classroom practices, assessment accommodations, color-coding, special education teacher's actions.

## Introduction

The potential of using formative assessment in mathematics classrooms to raise students' learning is well documented in many studies (Andersson, 2020; Heritage & Wylie, 2018). Even though, many studies argue that adapting formative assessment practices could be more effective and inclusive for specific group of learners such as learners with autism (e.g., Ravet, 2013), there is a limited number of studies that explore formative assessment in mathematics classrooms from a special education perspective.

In special education settings, teachers usually implement assessment accommodations to support students with special learning needs (Maccini & Gagnon, 2000). Assessment accommodations are changes made to an assessment procedure (e.g., scheduling, timing, task presentation), that aim to remove barriers and allow students to fully demonstrate their competencies and their abilities (Elliott et al., 1998). Maccini and Gagnon (2000) determined the type of assessment accommodations that special and general education teachers reported while enacting assessment practices. These types of accommodations may include visual tools for task presentation; reference materials such as cue cards or charts of strategy steps; or time extensions on tests.

The current study explores how a Special Education Teacher in Mathematics (SETM) implements a specific type of visual tools such as color-coding in order to support a student with autism. The research question (RQ) is: What are SETM's teaching actions while enacting the assessment accommodation of color-coding in a formative assessment practice?

Color-coding is the use of colors to represent data values on a task. This means that every data value is associated with exactly one color, and vice versa i.e. every color represents a fixed range of data values (Tominski et al., 2008).

<footnote n="1">¹National and Kapodistrian University of Athens; angeliki_ad@hotmail.com</footnote>

<footnote n="2">²National and Kapodistrian University of Athens; chrtriantaf@math.uoa.gr</footnote>
# Theoretical Background

## Socio-cultural perspective

We adopt a socio-cultural perspective since we view teaching and learning as a joint labor process where teachers and students are laboring together to produce knowledge (Radford, 2014). Thus, it is important to explore teachers’ in-the-moment responses to student mathematical contribution. In this study, we analyze SETM’s actions through the Teacher Response Coding (TRC) framework (Van Zoest et al., 2022). Some of the actions included in this framework are *allow* (creates an open space for interaction); *check-in* (elicits student’s self-assessment or understanding); *clarify* (asks the student to make more precise answer); *justify* (gives the student the opportunity to reason on his mathematical idea) and *evaluate* or *validate* students’ responses. Another important aspect of the TRC framework is the degree to which the teachers’ response aligns with students’ ideas and mathematical contributions.

## Inclusive formative assessment practice

Practice in a classroom, is formative to the extent that evidence about student achievement is elicited, interpreted, and used by teachers, learners, or their peers, to make decisions about next steps in instruction (Black & William, 2009). Thus, all studies share the defining characteristic of formative assessment: agents in the classroom collect evidence of students’ learning, and based on this information, adjust their teaching and/or learning (Andersson & Palm, 2017). In this study we use the term of ‘inclusive formative assessment’ (Andersson, 2020, p. 75) where inclusion means that students’ diversity and differences are seen as something natural and valuable. Based on this perspective, students with mild learning disorders are now taught in mainstream classrooms and not in special units and schools. To achieve inclusive formative assessment is a challenging issue. Ravet (2013, p. 961) argues that “inclusive formative assessment can be more successful when teachers abstract themselves from the straitjacket of normative thinking about learning, in order to understand the minds of students who function differently.”

In this study, we view formative assessment as a unified practice that involves teachers’ actions and students’ responses to these actions. We analyze SETM’s actions while enacting a specific assessment accommodation in a Parallel Support environment as well as the outcome of these actions on students’ learning.

## Literature review

A limited number of studies suggest empirically validated approaches for assessing students with learning disabilities in mathematics classrooms. Tay and Kee (2019) study mainstream teachers’ effective questioning and feedback in primary and secondary math and science classrooms that include high-functioning students with autism spectrum disorder (ASD). They identified three important characteristics of effective questioning and feedback for these students: *addressing students’ cognitive needs of* (e.g., precise and direct questions); *taking into consideration their socio-emotional needs* (e.g., affirmative feedback); and *using of supporting structures* (e.g., visual cues). Andersson (2020) documented 39 special education teachers’ views while implementing formative assessment practices in mathematics classrooms. Participants, referred to the potential of formative
assessment for students with learning disabilities as well as to the challenges they faced, while trying
to adjust the learning environment according to students' individual needs.

**Methodology**

**The Greek educational system**

The Greek educational system, based on the current legislation (Law 3699 of 2008, article 6) provides inclusive teaching support programs, such as the Parallel Support (PS) program, for students with learning disabilities (e.g., students with Autism Spectrum Disorder (ASD) or students with intellectual disability). Parallel Support is a co-teaching program where two teachers, a general education teacher and a special education teacher share the instruction for a single student in a single classroom setting. Mavropalias and Anastasiou (2016) explored the features of the Greek co-teaching model of Parallel Support (PS) in several Greek educational regions. Their study revealed that the PS program is similar to the One Teach, One Assist approach where the special education teacher typically sits next to the student with a disability, while the general education teacher delivers the lesson in the traditionally arranged classroom setting. The Special education teachers provide individualized support for these students during lessons in a regular classroom, not only to support them to follow the general education system curriculum, but also to reach their educational needs.

**The context of the study**

The research was carried out in a general education junior high school, during the 2023-24 school year, where one of the researchers works as SETM in a PS program and is responsible for implementing individualized instruction in mathematics, for students with learning disabilities. In this study the research data concerns one of these students, who attends the 8th grade mathematic classes, with an ASD diagnosis. For the needs of this research ethical issues were taking into consideration. SETM from the beginning of the school year had knowledge about important characteristics of the student's learning profile, through the official written diagnosis. This diagnosis, among others, provided useful instructive suggestions that were estimated to favor student's understanding. Specifically, some of these suggestions were that SETM should conduct a combined review of acquired knowledge and implement applications of mathematical skills; systematically pursue student's understanding of mathematical concepts; use information coding (e.g., acronyms, highlighting or color-coding) and positive reinforcement by rewarding student's effort.

**Research data and data analysis**

Research data is drawn from the research diary kept by the SETM, concerning her everyday actions as special education teacher. Data derived from the research diary included written notes of her daily schedule; photographic material from the student's notebook; short indicative dialogues with the students and the classroom teacher, written on field notes during the lesson or during the breaks; her teaching goals and her reflections after the lesson enactment; short reports/updates about the students' learning progress; information about students' daily homework tasks and the difficulties they faced. It also included short discussions with general education teachers, concerning assessment tools. The above information supports SETM to gain a better perception of the student's learning profile, learning needs as well as types of assessment accommodations that appeared to have a positive learning outcome for them.
The analysis of research data was carried out in three steps. Step 1: We traced all episodes in SETM's Research Diary where she implemented the specific assessment accommodation of color-coding in a number of lessons. Step 2: Two episodes were selected in which, the color-coding played a significant role in a problem-solving process. Step 3: In these episodes, SETM's actions were analyzed through the TRC framework (Van Zoest et al., 2022).

## Results

The following two episodes are from the same teaching hour and the teaching chapter covers the calculation of the area of known geometric shapes. The color-coding method was used during the problem-solving process.

### 1st episode

The 1st episode was concerning a textbook homework task (Task 1) assigned by the general education teacher to all students. The problem was asking students to calculate the area of the two roads (brown rectangles) and the lawn (green areas) as presented in Figure 1.

<figure><img src="image_25.png" /><figcaption>Figure 1: The textbook task (Vlamos et al., 2021, p. 125)</figcaption></figure>

At first SETM checked student's notebook to make sure that homework was done. From this check, SETM realized that there was a mistake in student's following response: 1000 - (24 + 20) = 1000 - 44 + 0.48 = 956.48. In the area calculation solution, student added 0.48 (the crossroad area) without any explanation. This prompted SETM to ask student to justify his answer. The student seemed to face difficulties in justifying his answers. SETM decided to copy the shape on student's notebook by using different colors for each road and the lawn area (Figure 2). Specifically, she took the following color-coding steps as appears in her new version Figure 1 (see Figure 2). Step 1: SETM drew the main rectangle (25W x 40L) and defined with green color all the areas that supposed to be planted with lawn, keeping the book's initial color coding, specifying that this was the area to be calculated. Step 2: Orange color was used to mark the horizontal road. Even though student pointed out correctly the largest and the smallest dimension of the orange rectangle, seemed to struggle when was asked to calculate this area. Step 3: Purple color was used for the vertical road. The main purpose of color-coding was to make the magenta rectangle visible to the student. Then, SETM asked again the student to calculate this area where similar difficulties appeared once more. Then SETM started to discuss with the student about the colors that appeared in the final design.
<figure><img src="image_26.png" /><figcaption>Figure 2: Modifying the textbook task by using color-coding accommodation</figcaption></figure>

1 SETM: So far, we have calculated the area of the orange road and the area of the purple road. Can you see what is happening with that little rectangle with the different color? Why do you think the color changes?

2 Student: Because is the purple above the orange. It has two colors [purple and orange]. This [rectangle] is part of the two roads!

3 SETM: Very nice! So practically what does this mean for us? When we calculated the orange area, we calculated the area of the little magenta rectangle for the 1st time, but we calculated the area of this exact same rectangle for the 2nd time when we calculated the area of the purple road. So, in the end it's like we have calculated magenta's rectangle area twice.

SETM summarizes what was done so far and asks student to focus on the color change that appears in the Figure 2 (Line 1). SETM allows student to respond and asks him to justify through the two colors. Student realized that the magenta area appeared twice in calculations during the solution (Line 2). Student came to this conclusion through the observation that the different color in the crossroad rectangle is due to the overlapping of the two colors. Then, SETM gives supportive feedback to student for the observation made, *repeats* and *extends* the whole solution process in detail (Line 3) and finally she relates the solution with the color-coding accommodation and presents it in mathematical terms by *evaluating* and *validating* student's response. Furthermore, she *honors* and *rewards* student's mathematical contribution.

## 2nd Episode

The 2nd episode concerned a geometrical problem that the general education teacher gave to the students as homework. This geometric task (Task 2) referred to the calculation of the area of a specific geometric shape. Student drew the shape in the notebook and solved the problem as appears in Figure 3. Then, the general education teacher asked students to provide an alternative solution. Student could not think of a different way to solve the task. The SETM decided to use color-coding to support student to identify another way to solve the problem.

SETM redesigned the geometric shape and filled with pink color the inner rectangle appeared, as shown in Figure 4.

1 SETM: So, can you tell me what is the geometric shape that you see here?

2 Student: A large rectangle.

3 SETM: What's the area of this large rectangle?

4 Student: 12x8
<figure><img src="image_27.png" /></figure>

In Lines 1, SETM starts with *check-in* student's understanding of the geometrical figure. Student possibly identifies two rectangles in the drawing, the large one and the small one colored in pink (Line 2). Then, SETM asks student to *clarify* his answer and make the relevant calculations. SETM wrote "E1 = ..." on the notebook to *allow* him to move to the next step i.e. to calculate the area of the small pink rectangle.

<table><tr><td>5 SETM:</td><td>So, can you tell me what is the length and the width of this little pink rectangle that was formed?</td></tr><tr><td>6 Student:</td><td>4 [points out the length of the rectangle].</td></tr><tr><td>7 SETM:</td><td>Oh, nice! And how did you find it?</td></tr><tr><td>8 Student:</td><td>At the left and at the right is also 4. There are 3 pieces that makes us 12.</td></tr><tr><td>9 SETM:</td><td>That is because our shape is rectangle, so the opposite sides are... [let the student finish her argument]</td></tr><tr><td>10 Student:</td><td>Equal.</td></tr><tr><td>11 SETM:</td><td>Perfect! So, you made the calculations 4+4=8 and then 12-8=4. And what a nice observation that in this case we have indeed 3 equal parts of 4, that makes us 12. What about the other dimension? Look at the shape above and then calculate the area of this pink rectangle.</td></tr><tr><td>12 Student:</td><td>2. So the area is 2x4.</td></tr><tr><td>13 SETM:</td><td>Finally, to calculate this area [outlines the shape with purple lines], what shall we do?</td></tr><tr><td>14 Student:</td><td>96-8.</td></tr><tr><td>15 SETM:</td><td>Very nice! So, from the area of the large rectangle we will subtract the area of the small one.</td></tr></table>

In line 5 SETM starts focusing on the small pink rectangle, asking him to *name* the dimensions of the small rectangle. In lines 6 to 8 student responds correctly and SETM asks him to *justify* his responses. It seems that color-coding facilitated student to reach the conclusion that the length is divided into 3 equal pieces (Line 8). SETM *validates* this response while mentioning the relevant theory (Line 9). In line 11 SETM provides positive feedback to the student and then *extends* and *repeats* the mathematical process leading to student's correct answers in lines 6, 8 and 10. Finally, in lines 12 and 14 student gives the correct numerical solution to the problem while SETM *validates* student's answer and keeps *honoring* and *rewarding* student's mathematical contribution (Line 15).

## Conclusions

In this paper we explore SETM's actions while enacting a color-coding assessment accommodation in a PS program in a Grade 8 mathematics classroom. These actions constitute parts of an inclusive formative assessment classroom practice. The outcome of SETM's actions was leading to student's understanding. The main SETM's actions while enacting color-coding accommodation are a) repeating and extending student's short answers; b) asking student to justify his responses; c)
evaluating and validating student's correct responses by underlining the mathematical reasoning behind these answers; d) honoring student's contribution by maintaining his mathematical idea. In this way the student could easily follow the whole discussion (Van Zoest et al., 2022); and e) creating a positive and engaging learning environment (Hill & Seah, 2023) by frequently rewarding student's responses and focused observations on the color-coding task presentation. SETM's actions seems to satisfy aspects of Tay and Key (2019) effective questioning and feedback. Specifically, SETM poses direct questions when she was asking student to *name* the dimensions of the rectangle; she provides affirmative and constructive feedback; and uses visual cues in color-coded form.

From our perspective, it is the dynamic nature of formative assessment, which makes this process challenging for special education teachers, as it requires continuous adjustments to create the appropriate inclusive conditions for students with learning disabilities (Andersson, 2020). The assessment accommodations -such as color-coding- create opportunities for the learners to demonstrate their mathematical competence and open ways to assess their abilities and not their disabilities (Elliott et al., 1998). Furthermore, despite the institutional limitations that the Greek educational system poses to SETM's teaching activities, as addressed by Mavropalias and Anastasiou (2016), SETM managed to overcome these limitations and to deliver a positive outcome for the student she is responsible for.

Finally, the limitations of this study include, the limited number of participants, one SETM and one student, and the small and by convenience selected sample of episodes does not allow us to generalize our conclusions. More research is needed concerning the enactment of assessment accommodation in inclusive formative assessment practices.

## References

Andersson, C. (2020). Formative assessment-from the view of special education teachers in mathematics. *Nordic Studies on Mathematics Education, 25*(3-4), 73-93.

Andersson, C., & Palm, T. (2017). Characteristics of improved formative assessment practice. *Education Inquiry, 8*(2), 104-122. https://doi.org/10.1080/20004508.2016.1275185

Black, P., & Willam, D. (2009). Developing the theory of formative assessment. *Educational Assessment, Evaluation and Accountability 21*(1), 5-31. https://doi.org/10.1007/s11092-008-9068-5

Elliott, J., Ysseldyke, J., Thurlow, M., & Erickson, R. (1998). What about assessment and accountability? Practical implications for educators. *Teaching Exceptional Children, 31*(1), 20-27. https://doi.org/10.1177/004005999803100103

Heritage, M., & Wylie, C. (2018). Reaping the benefits of assessment for learning: Achievement, identity, and equity. *ZDM - The International Journal on Mathematics Education, 50*(4), 729-741. https://doi.org/10.1007/s11858-018-0943-3

Hill, J. L., & Seah, W. T. (2023). Student values and wellbeing in mathematics education: perspectives of Chinese primary students. *ZDM-Mathematics Education, 55*(2), 385-398. https://doi.org/10.1007/s11858-022-01418-7
Law 3699. (2008). *Special education and education of people with disability or special educational needs*. Athens: FEK 199A/October 2nd, 2008.

Maccini, P., & Gagnon, J. C. (2000). Best practices for teaching mathematics to secondary students with special needs. *Focus on exceptional children*, 32(5).

Mavropalias, T., & Anastasiou, D. (2016). What does the Greek model of parallel support have to say about co-teaching?. *Teaching and Teacher Education*, 60, 224-233.
[https://doi.org/10.1016/j.tate.2016.08.014](https://doi.org/10.1016/j.tate.2016.08.014)

Radford, L. (2014). On teachers and students: An ethical cultural-historical perspective. In Nicol, C., Oesterle, S., Liljedahl, P., & Allan, D. (Eds.) *Proceedings of the Joint Meeting of PME 38 and PME-NA 36*, Vol. 38, pp. 1-20. Vancouver, Canada.

Ravet, J. (2013). Delving deeper into the black box: formative assessment, inclusion and learners on the autism spectrum. *International Journal of Inclusive Education*, 17(9), 948-964.
[https://doi.org/10.1080/13603116.2012.719552](https://doi.org/10.1080/13603116.2012.719552)

Tay, H. Y., & Kee, K. N. N. (2019). Effective questioning and feedback for learners with autism in an inclusive classroom. *Cogent Education*, 6(1), 1634920.
[https://doi.org/10.1080/2331186X.2019.1634920](https://doi.org/10.1080/2331186X.2019.1634920)

Tominski, C., Fuchs, G., & Schumann, H. (2008). Task-driven color coding. In *12th International Conference Information Visualisation* (pp. 373-380). IEEE.

Van Zoest, L. R., Peterson, B. E., Rougée, A. O., Stockero, S. L., Leatham, K. R., & Freeburn, B. (2022). Conceptualizing important facets of teacher responses to student mathematical thinking. *International Journal of Mathematical Education in Science and Technology*, 53(10), 2583-2608.
[https://doi.org/10.1080/0020739X.2021.1895341](https://doi.org/10.1080/0020739X.2021.1895341)

Vlamos, P., Droutsos, P., Presvis, G., & Rekoumis, K. (2021). *Grade 8 Mathematics textbook*. OECD. Athens, Greece.
Large language models as formative assessment and feedback tools? –
A systematic report

Frederik Dilling

University of Siegen, Germany; dilling@mathematik.uni-siegen.de

This paper discusses the use of large language models (LLMs) for formative assessment and feedback in mathematics education. First, a brief introduction to the research on LLMs in mathematics education is given. Subsequently, the LLM ChatGPT 4.0 is systematically evaluated with regard to the aspects 1) Input and localization, 2) Assessment-quality, 3) Content and form of feedback, and 4) Adaptivity and receiver of feedback. It is shown that ChatGPT has the potential to provide meaningful feedback on mathematical work, but that its use is associated with a number of challenges.

Keywords: Artificial intelligence, ChatGPT, Digital assessment and feedback, Formative assessment, Large language models

Introduction

Artificial intelligence (AI) is currently a highly debated topic. At the latest with the free publication of GPT-3 in November 2022, the discussion has arrived in society. The opportunities and challenges for the educational sector were also quickly addressed. For example, a study by Kung et al. (2023), which found that ChatGPT could pass the three-part American medical licence test (USMLE) without further training, received a great deal of media attention. Opportunities and challenges were also investigated in the field of mathematics education. Wardat et al. (2023) conducted interviews with students and teachers and found that ChatGPT is generally perceived as a useful educational tool, but that it does pose some challenges (e.g. development of misconceptions). Other authors, however, analyzed ChatGPT from a theoretical perspective and through extensive testing. For example, Buchholtz et al. (2023) come to rather negative conclusions on this basis and state that the generative AI ChatGPT is not yet suitable for use in mathematics classes.

While AI has been studied intensively in educational research from a technical perspective for around 10 years (see, e.g. "International Journal of Artificial Intelligence in Education"), mathematics education research has only recently begun to address the topic, as can be seen above. In particular, so-called large language models (LLMs) such as ChatGPT are being considered. LLMs are linguistic models that have been trained with a huge amount of text data and are intended to simulate communication. With the help of probability trees, answers to user requests (so-called prompts) are generated. Although the system was trained for linguistic knowledge, it can also contain rational knowledge from the training data (Petroni et al., 2019). However, knowledge databases are not accessed for the answers; the "knowledge" comes solely from the trained linguistic model, which can also result in the output of incorrect information. Kasneci et al. (2023) explain in the context of LLMs in education:

"Large language models can help teachers to identify areas where students are struggling, which adds to more accurate assessments of student learning development and challenges. Targeted instruction provided by the models can be used to help students excel and to provide opportunities for further development." (p. 3)
Initial studies on formative assessment with LLMs have already been carried out. Moore et al. (2022) used a fine-tuned GPT-3 model to evaluate student answers in chemistry education and concluded that it is a powerful tool to assist teachers in the quality of their evaluations of students. Zhu and Liu (2020) found that LLMs can support high school students in scientific reasoning in the area of climate activity. Sailer et al. (2023) observed in a teacher education program that the use of LLMs leads to better justifications of diagnoses of students' learning difficulties.

In this article, the opportunities and challenges of ChatGPT as a formative assessment and feedback tool in mathematics education will be discussed. For this purpose, the LLM is tested and analyzed in detail against the background of a previously described framework based on Fahlgren et al. (2021).

**Digital formative assessment and feedback – A framework for the analysis**

The basis for the analysis framework in this article is the survey report by Fahlgren et al. (2021) on technology-rich assessment in mathematics. In this report, the research or development projects STACK, STEP and SMART are analyzed and compared against the background of selected categories. The categories considered include localization, receiver, content and form, and adaptivity. Localization refers to whether the assessment and feedback takes place on a micro-level (e.g. task level) or a macro-level (e.g. overall performance). The receiver of feedback is the person to whom the feedback is directed (e.g. student, teacher). The content of feedback can also differ (e.g. right/wrong, hints or error information, worked out examples) as well as the form in which it is presented (e.g. language, pictures). Adaptivity refers to the extent to which aspects of the student response appear in the feedback.

For the analysis of ChatGPT as a potential feedback and assessment tool in this article, the above categories were slightly modified. For this purpose, the aspect of localization was expanded to include the possibilities of input by the user. The adaptivity and receiver categories were combined into one analysis aspect. The aspect of assessment quality, which is important to examine in the field of generative AI, has been added. The analysis in this article is therefore based on four aspects:

* Input and localization

* Assessment-quality

* Content and form of feedback

* Adaptivity and receiver of feedback

**Evaluation of the large language model ChatGPT**

The LLM analyzed in this article is the latest version of ChatGPT 4.0 at the time of the analysis. Extensive testing was carried out with this LLM in January 2024. The Wolfram plugin, which establishes a connection to the Wolfram Alpha computer algebra system (CAS), was used for all user requests. The testing was based on selected tasks in the field of linear algebra and analytical geometry from the publicly accessible mathematics secondary school examinations in North Rhine-Westphalia (Germany) (see https://www.standardsicherung.schulministerium.nrw.de/cms/zentralabiturgost/faecher/fach.php?fach=2). The tasks include, for example, an inner-mathematical task in which distances between two points had to be determined and a third point had to be chosen so that a right angle is formed, or an application task with a real-world context in which it had to be justified that the base of a pyramid lies in one of the coordinate planes, it had to be shown that three given corner
points of the pyramid approximately form an equilateral triangle with a certain edge length, and the plane in which these three points are located had to be determined. The tasks were translated into English and formulas were converted into LaTeX notation. In addition, sample solutions with different types of errors were created, which were then entered into ChatGPT with different prompts. The conversations with ChatGPT formed the data basis for the analysis.

The results of the analysis with regard to the four aspects mentioned above are presented below and explained at selected points with examples from the testing. For readability, the explication is limited to one task in which a linear system of equations is to be solved. It should be emphasized at this point that although the analysis framework used was precise and comparable with other studies, the data collection was rather exploratory and not systematic, which is why this is not a scientific study, but rather an experience report.

## Aspect 1: Input and localization

As described above, LLMs like ChatGPT are developed especially for processing text and are intended to simulate communication. Therefore, the input and also the output is mainly symbolic as text. If the Wolfram plugin is switched off, it is also possible to input images or sound recordings, although the analysis options in this case are very limited beside the recognition of text on the images or in the recordings.

Mathematical formulas can be entered in any programming language or as a kind of pseudo-code. In this experiment, formulas were entered as LaTeX codes. Various external software is available that automatically converts handwritten formulas or texts into equivalent LaTeX codes, which can then be copied into the text input field of ChatGPT.

To enable assessment and feedback by ChatGPT, both the task and the user's own solution must be entered. To complete the prompt, it is also necessary to explain what is to be done in relation to the task and the solution (e.g. "Can you tell me if this is correct?"). Characteristic for an LLM such as ChatGPT is the possibility to ask follow-up questions after the response of the system, which then also changes the feedback (e.g. "Please tell me where exactly the error occurs."). The further conversation automatically includes the previous requests and answers.

However, retaining the context of a conversation is only possible to a certain extent. The number of tokens that can be used (8192 tokens for ChatGPT 4.0 at the time of analysis) limits the number of analysis units included, whereby according to ChatGPT, one token corresponds to approximately four characters. This means that the localization of the assessment and feedback provided by ChatGPT is more likely to be at the micro level. Although several tasks and solutions can be included, a structured modeling of learning paths over a longer period of time is not possible.

## Aspect 2: Assessment-quality

In terms of mathematical correctness, the detailed testing revealed that many of the calculated results were correct and that the feedback on the user's solution was also correct on this basis. The Wolfram plugin establishes the connection between the LLM and the CAS Wolfram Alpha. This means that potentially all operations that can be performed by a CAS (e.g. transformation of equations, calculation of derivatives and integrals) can also be used by ChatGPT. Nevertheless, ChatGPT may
misinterprets information when analyzing the task and thus sends an incorrect request to Wolfram. It is also possible that ChatGPT misinterprets the results from Wolfram and thus provides incorrect feedback to the user. However, in the responses from ChatGPT it is made transparent which requests were sent to Wolfram and which responses were given. This means that errors can be quickly identified in most cases. Errors also frequently occur if the Wolfram plugin is not used. This can be problematic, as the reasoning around the incorrect calculations can still be plausible and therefore potentially not recognized as wrong by non-experts (Buchholtz et al., 2023). It is therefore recommended to always explicitly state in a request that the Wolfram plugin should be used.

In general, ChatGPT has proven to be quite reliable in processing math problems, even when real-world contexts occur. However, there are actually no reliable figures on mathematical correctness of ChatGPT responses. Some uncertainties occur in the answers of ChatGPT when conceptual questions are asked (e.g. "What is a probability?"). Problems also arise when mathematical processes outside the capabilities of a CAS are requested. For example, ChatGPT is comparatively unreliable at outputting mathematical proofs or evaluating given proofs. This is mainly due to the fact that LLMs are not mathematically logical systems. Although proofs of classical mathematical theorems can be generated, circular reasoning or incorrect derivations often occur.

## Aspect 3: Content and form of feedback

The feedback that ChatGPT provided on the user's solutions in the testing had different contents. If no further information on the desired content is provided (e.g. only asking "Can you tell me, if my solution is correct?"), ChatGPT usually presents the correct solution and uses this as the basis for judging whether the user's solution is correct or incorrect. Figure 1 shows an example in which a system of linear equations consisting of the equations $4x + 2y - 3z = 8$, $2x - 3z = 2$ and $6x + 2y - 5z = 10$ had to be solved. The system of equations was solved by hand on paper, whereby a transformation error was deliberately included in the first step. The entire solution was converted into a LaTeX code and entered together with the question "I have calculated like this. Can you tell me if this is correct? Use the Wolfram Plugin for your calculations.". The response from ChatGPT can be seen in Figure 1. The Wolfram plugin was used to calculate the solution, and this was displayed in the response. In addition, feedback was given that the calculated solution differs from the user's input.

<figure><img src="image_28.png" /><figcaption>Figure 1: Right-Wrong feedback by ChatGPT</figcaption></figure>

When determining the correct solution, the solution process is often presented directly as a kind of sample solution and provided with some explanations. However, this feedback can also be deliberately generated using suitable prompts, e.g. "Can you solve the task step by step and explain it?". Figure 2 shows such a query in relation to the task above. ChatGPT first writes down the system of linear equations to be solved and explains that this can be solved using the method of Gaussian
elimination. The subsequent solution process is divided into four steps. In the first step, the augmented matrix is written down. The second and third steps consist of transformations that convert the matrix into a triangular form. In the fourth step, the concrete values for x, y and z are determined by substitution. The solution is described in relatively great detail and some instructions are given on why the individual steps should be carried out.

<figure><img src="image_29.png" /><figcaption>Figure 2: Solving and explaining a task step by step by ChatGPT</figcaption></figure>

In most cases, an explicit prompt is required for a specific feedback content. For example, it is possible to ask at which point the error occurs. ChatGPT can specify this sometimes more and sometimes less precisely and correctly. One of the reasons for this is that the Wolfram plugin does not output the solution path to ChatGPT, but only the final results – against this background, the solution path must therefore be checked by ChatGPT itself.

In Figure 3, for the example above, the user asks where exactly the error is located in the calculation. ChatGPT then checks whether the row operations from the first step are correct and determines that an incorrect equation has been set up in the third line. This is actually the row in which the error was deliberately inserted for the testing. However, when determining the correct equation, ChatGPT makes a mistake, too, and states that the equation $0 + 2y - z = 6$ is created by double subtracting the second line from the third line. This operation actually results in $2x + 2y + 1 = 6$, which does not help to solve the system of equations.
<figure><img src="image_30.png" /><figcaption>Figure 3: Locating the error by ChatGPT</figcaption></figure>

Many other feedback contents can be generated using suitable prompts. For example, a mathematical process that is related to the task can be explained again. In relation to the task above, for example, you can ask how linear systems of equations can be solved. In a test, ChatGPT has listed different methods (graphical, substitution, elimination, Gaussian matrix elimination, Cramer's rule, software or calculators), briefly explained them and compared them according to the criteria "applicable for" and "best for".

In addition, further exercises including sample solutions can be generated. For this purpose, certain criteria can also be specified for the task, e.g. that only integers should occur in the matrix or that the task is integrated into a real context. A query for the above task resulted in an application, that concerns the relationship between costs and the manufacture of three different products in a company. This is certainly an authentic context in the field of linear equations. However, the task resulted in a system of linear inequalities. From a didactical perspective, it is questionable whether this is a good subsequent task for students struggling with solving systems of linear equations.

Finally, it should be emphasized at this point that it is also possible to deliberately avoid certain feedback content in ChatGPT. This is particularly interesting with regard to the output of the solution. For example, prompts can be formulated in which it is specified that the solution should not be given to the users under any circumstances, but only hints should be provided to help them solve the task or find the errors themselves. A customized version of ChatGPT called "Sostratic Tutor" is already provided specifically for this purpose. If the user asks the tutor how to solve the system of equations mentioned above, the tutor will ask suitable questions in order to help the user actively work on the task step by step (e.g. "Can you identify how many variables and how many equations are presented in this system? And why is this information important for solving a system of linear equations?").
In addition to the content of the feedback, ChatGPT also allows different forms of presentation. The focus is on feedback in written form. ChatGPT uses in particular mathematical terminology, which is characterized by the corresponding mathematical terms. However, isolated elements of colloquial language can also be identified in the conversation. In relation to the tasks, algebraic expressions can be found at many points. While the input by the user is carried out using LaTeX codes or similar codes, which are somewhat confusingly displayed, the formulas in the ChatGPT responses are easy to read (Figures 2-3). In addition to language feedback, ChatGPT can also output iconic representations. For plotting the above linear equation system, for example, it can use the Wolfram plugin and display an appropriate three-dimensional graphic of three intersecting planes.

## Aspect 4: Adaptivity and receiver of feedback

The intensive testing has shown that the level of adaptivity of ChatGPT can be very high. For example, the feedback always refers specifically to the task set at the beginning. Instead of giving a general description of the solution procedure, the solution path can be presented step by step according to the task. When checking the user's own solution, ChatGPT can even localize errors in the solution path to a certain extent and is not limited to comparing the final results with the results determined by the Wolfram plugin.

In addition to adaptivity in relation to the task and the solution to be checked, suitable prompts can also be used to make adjustments in relation to the user's characteristics. For example, a studied mathematician should receive different feedback than a student in secondary school. At the beginning of a chat or in the settings of the ChatGPT account, relevant information can be entered. The more detailed the information is, the more accurately ChatGPT can take it into account in the conversation. For example, language difficulties of the user can be pointed out so that responses use simple words and short sentences. It can also be emphasized as positive that users can work with ChatGPT in the language in which they feel most confident regarding mathematics.

## Conclusion

The previous analysis has shown that ChatGPT already offers a remarkable amount of potential for formative assessment and feedback in the field of mathematics. This is in line with the results already obtained in studies outside mathematics (Moore et al., 2022; Zhu & Liu, 2020; Sailer et al., 2023). The right prompting has proven to be an important success factor for appropriate feedback. The prompt largely determines the form and content of the feedback, the correctness of the performed calculations and the extent to which the response is adapted to the feedback receiver.

However, a number of challenges remain. Probably the most important one concerns the mathematical correctness of the calculations performed by ChatGPT, which is not guaranteed. This is particularly problematic because learners often do not have the competencies to recognize the errors. Students therefore need well-developed reflection skills, a critical handling of the system and close support from the teacher.

With regard to the analysis categories according to Fahlgren et al. (2021), the strengths of ChatGPT lie in particular in the adaptivity of the responses in relation to the task and the feedback receiver (Aspect 4) as well as the content and form of the feedback (Aspect 3). The possibilities for input are largely restricted to text and localization is limited to the micro level (Aspect 1). The assessment
quality is high for numerical and symbolic calculations as considered in this analysis, but there is
generally no certainty for the correctness of the responses (Aspect 2). Thus, LLMs such as ChatGPT
cannot replace the assessment and feedback systems developed specifically for learning mathematics
- but if used appropriately, they can be suitable additions. The future will show how the development
of generative AI will progress and how this will affect the opportunities and challenges identified
above.

References

Buchholtz, N., Baumanns, L., Huget, J., Peters, F., Pohl, M., & Schorcht, S. (2023).
Herausforderungen und Entwicklungsmöglichkeiten für die Mathematikdidaktik durch generative
KI-Sprachmodelle. *Mitteilungen der GDM*, **114**, 19–24.

Fahlgren, M., Brunström, M., Dilling, F., Kristinsdóttir, B., Pinkernell, G. & Weigand, H.-G. (2021).
Technology-rich assessment in mathematics. In: A. Clark-Wilson, A. Donevska-Todorova, E.
Faggiano, J. Trgalova, & H.-G. Weigand (eds.), *Mathematics Education in the Digital Age* (pp.
69–83). Routledge.

Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., ... & Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. *Learning and individual differences, 103*, 102274.

Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., ... & Tseng, V. (2023). Performance of ChatGPT on USMLE. *PLoS digital health, 2*(2), e0000198.

Moore, S., Nguyen, H.A., Bier, N., Domadia, T., & Stamper, J. (2022). Assessing the quality of student-generated short answer questions using GPT-3. *Proceedings of EC-TEL 2022*, 243–257.

Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., & Riedel, S. (2019).
Language models as knowledge bases? *arXiv preprint arXiv:1909.01066*.

Sailer, M., Bauer, E., Hofmann, R., Kiesewetter, J., Glas, J., Gurevych, I., & Fischer, F. (2023).
Adaptive feedback from artificial neural networks facilitates pre-service teachers’ diagnostic
reasoning in simulation-based learning. *Learning and Instruction, 83*, 101620

Wardat, Y., Tashtoush, M., AlAli, R., & Jarrah, A. M. (2023). ChatGPT: A revolutionary tool for teaching and learning mathematics. *EURASIA Journal of Mathematics, Science and Technology Education, 19*(7), em2286.

Zhu, O.M., & Liu, H.L. (2020). The effect of automated feedback on revision behavior and learning gains in formative assessment of scientific argument writing. *Computers & Education, 143*, 103668.
# From automatic diagnosis to lesson plannings: how teachers use elements of a digital formative assessment tool

Eumann, Anica; Klingbeil, Katrin and Barzel, Bärbel

University of Duisburg-Essen, Germany; anica.eumann@uni-due.de

As the effectiveness of formative assessment depends on the form of implementation in school lessons, we investigate in this study which elements of the digital formative assessment tool SMART teachers use for which purposes. In an exemplary qualitative analysis of two teacher interviews, we found that teachers use the given teaching suggestions and materials for designing their upcoming lessons. Furthermore, they use the test items and didactical information about (mis)conceptions to professionalize themselves by gaining deep insights into students' thinking.

Keywords: formative assessment, digital tool, algebra lessons, teacher professionalization

## Introduction

Empirical studies showed that formative assessment (FA) may have positive effects on students' learning, depending on the subject and the concrete form of implementation (McLaughlin & Yan, 2017). For this reason, Schütze et al. (2018) state a high need for research to find out more about different types of realization of FA in classroom practices.

In this paper, we present first results of a study examining the concrete use of a digital FA tool by teachers in algebra lessons in secondary schools in Germany. The tool used in this study is SMART (Specific Mathematics Assessments that Reveal Thinking) that results from a project of the University of Melbourne and is currently being adapted to German speaking countries. This tool aims at giving precise diagnosis of students' thinking and possible misconceptions for specific topics.

## Theoretical Background

FA is defined as an activity in which “evidence about student achievement is elicited, interpreted, and used by teachers, learners or their peers, to make decisions about the next steps in instruction”. (Black & William, 2009, p. 9)

In this process, the teacher as a decision maker plays a very important role. In their model of technology enhanced FA, Cusi et al. (in print) differentiate four areas of teachers' practice in FA (sharing goals and criteria, designing and implementing learning activities, fostering the quality of feedback, involving students in peer- and self-assessment) which they combine with the three phases of preparing lessons (pre-paration, paration and meta-paration) as well as with three functionalities of technology in the process of FA (communicating, analyzing and adapting).

The digital tool SMART that is used in this study consists of about 130 multiple-choice-tests in several mathematical topics that elicit students' understanding of mathematical issues by focusing on their conceptual knowledge. Thus, it allows communication through and with technology because information is displayed and submitted but the user may also interact with the elements of the tool. The tool carries out an advanced analysis on the basis of response pattern that is shown in individual stages of understanding and possible misconceptions displayed in the automatic evaluation for the teacher. Hereby, it allows teachers to get an insight into students' thinking. Adaptation in this tool
rests passive because it proposes teaching suggestions and materials for further instruction to the teacher. At least, they decide whether and how they use this supply (Price et al., 2013).

## Research Question & Methods

As we have seen, working with SMART demands to a high extend teacher activities. This leads to our research question: How and wherefore do teachers use the components of SMART after the implementation of a test? In this paper, we will focus on the second aspect.

To answer this question, we interviewed teachers after having used the tool in their lessons for the first time. These interviews are part of the project SMART[alpha] in which we investigate in a control-group-design the thinking of students and the development of FA competencies of teachers while working with the tool. In this interview, we followed a guideline consisting of two main issues of discussion: the teachers' impression of the test results and the consequences they draw from it for the upcoming lessons.

In the following, we present first results to this research question, arising from an exemplary analysis of two teacher interviews. These are male teachers that teach in North-Rhine-Westphalia in Germany at two different secondary schools in grades 8 resp. 7. Both teachers were picked from a group that did not participate in a professional development program. The only hints for use provided were on a technical level in the handling of the tool. Thus, they worked with the SMART tool independently and autonomously in their classes and used the automatic diagnosis in their sole discretion for their further lesson plannings.

The data analysis is carried out by a qualitative content analysis following Kuckartz (2018). Hereby, we developed a deductive-inductive category system, in which the deductive categories arise from the four areas of FA practice as well as the three phases of preparing lessons (see Cusi et al., in print).

## Results

The data analysis led to three essential aspects of using the tool which will be explained and illustrated by examples.

The first aspect, the *lesson design*, is realized on three different levels that are part of the three deductive categories arising from the phases of preparing lessons. On the level of *pre-paration*, Teacher 1 explains that in the future he will treat typical errors and misconceptions more explicit. Teacher 2 reports that he will use the given material much earlier in future lessons. On the level of *paration*, Teacher 1 reports on the one hand that he uses particular teaching material unchanged and on the other hand that he adapted information on misconceptions and some questions of the test items to discuss them in the classroom. Here, he focusses primarily on the results of the whole class. In addition, teacher 1 also develops new ideas for teaching as for example to work out strategies and rules with the students that might help to avoid typical errors. In contrast, teacher 2 reports that he used the material only in the form in which it is contained in the tool. An adaption or development of new ideas does not take place. Apart from these results, the interviews with both teachers also show that the tool encourages them to reflect their own teaching, which is part of the level of *meta-paration*: "well, as I now knew a bit what is in the tool, and payed a bit more attention to it, also in discussions in class, I noticed more." (Teacher 1).
This statement also indicates the second aspect of use, the *professionalization*. This is shown on two levels arising from two inductively developed categories: first the level of reflection of one's own thinking and practice and second on the level of an intentional knowledge acquisition. Both teachers describe trying to find explanations for particular answers in individual students' results, which is a form of *reflection on their own thinking and practice*. Teacher 2 explicitly names that he is aware of these reflection processes: "Also, ehm, I felt catched, to be honest, so that means, objectively spoken, I reflected obviously." In a similar way, Teacher 1 describes a non-intended professionalization as an "aha-effect" concerning his own language practice in classroom. He formulates the consequence that in future lessons he will pay more attention in the class to being a role model in the use of language and to pay more attention to the students' use of language as well.

Teacher 2 also shows a process of reflection in his work with the teaching suggestions and the information about the levels of understanding and misconceptions. He reports that first he was a little bit annoyed by the length of the texts, but then admits: "I would also say afterward, that I have taken some time to look at it, I think it's great that there is some didactical background. Ehm, I have to admit, that I also took a lot with it." This citation also shows the professionalization on the second level of *intentional knowledge acquisition*. Teacher 1 also mentions this very specifically in the report that he initially had very few ideas about misconceptions, but in the end, recognized them in his own lessons. The most intensely the aspect of knowledge acquisition becomes obvious in the end of the interview with Teacher 2 when he states that the work with the tool was very helpful for him on different levels and that he would use the tool once again but with constraints. The decision to work with the tool depends for him on two factors: the subjective relevance as well as his own pedagogical content knowledge of the concrete topic. This leads to the conclusion that he actively and intentionally uses the tool to gain new knowledge.

The third aspect, the *exploration of the automatic diagnosis*, also arises from inductively developed categories. On the one hand, teacher 1 tries to retrace the automatic evaluation by trying to find connections between different response options in the items and the information about the stages of understanding and misconceptions. On the other hand, he notices that some students do not show the results he would have expected so that he develops explanations that take into account students' thinking: "Well, I found that very fascinating. [...] that you can somehow understand what could be the way of thinking that a student had." Teacher 2 also recognizes a high discrepancy between the results of the automatic diagnosis and his own observations in classrooms. This is why he looks at some individual answers given by students to comprehend the automatic analysis. He looks deeply into the content of the items and the didactic information which allows him to understand that it is not the number of correct answers that leads to a certain stage of understanding but the type of answers given. Thus, he also gains deep insights into students' thinking that he tries to put into relation with his own practice in classroom.

All in all, we find that both teachers put into effect similar activities while working with the tool that differ clearly in their concrete specification. While Teacher 1 focuses on the teaching suggestions and materials to design upcoming lessons by planning new impulses and developing new ideas out of the given materials, Teacher 2 concentrates on the test items and the didactical information to understand
the automatic diagnosis, reflect his own practices and educate himself on a didactical level. But in
both cases, we can see an active change of lesson design and a process of teacher professionalization.

Discussion & Outlook

The results give a hint that the teaching materials are actively used in mathematics lessons but that we have to differentiate between an unmodified and an adapted use. Moreover, the case of Teacher 1 shows that the test items as well as the didactical information about students' (mis)conceptions are used to plan and reflect lesson practices. Particularly, the information is able to encourage teachers to develop their own new ideas of lesson activities. We have also seen that the test items as well as the teaching suggestions and materials support teachers in gaining new pedagogical content knowledge and reflecting their own thinking and practice so that the work with the elements of the tool may contribute to teacher professionalization.

Didactically deep founded diagnostic items as well as didactical background information and fitting
teaching suggestions and material therefore seem to be important elements of digital FA tools that
teachers use intensively. It is to be underlined that the passive adaptation of SMART is sufficient for
such an intensive use.

The presented study nevertheless underlies some restrictions. At the moment of the interviews, the
teachers temporarily had a restricted access to the results of the diagnosis so that they had to
reconstruct some aspects based on their memory. Moreover, these are all only self-reported practices
where it is impossible to check their real implementation. But this also offers a new research
perspective to a project in which teachers will be attended in their work with SMART and the use
and implementation of the tool will be examined by lesson videography.

References

Cusi, A., Aldon, G., Barzel, B. & Olsher, S. (in print). Rethinking teachers’ formative assessment practices within technology-enhanced classrooms. In B. Pepin, G. Gueudet & J. Choppin (Eds.), *Handbook of Digital Resources in Mathematics Education, Ed 1*.

Black, P. & William, D. (2009). Developing the theory of formative assessment. *Educational Assessment, Evaluation and Accountability, 21*(1), 5–31. https://doi.org/10.1007/s11092-008-9068-5

Kuckartz, Udo (2018). Qualitative Inhaltsanalyse. Methoden, Praxis, Computerunterstützung. Beltz Juventa.

McLaughlin, T. & Yan, Z. (2017). Diverse delivery methods and strong psychological benefits. A review of online formative assessment. *Journal for Computer Assisted Learning, 33*, 562–574.

Price, B.; Stacey, K.; Steinle, V. & Gvozdenko, E. (2013). SMART ONLINE ASSESSMENTS FOR TEACHING MATHEMATICS. *Mathematics Teaching, 235*, 10–15.

Schütze, B.; Souvignier, E. & Hasselhorn, M. (2018). Stichwort – Formatives Assessment. *Zeitschrift für Erziehungswissenschaften, 21*(4), 697–715. https://doi.org/10.1007/s11618-018-0838-7
# Peer assessment in an undergraduate geometry course: Fostering proof competency in teacher students

Yael Fleischmann<fnref n="1" />, Antoine Julien<fnref n="2" /> and Alexander Schmeding<fnref n="1" />

Proofs are integral to mathematics as a science, but they are difficult to learn and provide challenges for instruction. This is a particular problem in teacher education, where proofs are a topic that many students experience being disconnected from teachers' day-to-day work in schools. In this note, we report on a course development and research project exploring the use of peer assessment as a tool to foster proof competency in pre-service teacher students.

Keywords: Assessment, teacher education, mathematics education, proof competency, mathematical proofs, peer assessment

## Introduction

Proofs are both a defining product of mathematicians' activity and a notoriously difficult topic for undergraduate students of mathematics. The usual mathematics curriculum in schools puts little emphasis on proofs, arguments, and formal explanations which is a source of tension for the design of courses for pre-service teachers (PSTs). On the one hand, students need to acquire competency in producing and understanding proofs and arguments; on the other hand, the level of formality of tertiary education proofs is often viewed by PSTs as irrelevant to the practice of teaching in schools.

In this article, we present a course development project aimed at training PST's proof competency and changing their attitudes on the topic. Our method to attain these goals is to use peer assessment. The study detailed in this paper is set to commence in early 2024. Note that this paper therefore will not feature results or data from this ongoing study. We anticipate sharing preliminary findings at the FAME conference. The design for the study draws inspiration from an earlier study which we conducted with the aim to improve computational skills of PSTs through peer assessment. Our overarching aim is to develop peer assessment as a tool to enhance *proof competency* (refer to the theory section for the definition of this term used throughout the paper) of the PSTs. The present article more modestly restricts to the following research questions:

1. To what extent does peer assessment contribute to the improvement of proof understanding and proof construction skills among PSTs participating in an undergraduate geometry course?

2. How effectively can PSTs assess the clarity and logical soundness of proofs and arguments generated by their peers during a peer assessment activity?

To this end, we incorporate peer assessment into a geometry course for teacher students. In the implemented activity, the PSTs are asked to evaluate each other's mathematical argumentation. Our interest here lies in the effects of peer assessment on proof understanding and construction abilities of the PSTs. Note that the evaluation of mathematical arguments is typically an unfamiliar task for PSTs in Norway.

<footnote n="1">¹Norwegian University of Science and Technology, Trondheim, Norway;
yael.fleischmann@ntnu.no, alexander.schmeding@ntnu.no</footnote>

<footnote n="2">²Nord universitet i Levanger, Norway; antoine.julien@nord.no</footnote>
**Theory and background**

Assessment and feedback can be effective tools teachers can use to promote students' learning (Hattie, 2008). For this study we are interested in peer assessment as a tool. Topping (1998, p. 250) defines peer assessment as being an arrangement in which students evaluate the work of peers of similar status. In a previous study (Julien, Romijn, Schmeding 2023) we investigated how peer assessment, in particular giving and receiving of feedback, enhanced mathematical knowledge of PSTs. There, assessment activities which involved giving feedback were shown to have potential to both enhance students computational and professional skills. These two objectives can be related to Shulman's distinction between pedagogical content knowledge (PCK) and subject matter knowledge (SMK), see e.g., Berry et al. (2016). We focus on assessment in mathematical tasks which address SMK as in mathematical competency but not PCK. In the literature, the effect of peer evaluation on PCK for pre-service mathematics teachers is discussed in Ayalon & Wilkie (2021). In contrast, there is little research on the influence of peer assessment on SMK in mathematics teacher education.

The purpose of the present paper and the associated study is to investigate the effects of peer assessment on proof understanding and construction in an undergraduate geometry course. As Lin et al. (2012) stress, teachers professional learning of proofs and teaching proofs depends on their knowledge, practice, and beliefs about proofs. Our main goal is to investigate how peer assessment among PSTs can be used to develop those aspects.

So far in this article, the term “proof competency” was used as an umbrella term which has not yet been defined; we shall remedy it now. The teaching of mathematical arguments and proofs in higher education and in teacher education has been an active research subject for quite some time. With a view towards teacher education, there are three different aspects to be considered: knowledge of proof, practice of proof and beliefs about proof. These are interdependent and need to be addressed simultaneously to improve proof competency (Lin et al., 2012). For our study, the construct of proof competency consists of four related aspects (see Selden and Selden, 2015): proof comprehension, proof construction, proof validation and proof evaluation.

Proof comprehension is the ability to read and understand written proofs. The “big difference” (Selden and Selden 2015, p. 4) between proof comprehension and proof validation is that in a proof comprehension situation, it can be assumed that the presented proof is correct, while this is not the case in proof validation situations. The distinction is of particular importance for us, as PSTs train for situations in which they are asked to validate and evaluate arguments and proofs. Proof evaluation describes the assignment of a value judgement to a proof (attempt). For professional mathematicians this often means judging a proof on its merits of conveying ideas and concepts. We view it as equally important for PSTs to be able to assess the presentation and clarity of a proof.

The three aspects of proof competency described in the last paragraph have in common that they apply to proofs presented to the PSTs. In contrast, proof construction asks for the creation of new proofs, usually to a statement provided to the learners. This activity in general is inherently more challenging than the other aspects and often requires substantial SMK. However, in a sense also the PCK is called upon in the construction and presentation of arguments to convince the reader of the validity of a claim. This social dimension can be viewed through the lens of communities of practice
(here the PSTs in the course), see e.g. Selden (2012, section 3.1.2). Proofs need to provide an acceptable level of conviction that the mathematical statement is true. Following Mason et al. (1982) these levels are (in ascending order of sophistication): 1. Convince oneself, 2. Convince a friend, 3. Convince a sceptic. What is viewed as sufficient to qualify for the different stages in the model will depend on the social norms and practices within the community for which the proof is constructed.

**Methodology and setting**

The setting in which we will carry out the peer assessment is a geometry course in a large Norwegian university, which has a particular focus on axiomatic constructions. One of the main learning goals of this course is to revisit classic geometric results from a higher standpoint. This includes explicit proofs in an axiomatic setting. Thus, the course aims at developing students’ geometric understanding as well as their understanding of and ability to produce proofs. The main stated public for this course is pre-service high-school teachers in their second or third study year.

In one of the assignments which the students need to hand in, they will assess an educator-made proof (based on student deliveries from a previous iteration of the course). This preparation task was selected to display subtle aspects of proving, which students usually find difficult, such as the need to prove that a condition is both necessary and sufficient. Two actual “peer assessment events” will then be carried out during the semester. Such an event consists of the following: students will solve a task knowing that it will be assessed by another student, and hand it in. In the next homework assignment, they will receive one of their peers’ solutions from the previous assignment and assess it. For this, the PSTs will be given a grading guide. Our design aim for the guide is to strike a balance between general and specific instructions: the guide needs to be specific to provide scaffolding for the assessment; nevertheless, it should also not be a step-by-step solution as we want students to exercise their own judgment and autonomy in assessing statements and justifications.

To answer our research questions, we will evaluate both the quality of the students’ proofs and the product of the peer assessment of these proofs. For the second research question, we will focus on clarity and logical soundness of arguments, as reflected in the proof and in the peer assessment of the delivery. We will then compare our own assessments with those obtained by the students in the peer assessment process. In addition, we will conduct interviews with participants. This qualitative data together with the results of a formal written assessment of the proof understanding and construction (conducted three times during the semester) will allow us to create a holistic description of the PSTs abilities. The assessment of proof understanding and construction is also part of a second research project focusing on the influence of learning videos on proving skills. The results from the assessments and additional interviews will shed light on the first research question.

**Discussion**

In designing our peer assessment experiment and data collection, we needed to clarify our goals: a large aspect of learning proofs is a question of students’ autonomy. As Robert and Schwarzenberger point out “tertiary students need to learn to distinguish between mathematical knowledge and meta-mathematical knowledge of the correctness, relevance and elegance of proof and take responsibility for their own mathematical learning.” (cited from Guzman et al., 1998, p.755). With that in mind, we believe that students having the responsibility of establishing the correctness of a peer’s proof might
be of value. In addition, reading the grading guide will make explicit and visible to them the set of
demands that mathematicians make on what can be called a “proof”. This relates to the
“enculturation” aspect of learning proofs: evaluating other people’s reasoning is an authentic activity
both for mathematicians and mathematics teachers.

We are convinced that peer assessment activities can be developed to become a valuable tool for the
acquisition of proof competency and the professionalization of teacher students.

References

Ayalon, M., & Wilkie, K. J. (2021). Investigating peer-assessment strategies for mathematics pre-service teacher learning on formative assessment. *Journal of Mathematics Teacher Education, 24*(4), 399–426. https://doi.org/10.1007/s10857-020-09465-1

Berry, A., Depaepe, F., & van Driel, J. (2016). Pedagogical content knowledge in teacher education. In J. Loughran & M. L. Hamilton (Eds.), *International handbook of teacher education: Vol. 1* (pp.347–386). Springer. https://doi.org/10.1007/978-981-10-0366-0_9

De Guzmán, M., Hodgson, B. R., Robert, A., & Villani, V. (1998). Difficulties in the passage from secondary to tertiary education. In *Proceedings of the international Congress of Mathematicians* (Vol. 3, pp. 747–762). Documenta Mathematica. https://doi.org/10.4171/dms/1-3/72

Hattie, J. (2008). *Visible learning*. Routledge.

Julien, A., Romijn, E. & Schmeding, A. (2023). *Peer and self-evaluation as a tool in teacher education*. [Submitted for publication]. http://dx.doi.org/10.13140/RG.2.2.13083.64803

Lin, F.-L., Yang, K.-L., Lo J.-J., Tsamir, P., Tirosh, D. & Stylianides, G. (2012). Teachers’ professional learning of teaching proof and proving. In G. Hanna and M. de Villiers (Eds.), *Proof and Proving in Mathematics Education* (pp. 327–346). Springer. https://doi.org/10.1007/978-94-007-2129-6_14

Mason, J., Burton, L., & Stacey, K. (1982). Thinking mathematically. Addison-Wesley.

Mejia-Ramos, J. P., Fuller, E., Weber, K., Rhoads, K., & Samkoff, A. (2012). An assessment model for proof comprehension in undergraduate mathematics. *Educational Studies in Mathematics, 79*(1), 3–18. https://doi.org/10.1007/s10649-011-9349-7

Selden, A. (2012) Transitions and proof and proving at tertiary level. In G. Hanna and M. de Villiers (Eds.), *Proof and Proving in Mathematics Education* (pp. 391–420). Springer. https://doi.org/10.1007/978-94-007-2129-6_17

Selden, A. & Selden, J. (2015). A comparison of proof comprehension, proof construction, proof validation and proof evaluation, In R. Göller, R. Biehler, R. Hochmuth, and H.-G. Rück (Eds.). *Didactics of Mathematics in Higher Education as a Scientific Discipline – Conference Proceedings* (pp. 339–345). Universitätsbibliothek Kassel. http://www.urn.fi/urn:nbn:de:hebis:34-2016041950121

Topping, K. (1998). Peer assessment between students in colleges and universities. *Review of Educational Research, 68*(3), 249–276. DOI: 10.3102/00346543068003249
# Formative assessment: quick surprise quizzes online in class in mathematics higher education

Sandra Gaspar Martins

Instituto Superior de Engenharia de Lisboa, Instituto Politécnico de Lisboa, Portugal;
[sandra.gaspar.martins@isel.pt](mailto:sandra.gaspar.martins@isel.pt)

Centro de Investigação em Ciências Sociais, Universidade Nova de Lisboa, Portugal;
[sicgm@campus.fct.unl.pt](mailto:sicgm@campus.fct.unl.pt)

In literature, moments of Active Learning in which all students are working effectively are considered an added value. Even better if, at the same time, they provide a formative assessment opportunity in which students receive immediate feedback. Thus, in a Calculus curricular unit, 31 online surprise quizzes were administered in Moodle to 112 students, distributed across nearly all classes. Students can repeat the quizzes as many times as necessary until they obtain the correct answer, without grade deductions. In an anonymous online survey, almost every student indicated that the quizzes were useful. Although they were not mandatory, the participation rate was very high. Many mentioned they were more attentive due to the quizzes, finding them useful for feedback, for understanding the level they were reaching, and for learning new things. Teachers also confirm that it is a pedagogical strategy worth maintaining.

Keywords: Quizzes, Moodle, formative assessment, active learning, feedback.

## Introduction

Quizzes are part of several successful approaches with different kinds of students, both in top universities and in other higher education institutions. Examples of these include TEAL (Dori & Belcher, 2004) at the Massachusetts Institute of Technology; SCALE-UP (Beichner et al., 2007) at North Carolina State University; Peer Teaching (Lasry et al., 2008) at Harvard University; Online Learning Modules (Hill et al., 2015) at the University of Sydney.

Particularly, in the teaching of mathematics in Higher Education, there are many different strategies to apply quizzes: as either formative or summative assessment; online or in-class; mandatory or optional; weekly or with other periodicity; generate new instances for each student or not; give penalties for submitting the answer more than once or not; only multiple-choice questions or more sophisticated ones, etc. Researchers are still looking for the best combination. Some approaches can be found in Siew (2003), Varsavsky (2004), Blanco et al. (2009), Lim et al. (2012), etc.

The National Center for Public Policy and Higher Education in the U.S.A. (Twigg, 2005) refers to computer-based continuous assessment and feedback as a key strategy for quality improvement. Shorter and Young (2011) made a comparison of three assessment methods: (1) daily in-class quizzes, (2) online homework, and (3) project-based learning. They found ‘daily in-class quizzes’ to be the best predictors of students’ learning for 117 undergraduate Calculus students.

Making surprise quizzes carried out during classes makes quizzes an Active Learning activity, that is, a moment in which all students are actively working. This adds even more value since it provides
immediate feedback to students (Booth, 2012), forces students to be more attentive in classes and increases competitiveness (Nadeem & Al Falig, 2020).

Taking all this into account, the online surprise quizzes described below were applied in class.

## Context

This research took place in the school year of 2023/24 in the course of Mathematics Applied to Engineering (Calculus with applications) belonging to the first year, first semester of the graduation in Computer and Multimedia Engineering of the Polytechnical Institute of Lisbon, Portugal. The researcher has been responsible for this course for some years and this year surprise quizzes were introduced in class. Weekly quizzes have been held for several years, but outside of classes. These quizzes appear from time to time in classes, typically after finishing a subject. In general, there is one quiz per class, sometimes at the end of the class, sometimes in the middle and, sometimes at the beginning – referring to the material that was taught in the previous class. We have three classes per week, lasting 1h30 each and we made a total of 31 quizzes.

Some of these quizzes have different questions for each student (randomly generated using some variables), but most are the same for all students. Everyone can try to answer the quiz as many times as they want, the grade will not be discounted. It is natural for students to talk among themselves, compare the results, and end up all having the full score. This is not discouraged, on the contrary, mutual help is encouraged. These quizzes are not mandatory and count very little towards the final grade. They only count if the grade in the exams is higher than 9.0 and if they improve their grade, in which case, counts 5% of the final grade.

The quizzes were mostly used from the weekly quizzes that had already been created for the course, some were new. Both these quizzes were made available on Moodle -- the Learning Management System of the Institute. These quizzes were simple, we tried to choose or create simple questions that do not use much class time. The quizzes were mostly answered in class, but sometimes there was no time to finish and then the students finished at home. The quizzes were only available to students of the class that was taking that quiz, from the time it opened, until the end of that class (typically).

The course had 112 subscribed students, with 6 being considered ghost students since never answered any class quiz, weekly quiz, test, or exam. Classes are not mandatory. The students who went to any class were 103, distributed this way: the researcher was a teacher in two daytime classes, T11D with 37 students and T12D with 36 students. The other teacher on the course had a night class T11N with 10 students and a daytime class T13D with 20 students.

The aim of the quizzes was not to assess students, but to make them be more attentive in class, study more, not to postpone, not to study first the other subjects that naturally are more pleasant to them (since they belong to their study area), and to make students be aware of their level of understanding (often students only realize that they cannot solve the exercises when they get the first test -- in the middle of the semester). Students usually are optimistic about their capabilities (Wandel, 2015) and quizzes help them to be realistic. It was written in Moodle and teachers repeatedly reminded students that the aim of quizzes was to make students be more attentive, study more and be aware of their level of understanding; students may copy all quizzes, but probably will not get the requested values in 'regular' assessment and it will not be worthwhile.
One of the advantages of quizzes being online is that they are self-corrected, and we can present them to large classes with little effort to create them and no effort to correct them. Maintaining the advantage of scaffolding the questions as well as in paper and pencil.

There were quizzes on all subjects. Typically, the teacher explained a subject and at the end made a quiz about that subject. It's interesting that in some classes, even in the last few days, when the quizz opened, everyone was attentive and working.

## The quizzes

The quizzes were produced through the 'Moodle activity': 'test'. We created one quiz and then use the "duplicate" option to replicate it to all the four classes (Figure 1.a)). However, for the next year, we will create just one quiz and open and close the same test every time we are in a class to create just one quiz. Students don't know what the quiz will be about (it's hidden, only visible to the teacher). Quizzes allows the introduction of images, and mathematical symbols using LaTeX, see Figure 2. Some students answer it on the smartphone others on laptops. It perfectly fits a smartphone, as we can see also in Figure 1.b) and c).

<figure><img src="image_31.png" /><figcaption>Figure 1: a) The same quiz for the 4 classes, with the theme hidden for the students. b) and c) Quiz including a figure and mathematical symbols. Screenshot of a smartphone</figcaption></figure>

Each quiz had just one question. Sometimes the questions are multiple choice like in Figure 2b) and c); other times they are numerical response as in Figure 2, and others have both as in Figure 3. Sometimes the question is the same for all students; other times it is an instance of the question with some randomly generated values, see Figure 2.
<figure><img src="image_32.png" /><figcaption>Figure 2: Two instances, aleatorily generated, of the same question, using numerical questions</figcaption></figure>

By “submitting”, students receive feedback, knowing which answers are correct and which are wrong.
The next attempt already includes these answers; they just need to change them, see Figure 3.

<figure><img src="image_33.png" /><figcaption>Figure 3: A question with numerical and multiple choice questions. And the feedback.</figcaption></figure>

We were creative when making the questions allowing to evaluate all subjects, even those who seem
difficult to evaluate using only numerical and multiple choice questions. The grades are all saved in
Moodle, and it is very easy to export them.

**Methodology, quantitative and qualitative data, and discussion**

The research question is whether these quizzes are seen by students and teachers as an added value.
To this end, a quantitative and qualitative study was carried out. The two research methods used were
Survey and Interviews (Cohen, 2007). As tools, an anonymous survey, online in Moodle, was made
available to all students on the course. And an Informal Conversational Interview of teachers
regarding this new pedagogical practice was also collected.

There was a total of 31 surprise quizzes in 39 classes of 1h30 each class. The course had 112
subscribed students, with 6 being considered ghost students since they never answered any class quiz,
weekly quiz, test, or exam. Classes are not mandatory. The students who went to any class were 103.
All 106 students (non-ghost) answered at least one quiz; of those, 96 students (91%) obtained a quiz
average above 5 out of 20; 77 students (73%) got a quiz average above 15 and 43 students (41%)
obtained a quiz average above 19 out of 20 having answered almost all the quizzes with almost everything correct – as it is expected, since they may go to all classes/quizzes and when they have doubts solving, they can ask for help from the professor and from the classmates. This shows that the student participation rate was very high (see Graphic 1).

<figure><img src="image_34.png" /><figcaption>Graphic 1: Quizzes average grades histogram</figcaption></figure>

The respondents to the survey were 47, representing 46% of the students who went to at least one class, with a representative distribution of student grades, including approved and failed students. About the question "Do you find the quizzes:" (Graphic 2), 43 find it useful, 3 indifferent and 1 didn't answer. So, 92% find it useful.

<figure><img src="image_35.png" /><figcaption>Graphic 2: Answers to the question "Do you find the quizzes:"</figcaption></figure>

About the question "If there were no quizzes, do you think your grade would be different?" 29 think that it would be worse, 16 think that it would be the same and 2 didn't answer. Then, 62% believe that it makes them have a better grade (see Graphic 3).

<figure><img src="image_36.png" /><figcaption>Graphic 3: Answers to the question "If there were no quizzes, do you think your grade would be different?"</figcaption></figure>

About " If the quizzes didn't exist it would have been: more or less attentive in class” (Graphic 4), 18 think that without quizzes they would be less attentive in class, 28 think that would have the same attention and 1 didn't answer. Thus 38% believe that it makes them be more attentive (there are those who would already be attentive anyway, but, in principle, these are not counted).
<figure><img src="image_37.png" /><figcaption>Graphic 4: Answers to the question " If the quizzes didn't exist it would have been:"</figcaption></figure>

Abou the next questions in Graphic 5, the number of respondents were (from top to bottom) 6, 1, 0, 0, 1, 30, 38, 20, 36, 30. In short, students believe that quizzes make them learn new things, study more, keep up with their studies, pay more attention, and be aware of the level they are reaching. Six students say that it gives them too much stress.

<figure><img src="image_38.png" /><figcaption>Graphic 5: Answers to the question " Please indicate ALL statements with which you agree:"</figcaption></figure>

The open question of the survey was: "What do you like/dislike about quizzes?" we obtained 11 responses, all positive, reinforcing that it helps them to test whether they understand the material with basic exercises, also indicating that it forces them to pay attention in class and helps them learn more.

In an Informal Conversational Interview (Cohen, 2007) qualitative data was collected: the feedback of the two teachers of the course about this pedagogical practice. The teacher (who is the researcher) considers that the quizzes were a positive strategy. They made the students pay more attention in class, whenever she said "next comes a quiz", the students studied more quietly and more attentively. It was interesting, that in one of the classes, when the quiz was launched, the students were all working hard, whether at the beginning of the year or until the end. In the other class, there were some who were disinterested. She believes that some students felt some stress, via that some were very concentrated and asked straight away if they didn't know how to solve a part and it was clear that they were anxious. Although there was no reason for that, because if the answer wasn't correct at the beginning, they were allowed to change it and there was never a lack of time: the teacher opened the test and only moved on to the next subject if almost everyone had already answered, and even let it
open it until a bit after the end of the class. The teacher felt that it is time consuming, and sometimes is difficult to leave time for students to answer it.

The other teacher stated that quizzes are very important because they encourage all students to work and not leave studying until the end. He felt that they were important, because in his classes all the students worked when a quiz was launched, except for two students, who had a lot of difficulties, who pretended [note the pressure that quizzes generate] that they were solving it, and then ended up not handing it in. The teacher indicates that the time consumed by quizzes can be a problem, suggests to make shorter quizzes and/or fewer quizzes and that some might be done at home. Due to lack of time, he tried leaving some quizzes to finish at home but there were less students responding.

As discussion, first be aware that the surveys were automatically anonymous, meaning there was no pressure for students to respond positively to the survey, despite the researcher being their teacher. Student's participations rate in quizzes was very high with 72% of students obtaining 15 or more average values. And 41% over 19, that is, with almost all quizzes answered with full marks.

In the responses to the survey, an important result was that nearly all students find quizzes useful, 62% consider that the quizzes helped them to get a better grade, and 38% refer that were more attentive in class due to the quizzes. It should be noted that 6 students consider that the quizzes cause them a lot of stress. Students' opinion is that quizzes made them to learn new things, help them to have a better understanding of the level that they are reaching, make them pay more attention, and remind them to catch up on the subject.

Teachers also consider quizzes to be an effective way to make students more attentive in concordance with Nadeem & Al Falig (2020), to have a moment where they all work hard, and where they receive immediate feedback, again in concordance with literature (Booth, 2012). In the future, given that quizzes take up a lot of class time, the intention is to move towards shorter questions or fewer quizzes.

## Conclusions and future work

In order to create moments of active learning that at the same time provided immediate feedback to students, 31 online surprise quizzes were administered in class, in Moodle, around one per class, in a Calculus course to 112 students. Student participation in these quizzes was very high, answering almost all quizzes, with almost everything correct. In an anonymous online questionnaire with a very significant number of responses, student feedback was that almost everyone considered the quizzes to be useful. That makes them work harder, learn more and be more attentive. And that allows them to receive feedback and become more aware of the level they are reaching. It also makes them to be always updated.

Teachers also consider that quizzes are useful, make students more attentive and that quizzes also allow students to receive more constructive feedback as they can repeat until the result is correct. Due to the lack of class time, in the future we will have fewer quizzes or quicker quizzes to answer. Our findings are in line with literature: quizzes make students more attentive (Nadeem & Al Falig, 2020), and give immediate feedback which is positive for students (Booth, 2012).

In short, given the high participation rate, the positive rating of the students and the encouraging reflection of the teachers, quizzes are undoubtedly a pedagogical strategy to maintain.
## References

Beichner, R. J., Saul, J. M., Abbott, D. S., Morse, J., Deardorff, D., Allain, R. J., ... & Risley, J. S. (2007). The student-centered activities for large enrollment undergraduate programs (SCALE-UP) project. *Research-based reform of university physics, 1*(1), 2-39.

Blanco, M., Estela, M. R., Ginovart, M., & Saà, J. (2009). Computer assisted assessment through moodle quizzes for calculus in an engineering undergraduate course. *Quaderni di Ricerca in Didattica, 19*(2), 78-83.

Booth, D. J. (2012). Managing Mathematics with CALMAT. *International Journal of Innovation in Science and Mathematics Education* (formerly CAL-laborate International), 2(1).

Cohen, L., Manion, L. & Morrison, K. (2007). *Research methods in education* (6th ed.). Taylor and Francis Group. http://repository.unmas.ac.id/medias/journal/EBK-00127.pdf

Dori, Y. J., & Belcher, J. (2004). Improving Students' Understanding of Electromagnetism through Visualizations-A Large Scale Study. Paper submitted to the *2004 NARST Annual Meeting*, Vancouver, Canada. Online at http://web.mit.edu/jbelcher/www/TEALref/dori.pdf

Hill, M., Sharma, M. D., & Johnston, H. (2015). How online learning modules can improve the representational fluency and conceptual understanding of university physics students. *European Journal of Physics*, 36(4), 045019.

Lasry, N., Mazur, E., & Watkins, J. (2008). Peer instruction: From Harvard to the two-year college. *American Journal of Physics*, 76(11), 1066-1069.

Lim, L. L., Thiel, D. V., & Searles, D. J. (2012). Fine tuning the teaching methods used for second year university mathematics. *International Journal of Mathematical Education in Science and Technology*, 43(1), 1-9.

Nadeem, N. H., & Al Falig, H. A. (2020). Kahoot! quizzes: A formative assessment tool to promote students' self-regulated learning skills. *Journal of Applied Linguistics and Language Research*, 7(4), 1-20.

Shorter, N. A., & Young, C. Y. (2011). Comparing assessment methods as predictors of student learning in an undergraduate mathematics course. *International Journal of Mathematical Education in Science and Technology*, 42(8), 1061-1067.

Siew, P. F. (2003). Flexible on-line assessment and feedback for teaching linear algebra. *International Journal of Mathematical Education in Science and Technology*, 34(1), 43-51.

Twigg, C. A. (2005). Course Redesign Improves Learning and Reduces Cost. Policy Alert. *National Center for Public Policy and Higher Education*.

Wandel, A. P., Robinson, C., Abdulla, S., Dalby, T., Frederiks, A., & Galligan, L. (2015). Students' mathematical preparation: Differences in staff and student perceptions. *International Journal of Innovation in Science and Mathematics Education (CAL-laborate International)*, 23(1), 82-93.
# Assessment of students' learning mathematics with technology using video-based activities in an online course

Eirini Geraniou<fnref n="1" /> and Cosette Crisan<fnref n="2" />

Using technology for mathematical learning, but also for assessing students' mathematical learning has proven to enhance, support and impact mathematics education in innovative, yet challenging ways. One of the goals of the online asynchronous master's module we designed is to prepare postgraduate students (who are either prospective or practicing mathematics teachers) for assessing school students' mathematical learning when using digital technologies. Fostering postgraduate students' noticing and interpreting skills when analysing and assessing mathematical learning while a digital tool is used, has become a key priority for the design of our module's activities. This paper presents: (a) our current research study for investigating how best to support postgraduate students develop skills for assessing mathematical learning when using digital technologies; and (b) an innovative video-based activity that addresses this developmental need.

Keywords: Digital technologies, assessment, mathematical learning, video-based activity, professional development.

## Introduction

Assessment plays a crucial role in the learning process, and in the digital era, we cannot underestimate the potential impact that digital technologies can have on mathematics education. Incorporating digital technologies into the assessment of mathematical learning opens up new possibilities, but also presents new challenges (e.g., Jankvist, et al., 2021; Drijvers & Sinclair, 2023). For instance, while automated assessments can offer immediate scoring, feedback, and adaptivity (e.g., Hoogland & Tout, 2018), there is a risk of overemphasising procedural fluency at the expense of capturing the depth and reasoning behind a student's response.

Over the past 15 years, there have been discussions about how ‘slow’ the transformation of assessment practices and policies in education with the support of digital technologies has been, despite the advancements in digital technologies (e.g., Timmis et al., 2016). For example, the rapid integration of Artificial Intelligence (AI) and tools like ChatGPT into educational assessments without proper research evidence and consideration of the implications is concerning. Therefore, it is important to gain an understanding based on rigorous research evidence, of how mathematical learning can be assessed when doing and learning mathematics with digital technologies. One of our goals as mathematics educators is to contribute to this research field whilst supporting the professional development of mathematics teachers in the digital era. In other words, we are carrying out a research study that investigates how best to support prospective and practicing mathematics teachers in assessing mathematical learning when a learner interacts with a digital tool. The context of our study is a ten-week online asynchronous master's module that introduces postgraduate students

<footnote n="1">¹IOE, UCL's Faculty of Education and Society, University College London, London, UK;
[e.geraniou@ucl.ac.uk](mailto:e.geraniou@ucl.ac.uk)</footnote>

<footnote n="2">²IOE, UCL's Faculty of Education and Society, University College London, London, UK;
[c.crisan@ucl.ac.uk](mailto:c.crisan@ucl.ac.uk)</footnote>
to several dynamic and interactive digital technologies for mathematical learning via numerous
innovative activities. One such activity is the use of video-based activities, which are short videos
that show school students working on mathematical tasks in GeoGebra and Desmos. Our module's
postgraduate students are asked to analyse the videos and assess the learning of mathematics.

In this paper, we present some details about our research study, before moving on to describe the
design and rationale of the innovative video-based activity. We conclude this short paper by
discussing future outcomes and contributions of our study.

**Research study and context**

We are interested in identifying the best pedagogic strategies for supporting our postgraduate
students, most of whom are prospective or practicing mathematics teachers, in developing skills for
assessing school students' learning of mathematics while they interact with digital technologies.
Before we go into more detail about the research work, we need to give a brief presentation of our
master's module, which we refer to as the 'Digi' module.

The Digi module is taught online, with participants being given a series of tasks over a ten-week
period. The weekly tasks are signposted on a virtual learning environment (Moodle) at the beginning
of each week and include offline tasks such as: familiarisation with a piece of software and example
problems using specific software, designing a maths activity using the specific digital environment,
and trialling out the activity with learners. In our context, learners could be either school students
from the schools that some of our postgraduate students (practising teachers) work at or school
students from our postgraduate students' own personal networks. Online tasks include engaging with
the ideas in the key readings of this module, reading one of the essential reading articles and writing
a response about the points agreed or disagreed with from the article, and also contributing to online
discussion forums with written observations on views and perspectives of their module peers. For
example, in the third, sixth and ninth weeks, our postgraduate students are required to choose a
software tool introduced in the prior two weeks, design a learning activity using features of good
practice identified from the literature, use the activity they designed with students and analyse its
implementation through engagement with research and the ideas assimilated from the literature
reviewed to evaluate and justify the implications of using digital technology for students' learning.
Being an asynchronous online course, our postgraduate students' contributions are solely in written
format and consist of their weekly written task submissions, forum contributions such as written
comments to peers' tasks, reflections on their own learning and peer assessed work, peer reviews and
peer assessment.

The research study involves three student cohorts enrolled on our Digi module in 2024-2026 and it
focuses on how students develop critically reflective and interpretative skills for assessing
mathematical learning that takes place when a learner interacts with a digital tool. We will focus on
different activities from the Digi module to answer the following main research question: *How are
postgraduate students' skills for assessing mathematical learning that takes place when a learner
interacts with a digital tool developed and supported by different activities?*, and our secondary
research question: *In what ways do video-based activities develop postgraduate students' skills for
assessing mathematical learning during interactions with a digital tool?*
## Online video-based activity

To assist our postgraduate students in critically engaging with research and applying it to reflect on classroom practices involving digital technology, we are experimenting with the use of online video-based activities in one of the weekly activities. We have created videos that showcase pairs of school students actively engaging in mathematical tasks within a digital setting. We wanted to provide our postgraduate students with a simulation of a classroom-based scenario where two pupils worked together on a maths task involving digital technology. For this reason, the videos were not edited, and our students were invited to select their own segments of the recordings to analyse. Inspired by Van Es and Sherin's (2002) research, we explored the utilisation of video-based activities to offer our postgraduate students a shared learning episode for analysis. Video cases have been employed by numerous mathematics educators and researchers to guide teachers in focusing on students' learning and the decisions made by teachers during lessons. Van Es and Sherin (2002) suggested that videos could be effective tools in enhancing teachers' ability to observe, notice and interpret classroom interactions.

Among the various features of videos extensively documented in literature (Calandra et al., 2009; Van Es & Sherin, 2002), we highlight the capability of a video to be paused, rewound, and replayed multiple times, allowing viewers to focus on specific segments strategically chosen for their relevance to the viewers' goals, which in the case of the activity we present in this paper, is assessing school students' learning of mathematics during their interactions with Desmos and/or GeoGebra. The design of the video-based activity was guided by recommendations from researchers (Van Es & Sherin, 2002) emphasising that video clips could help shift attention away from teachers and classroom events, redirecting it toward students' work. In our research study, the videos produced are recordings of the collaborative efforts of a pair of school age students, narrowing the focus to the pedagogical activity of noticing significant episodes and analysing students' learning. The videos for this online module feature two Year 8 students, Tim and Tom (pseudonyms), both 12 years old, attending different secondary schools in a large city in the UK. Given the importance of understanding how students interacted with the provided digital environment, we utilized screencast video-recording software to capture on-screen work and audio recordings of student-student interactions during the mathematics activity. The maths activities presented to Tim and Tom were related to plotting points in a graphical environment that satisfy the equations of given straight lines; finding the equations of straight lines graphs already plotted; investigating and proving properties of quadrilaterals constructed in specific ways. Our postgraduate students were encouraged to watch these short videos and analyse how Tim and Tom used the digital tools to investigate the mathematics task. They were then invited to submit a piece of writing (800 words) where they assess and justify Tim and Tom's mathematics learning in a digital environment as portrayed by the videos.

It is worth adding that the ethical aspect of producing and utilising the videos underwent careful consideration (Flewitt, 2005). To ensure ethical standards, explicit consent from participants and parents was obtained, wherein the researcher transparently communicated the intended use of the video material and its purposes.
## Concluding remarks

The first set of data collection from the first student cohort ends in March 2024 and we intend to present the findings of one of our enquiries: *In what ways do video-based activities develop postgraduate students' skills for assessing mathematical learning during interactions with a digital tool?* at the FAME1 conference. The data analysis will focus on postgraduate students' interpretations of the learners' interactions with GeoGebra and Desmos and assessment of potential learning outcomes, as observed in the video episodes, and reported in their written contributions. Evidence of development of critical reflection skills will also be sought in the postgraduate students' assignments where they are required to design, trial, evaluate and critically analyse a series of mathematical activities in their own practice, utilising the potential of digital technologies.

## Future steps

In collaboration with our university partnership schools, the module tutors intend to produce a series of videos, with real classroom settings, featuring school students engaging in mathematics activities within a digital learning environment. These video recordings will then be edited to highlight key segments, transforming them into video-based activities for professional development. We are hoping to be able to offer guidance for the creation of such innovative video-based professional development activities with a focus on promoting and supporting effective assessment strategies when using digital technologies in mathematics learning.

## References

- Drijvers, P., & Sinclair, N. (2023). The role of digital technologies in mathematics education: purposes and perspectives. ZDM - Mathematics Education. https://doi.org/10.1007/s11858-023-01535-x

- Flewitt, R. (2005). Conducting research with young children: some ethical considerations. Early Child Development and Care, 175(6), 553-565. https://doi.org/10.1080/03004430500131338

- Hoogland, K., & Tout, D. (2018). Computer-based assessment of mathematics into the twenty-first century: Pressures and tensions. ZDM - Mathematics Education, 50, 675-686. https://doi.org/10.1007/s11858-018-0944-2

- Jankvist, U. T., Dreyøe, J., Geraniou, E., Weigand, H.-G., & Misfeldt, M. (2021). CAS from an assessment point of view: Challenges and potentials. In A. Clark-Wilson, A. Donevska-Todorova, E. Faggiano, J. Trgalova, & H-G. Weigand (Eds.), Mathematics Education in the digital age: Learning, Practice and Theory, (pp.99-120). Routledge.

- Timmis, S., Broadfoot, P., Sutherland, R., & Oldfield, A. (2016). Rethinking assessment in a digital age: opportunities, challenges and risks. British Educational Research Journal, 42(3), 454-476. https://doi.org/10.1002/berj.3215

- Van Es, E., & Sherin, M. (2002). Learning to notice: Scaffolding new teachers' interpretations of classroom interactions. Journal of Technology and Teacher Education, 10(4), 571-596.
# Examining ChatGPT responses to TPCK assessment items

Peter Gonschwerowski<fnref n="1" />, Edith Lindenbauer<fnref n="2" /> and Benjamin Rott<fnref n="3" />

The development and, thus, the objective and valid assessment of the skill of selecting digital learning material (dLM) is essential for pre-service teachers. In this paper, we compare ChatGPT 3.5's responses with responses from pre-service teachers to items for assessing this skill to gain insights into ChatGPT's capabilities and the longevity of the items for its assessment. The results reveal that, for one, ChatGPT 3.5, so far, cannot analyze dynamic dLMs, and second, it does not make a decision on the use of the dLM but provides predominantly TPK but appropriate reasoning for using or not using it. ChatGPT's TPK responses are comparable to pre-service teachers' responses, and further studies are required to understand its impact thoroughly. Still, the presented results support the projected effects of ChatGPT on assessments in teacher education and the evaluation of the skill of selecting dLM.

Keywords: Artificial intelligence, performance-based assessment, teacher education, pre-service teachers, teacher evaluation.

## Introduction

Selecting digital learning materials (dLMs) is a crucial skill for educators due to the possibilities dLMs offer in teaching and the varying quality of the many freely available dLMs. Therefore, this skill needs to be developed in teacher training, and valid and objective assessment instruments are required to assess the success of such development processes (König et al., 2022; Redecker & Punie, 2017). In this context, the most frequently used instruments so far are self-assessment instruments, which are based on the TPACK framework (Mishra & Koehler, 2006), and only a few valid and objective TPACK assessment instruments have been published, one of them using open-text items developed by Gonscherowski et al. (in review). With the free public availability of text-based natural language processing and multimodal artificial intelligence (AI) models, examining such assessment instruments and respective items becomes increasingly important because we must understand how AI chatbots respond to such items and potentially undermine their assessment purpose. Therefore, this paper examines the ability of ChatGPT 3.5 to respond to items developed and validated in Gonscherowski et al. (in review) for assessing the (pre-service) teachers' skill of selecting dLM by having to reason for or against the use of a given dLM for specific learning content in the context of a specific learner age and potential special educational needs.

Existing multimodal AI systems process text and images and either generate new images, modify images, or categorize images (Livberber & Ayvaz, 2023). However, such systems do not currently interpret dynamic dLMs and their functionality or intended learning goal. We chose ChatGPT 3.5, although it is a text-based AI model and cannot directly analyze or interpret images because of its popularity and free use (Livberber & Ayvaz, 2023). We developed a textual description of a dLM and its functionality and provided it to ChatGPT to answer the following research questions.

<footnote n="1">¹University of Cologne, Mathematics & Natural Sciences, Germany; pgonsche@uni-koeln.de</footnote>

<footnote n="2">²University College of Education Upper Austria, Austria; edith.lindenbauer@ph-ooe.at</footnote>

<footnote n="3">³University of Cologne, Mathematics & Natural Sciences, Germany; brott@uni-koeln.de</footnote>
RQ1: How does ChatGPT reason for or against using a dLM for a specific group of learners, characterized by their age and special learning needs, when provided with a text-based description of said dLM?

RQ2: How does ChatGPT's reasoning compare to that of pre-service teachers who evaluated the same dLM?

By answering these research questions, we want to understand the longevity of the assessment items and gain insights on how to refine them in the future so that ChatGPT or similar tools cannot undermine an assessment of pre-service teachers using the items.

## Theoretical framework: TPACK

To assess the skill of selecting dLM, we rely on the TPACK framework by Mishra & Koehler (2006). The framework is frequently used to describe and assess the digital competence of pre-/in-service teachers (Gonscherowski & Rott, 2023). It describes the content knowledge (CK), pedagogical knowledge (PK), and technological knowledge (TK) that teachers require to integrate technology into teaching successfully. In the framework, pedagogical content knowledge (PCK), technological content knowledge (TCK), technological pedagogical (TPK), and technological pedagogical content knowledge (TPCK) detail the knowledge required because of the interplay of CK, PK, and TK. Mishra and Kohler (2006) define TK, TCK, TPK, and TPCK as follows:

TK is knowledge about standard technologies, such as books, chalk and blackboard, and more advanced technologies, such as the Internet and digital video. (ibid., pp. 1027–1028)

TCK is knowledge about the manner in which technology and content are reciprocally related. Although technology constrains the kinds of representations possible, newer technologies often afford newer and more varied representations and greater flexibility in navigating across these representations. (ibid., p. 1028)

TPK is knowledge of the existence, components, and capabilities of various technologies as they are used in teaching and learning settings, and conversely, knowing how teaching might change as the result of using particular technologies. (ibid., p. 1028)

TPCK [...] is the basis of good teaching with technology and requires an understanding of the representation of concepts using technologies; pedagogical techniques that use technologies in constructive ways to teach content. (ibid., pp. 1028-1029)

We refer to Mishra and Kohler (2006, pp. 1026-1027) for the definitions of CK, PK, and PCK. The descriptions of TCK, TPK, and TPCK entail reasons for using dLM in a teaching setting and can also be used to categorize reasons for or against using a dLM in a teaching situation.

## Method

We apply a qualitative case study method to answer the research questions. To do so, we compare ChatGPT 3.5's responses to items one to four (see Table 4) developed by Gonscherowski et al. (in review), with the responses by pre-service teachers when assessing the skill of selecting dLM using a specific dLM (see Figure 1). First, we outline the coding and scoring of the items, and then we compare ChatGPT and pre-service teachers' results.