## Assessing the skill of selecting dLM

In Gonscherowski et al. (in review), four items were developed and validated to assess the skill of selecting dLMs (see Table 4). These items aim to evaluate pre-service teachers’ understanding of the learning content that a particular dLM is intended to deliver (one open-ended item), the learner age group (grades 1-13, in two-year increments), and the special learner needs with which the dLM would be used (two closed items). A fourth open text item inquires about the reasons for or against using the dLM.

To evaluate the items’ ability to assess the skill of selecting dLM reliably, validly, and objectively, they were integrated into an online test using a specific dLM (see Figure 1). The online test was distributed among mathematics pre-service teachers, and participation was voluntary.

Maxi and the tree

<figure><img src="image_39.png" /></figure>

Maxi and the tree

<figure><img src="image_40.png" /><figcaption>Figure 1: Starting and end point of the dLM</figcaption></figure>

There were 379 participants each from one university in Germany (n = 314) and one in Austria (n = 65). The German pre-service teachers were distributed in programs on primary education (n = 110), special education (n = 173), and lower secondary education (n = 31); the Austrian pre-service teachers were enrolled in a combined program for lower and upper secondary levels. Furthermore, the participants covered all relevant semesters of the mathematics education program: first year (n = 57), second year (n = 149), third year (n = 93), and year seven or higher (n = 80).

The responses to the items were coded following qualitative content analysis and summarized in scores: items one to three were analyzed with scores from zero to three, and item four on a scale from zero to four. Table 1 shows example responses for items one to three for a generic and detailed description of the learning content appropriate for the learner age group and the special learner needs; the former scored two, and the latter scored three. Inadequate descriptions of the learning content and generic or detailed descriptions of the learning content but that were inappropriate for the selected learner age group or the selected special needs were scored zero and one, respectively.
<table><caption>Table 1: Example responses by pre-service teachers (items one to three)</caption><thead><tr><td>Definition of the score (items 1 to 3)</td><td>Example response</td></tr></thead><tbody><tr><td>A generic description of the learning content appropriate for learner age/special educational needs</td><td>"Introduction to the definition circle' Learner age: 3-4 and hearing and communication learning needs" (WS2022/2023_ANJA07B_Pre)</td></tr><tr><td>A detailed description of the learning content and appropriate for learner age/special educational needs</td><td>"To derive the definition of a circle line. All points on the circumcircle have equal distance from the center.' Learner age: 5-6 no special educational needs" (WS2022/2023_EDJO06B_Pre)</td></tr></tbody></table>

The responses to item four, “Justify why or why not you would use the digital material,” were coded and scored using the codebook developed based on the TPACK framework (Mishra & Koehler, 2006) and other validated and generally accepted studies. The reasons for or against the use of dLM were categorized as TCK, TPK, and TPCK-based arguments. Coding of the TCK reasoning entails arguments such as different or dynamic representations of the learning content and reducing or increasing extraneous cognitive load because of the dLM. The coding of the TPK reasoning encompasses arguments such as self-directed learning, differentiation, learner motivation, exploring, teacher efficiency, and learner distraction.

<table><caption>Table 2: Single TCK and TPK example reasoning by pre-service teachers (item four)</caption><thead><tr><td>Example response 1</td><td>Example response 2</td></tr></thead><tbody><tr><td>TCK: "My students (with special needs) would be overwhelmed with the dLM..." (WS2022/2023_MAHA10L_Pre)</td><td>TPK: "Because by moving the crosses around, the children can discover for themselves what properties all the points of a circle have." (WS2022/2023_DOMA15S_Pre)</td></tr></tbody></table>

The example responses shown in Table 2 were scored as two, containing a single TCK or TPK reason for or against using the dLM. Responses with either two TCK or TPK reasons were scored three, and responses with both TCK and TPK, thus TPCK reasoning, were scored four, see Table 3.

<table><caption>Table 3: TCK and TPK, thus TPCK example reasoning by pre-service teachers (item four)</caption><thead><tr><td>Example response 1</td><td>Example response 2</td></tr></thead><tbody><tr><td>"...provides a different way of practicing and illustrating the properties of a circle. However, it is not necessarily suitable for all learners with special needs. The tree and the learning atmosphere outside the classroom (not in the usual environment) can lead to too much distraction." (WS2022/2023_CHGU24M_Pre)</td><td>"...Changing environments (digital/non-digital) with students and exploring other learning environments encourages interest. Experiencing on their own how these mathematical relationships are connected makes understanding easier." (WS2022/2023_SAHE17B_Pre)</td></tr></tbody></table>
Example one in Table 3 entails “...a different way of practicing and illustrating...”, a TCK argument, and “...lead to too much distraction...”, a TPK argument, thus constituting TPCK reasoning. No reasoning was scored zero, and generic arguments were scored one.

The coding and the scoring of the suitability of the dLM for a particular learning age, learning needs, and learning content were derived from the local curriculum (items one to three). The arguments for or against the dLM (item four) were coded based on the codebook as outlined. For the skill of selecting dLM, the scores were combined on a scale from zero to seven.

## The development process of a textual description of the dLM for ChatGPT

The dLM presented in Figure 1 encompasses the mathematical topic of a circle. Learners should discover the concept of the circle based on its defining property: a figure consisting of all the points in the plane that are a specific distance (radius) from a certain point (the center). Since ChatGPT applies a language-processing AI model and only accepts text input, the three authors crafted a textual description of the dLM using collaborative editing and multiple review cycles. In the processes of crafting the description, the authors applied the following guiding rules: a) explicitly state that the description is one of dynamic learning material, b) describe the activity the learners need to perform, c) not use the term circle or properties of the circle, and d) use as much of the wording of the task description included in the dLM as possible. The process resulted in the following description of the dLM and its dynamic functionality, including an introduction. “The dynamic learning material we want to evaluate presents a task in a dynamic digital applet showing a tree, a child named Maxi, and further children represented by crosses. Maxi is at a pre-set distance from the tree. The learners should move the crosses representing the children so they all have the same distance from the tree as Maxi. Finally, learners can press a button named “solution,” and the solution to the task is revealed.” ChatGPT was provided with the description and the four assessment items.

In Table 4, abbreviated ChatGPT responses are contrasted with example responses of pre-service teachers. The full transcript of the ChatGPT session and the text description of the dLM were recorded (OpenAI, 12/25/2023). ChatGPT responses are coded and scored as outlined in the previous section (see Tables 1, 2, and 3).

## Results

ChatGPT’s answer to item one, the learning content the dLM is intended to provide, is incorrect as the dLM is not intended for spatial reasoning in geometry or possibly proportional reasoning (see Table 4). The correct learning content contains the characteristics of a circle as a shape consisting of all points of equal distance from a given point, the center. The assigned grade level by ChatGPT (early to middle stage of primary education) is also incorrect. Although the circle is covered in the middle stages of primary school, the concept of equal distance from the center to the circle’s circumference is only covered in lower secondary school, according to the curriculum in Germany and Austria. ChatGPT’s responses to item three, the special needs of learners, contain multiple impairments. Each impairment is described with a sentence, of which “fine motor skills,” “cognitive challenges,” and “attention challenges” are appropriate. However, ChatGPT suggested that the dLM could be used with auditory learners if the dLM would be auditive narrated, which was not indicated in the description of the dLM. Although correct, it is conjecture and inappropriate.
The scoring as outlined previously resulted in a score of zero pt. for items one to three. Within the text responses, ChatGPT references elements of the description of the dLM, e.g., “Maxi and other children represented by the crosses” or “solution button suggests [...] immediate feedback,” providing specificity and context to its responses, however not all are suitable in response to the items.

The justifications for using or not using the dLM, required in item four provided by ChatGPT, are appropriate but predominantly correspond to TPK reasoning, namely “engaging and interactive” (motivation), “immediate feedback” (self-directed learning), the potential for “inclusivity” or “sensory overload” (both differentiation), and “motor skill challenges” (inclusion). Other reasons provided are coded as overly general arguments (e.g., “overreliance on technology” and “lack of flexibility”) and incorrect argumentation, such as “concept reinforcement,” as the dLM is for the discovery of the learning content and not for its reinforcement. ChatGPT references elements of the text description of the dLM, for example, “arranging crosses around the tree...” and “...button providing immediate feedback...” to support its arguments.

<table><caption>Table 4: Responses to items assessing the skill “selecting dLM” by pre-service teachers and ChatGPT</caption><thead><tr><th>#</th><th>Items</th><th>Example responses of pre-service teachers</th><th>pt</th><th>ChatGPT 3.5 responses</th><th>pt</th></tr></thead><tbody><tr><td>1</td><td>Describe the learning content for which you think the digital material is intended.</td><td>“To derive the definition of a circle line. All points on the circumcircle have equal distance from the center.” (WS2022/2023_edJO06B_pre)</td><td>3</td><td>“...it seems that the dynamic learning material is designed for teaching and practicing concepts related to spatial reasoning, geometry, and possibly proportional reasoning in a mathematics education context....”</td><td>0</td></tr><tr><td>2</td><td>For which learner age do you think the digital material is suitable?</td><td>“5-6 grade” (WS2022/2023_edJO06B_pre)</td><td></td><td>“...the digital material seems to be designed for learners in the early to middle stages of primary education...”</td><td></td></tr><tr><td>3</td><td>In your opinion, is the digital material suitable for learners with special educational needs? If so, which ones?</td><td>“No special educational needs.” (WS2022/2023_edJO06B_pre)</td><td></td><td>“...the dynamic learning material seems to have characteristics that could make it inclusive and adaptable for a range of learners, including those with certain special educational needs: visual learners, auditory learners, fine motor skills, cognitive challenges, customization and adaptability, attention challenges...”</td><td></td></tr></tbody></table>
<table><tr><td>4</td><td>Justify why or why not you would use the digital material.</td><td>“...provides a different way of practicing and illustrating the properties of a circle. However, it is not necessarily suitable for all learners with special needs. The tree and the learning atmosphere outside the classroom (not in the usual environment) can lead to too much distraction.” (WS2022/2023_CHGU24M_Pre)</td><td>4</td><td>“Certainly, the decision to use the digital material depends on the specific educational context, learning objectives, and the needs of the learners. Here are justifications both for and against using the digital material: Reasons to use the dLM: Engaging and interactive, visual learning, immediate feedback, concept reinforcement, potential for inclusivity; Reasons not to use the dLM: motor skill challenges, overreliance on technology, sensory overload, lack of flexibility.”</td><td>3</td></tr></table>

Without making an actual decision on the use of the dLM, ChatGPT's reasoning for and against using the dLM corresponds to a score of three points, as previously outlined.

Regarding RQ1, “How does ChatGPT reason for or against using a dLM,” our analysis reveals that ChatGPT does not adequately capture the learning content and the learner age range of the learning content. This is not surprising, as the specifics of the curriculum for one differ by local and are not necessarily consistent. Advances in ChatGPT responses in this regard are expected, particularly with the ChatGPT 4.0 personalized memory feature, which allows localized information like the curriculum to be added to a ChatGPT user profile. Further, ChatGPT 4.0 does not require a textual description of the dLM. In addition, ChatGPT does provide generic and predominant correct TPK reasoning to justify using or not using the dLM.

Regarding RQ2, “How does the reasoning of ChatGPT compare to pre-service teachers who evaluated the same dLM?” Table 4 reveals that, for item four, ChatGPT's responses are comparable to those shown in Tables 2 and 3. However, ChatGPT does not decide and only provides reasons for and against using the dLM. The combined score for the entire sample of pre-service teachers (n = 379), as presented in Table 5, shows a total mean score of 2.18 (SD = 1.59), which underscores the good results of ChatGPT (score of 3) and echoes its reported capabilities and impact on assessments in higher (teacher) education (Livberber & Ayvaz, 2023).

<table><caption>Table 5: Scores of pre-service teachers and ChatGPT</caption><thead><tr><td>mean; SD; max score</td><td>pre-service teachers (n = 379)</td><td>ChatGPT</td></tr></thead><tbody><tr><td>Items one-three (learning content)</td><td>0.93; 1.06; 3.00</td><td>0.00</td></tr><tr><td>Item four (reasoning)</td><td>1.25; 1.03; 4.00</td><td>3.00</td></tr><tr><td>Σ representing the score for “selecting dLM”</td><td>2.18; 1.59; 6.00</td><td>3.00</td></tr></tbody></table>

Particular to note is ChatGPT's capability of providing subject-unspecific (TPK) reasoning and its lack of TCK reasoning, which needs to be leveraged when using the four items and a dLM for assessing the skill “selecting dLM.” One should use a dLM that is complex enough to provide various arguments, and we recommend dLMs for discovering (mathematical) concepts, as such a dLM
enables potentially more argumentation, specifically more TCK arguments, as, for example, a dLM for drill and practice.

## Outlook and Limitations

As ChatGPT 3.5 cannot examine dynamic dLM, an inherent limitation of the case study lies in the authors' description of the dLM. Higher-quality responses by ChatGPT could have potentially been achieved with a more elaborate description of the dLM and by further refinement of the items to cater to ChatGPT. However, we did not want to optimize the responses of ChatGPT, but rather the opposite, as one can conjecture if pre-service teachers can precisely describe the functionality of a dLM and inquire ChatGPT in a way to achieve high-quality responses, they may also possess the skill of selecting dLMs we want to assess with the items. The use of dLM for evaluating the skill of selecting dLM increases the technology requirements of the assessment. In environments with limited online access, the risk posed by using ChatGPT may also be lower. In the future, further testing with other and newer AI models (ChatGPT 5.0) and dLMs is required to understand the impact of assessing the skill with the developed items thoroughly. In addition, the responses of ChatGPT should be compared with responses by in-service teachers whose responses potentially exceed those of pre-service teachers, as hypothesized in Gonscherowski and Rott (2022).

## References

Gonscherowski, P., Lindenbauer, E., Kaspar, K., & Rott, B. (in review). Selecting digital learning material as an approach to assess pre-service teachers' digital competence.

Gonscherowski, P., & Rott, B. (2023). Selecting digital technology: A review of TPACK instruments. In M. Ayalon, B. Koichu, R. Leikin, L. Rubel, & M. Tabach (Eds.), *Proceedings of the 46th conference of the International Group for the Psychology of Mathematics Education* (Vol. 2, pp. 378–386). PME.

Gonscherowski, P., & Rott, B. (2022). How Do Pre-/In-Service Mathematics Teachers Reason for or against the Use of Digital Technology in Teaching? *Mathematics, 10*(13), 2345. https://doi.org/10.3390/math10132345

König, J., Heine, S., Jäger-Biela, D., & Rothland, M. (2022). ICT integration in teachers’ lesson plans: A scoping review of empirical studies. *European Journal of Teacher Education, 1–29*. https://doi.org/10.1080/02619768.2022.2138323

Livberber, T., & Ayvaz, S. (2023). The impact of Artificial Intelligence in academia. *Heliyon, 9*(9). https://doi.org/10.1016/j.heliyon.2023.e19688

Mishra, P. & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. *Teachers College Record, 108*(6), 1017–1054. https://doi.org/10.1111/j.1467-9620.2006.00684.x

OpenAI. (12/25/2023, 4:29:14 PM). ChatGPT [Large language model] https://chat.openai.com. https://chat.openai.com/share/fce6ba96-b89a-4802-9093-04fee07bee5b

Redecker, C., & Punie, Y. (2017). European framework for the digital competence of educators – DigCompEdu. *Publications Office of the European Union*. https://doi.org/10.2760/159770
# Open-ended items in digital formative assessments: Decision Trees as (AI-compatible) approach to reliably code students' understanding?

Corinna Hankeln

TU Dortmund University, Dortmund, Germany; [corinna.hankeln@math.tu-dortmund.de](mailto:corinna.hankeln@math.tu-dortmund.de)

Open-ended items, in which students draw images, explain meanings or argue, allow them to express their own mental representations of situations and make it possible to grasp even fragile concepts in nuances and details. However, those answers-types are rarely found in digital formative assessment, also because they are often difficult to evaluate. This paper reports on the integration of open-ended items into the digital formative assessments of the Mastering Math – Online-Check and exemplifies for an item on conceptual understanding of multiplication how current approaches of category-based scoring could be optimized by using decision trees to rate features of responses. In preparation for the integration of an automatic pre-coding by an artificial intelligence, an exploratory study is presented on the functioning of prompt-based classification of students’ answers by ChatGPT.

Keywords: Evaluation methods, Digital formative assessment, conceptual understanding, decision tree, AI-prompt.

## Open-ended items are “worth the effort” in digital formative assessments

Formative assessments have the potential to promote the implementation of conceptual learning in classrooms (Burkhardt & Schoenfeld, 2018). However, many digital formative assessment (DFA) platforms hold a dominant procedural focus (Hoogland & Tout, 2018), partly because procedural items are easier to (automatically) evaluate. In order to uncover shallow understanding and to assess deep conceptual understanding, items are needed where students translate a concept between different (e.g., verbal, graphical, symbolical, or contextual) representations, explain the meaning of particular concept elements and the connection between representations, or connect different concept elements in a wider network of elements (Hiebert & Carpenter, 1992). There are proofs of existence that students’ thinking can be assessed by well-designed multiple-choice formats (e.g., the SMART test, Stacey et al., 2018), but open-ended long-answer or complex graphical formats give students more opportunities to express their own mental representations of situations (without being influenced by distractors), allowing thus the demonstration even fragile concepts in details and nuances (Hankeln et al., submitted). Furthermore, students’ language production for describing mathematical structures or explaining meanings are relevant learning goals (Götze & Baiker, 2021; Prediger, 2022) that should not be excluded in assessments, also because those responses provide valuable resources for subsequent communication processes between students and teachers.

## Typical challenges in coding open-ended items

Well-designed open-ended items come with the price that the evaluation of those responses requires topic-specific epistemic background knowledge, taking into account the current position of students’ learning progression, knowledge about the relevant components of the assessed topic, like concept elements, representations, and language needed to explain them (Siemon, 2019) and typical misconceptions. In their meta-study of 14 DFA tools, Çekiç and Bakla (2021) state
"As for open-ended items, no fully reliable methods of grading have been created so far, but there have been significant developments in this area. Several tools have put an effort in developing systems to grade open-ended items. There have been four methods of grading: (1) autoscoring of short-response questions [...], (2) auto-grading based on the existence of a set of pre-determined keywords [...] (3) assigning numerical scores manually [...] and (4) the use of artificial intelligence for scoring open-ended items. Each of these methods is valuable in a time when we desperately need ways to deal with open-ended responses. [...] Obviously, the success of the keyword method or artificial intelligence is open to debate and should be tested empirically, yet they seem to be good starting points for further developments." (p.1477)

This paper presents a small-scale exploratory study to address this research gap, contrasting a combination of (2) and (3), namely manually classifying open-ended items based on different coding schemes with (4), the use of few-shots prompts to ChatGPT to code students' responses, all within the DFA Mastering Math – Online-Check.

## Open-ended items in the Mastering Math – Online-Check

<figure><img src="image_41.png" /><figcaption>Figure 1: Exemplary open-ended items in the Mastering Math Online-Check</figcaption></figure>

The Mastering Math – Online-Check (Hankeln et al., submitted) is currently developed as a DFA that is integrated in the 15-year-long Mastering Math project which aims at Grade 5–7 (10- to 13-year-old) students who struggle in mathematics and need a second learning opportunity for understanding basic arithmetic concepts such as the place value understanding or meanings of multiplication and division (Prediger et al., 2019). Each of the 45 Online-Checks is linked to teaching material, and the results of every Online-Check provides support for the prioritization of learning tasks and communicative prompts in remediation classes. The Online-Checks are administered in the newly created platform *alea.schule*. When teachers have chosen an Online-Check for their students in this platform, students can access the assessment via any browser on a tablet or computer. When students
have filled out an Online-Check, their answers get send to the teacher-platform alea.schule. All items in closed formats (multiple-choice or single-choice items, short answers, drag-and-drop answers, etc.) are automatically coded as correct or incorrect regarding typical misconceptions. Open answers need to be manually coded by teachers, supported by suggested item-specific categories entailing typical solutions and errors (Figure 2). So the items can be evaluated not just if they are right or wrong, but “wrong in a specific way” (Stacey et al., 2018, p. 246). The evaluation outcomes can be displayed in different evaluation dashboards with varying degrees of details and focus. The Online-Check thus aims at informing teachers to support their planning of subsequent lessons and does not provide any direct feedback to students.

<figure><img src="image_42.png" /><figcaption>Figure 2: Coding area in the platform alea.schule: Category-based coding to evaluate students' responses</figcaption></figure>

## Evaluation of category-based scoring and proposition of decision trees

A necessary (but insufficient) precondition for the validity of the conclusions drawn from the classification of students' responses is that the coding of open items is reliable (Çekiç & Bakla, 2021). However, a pilot study with 15 pre-service teachers (in their mathematics teacher master program) who coded 50 responses to the item “Bottles” in Figure 2 following the proposed categories revealed low interrater-reliabilities (3 rater per student response, κ = .21).

There can be various reasons for this observation: The correct choice of a category is highly dependent on the raters' pedagogical content knowledge (Prediger et al., 2023). Without an accurate understanding of the categories, raters cannot identify central indicators for these categories within student responses. Whereas research projects overcome this challenge by detailed rater preparation, teachers – in their daily use of the tool – need to be able to code different items without detailed instruction. That is why proposed buttons for selecting categories have to be labeled precisely, taking into account frequent misconceptions. While there are ideas how to improve the comprehensibility, for example by including category descriptions, another approach is to integrate a feature-based scheme to evaluate the answers in form of decision trees (Kingsford & Salzberg, 2008). Students' responses are thus seen as texts that have to be labelled, which makes the coding of students' answers to a form of text-classification problem (Gasparetto et al., 2022). There are various approaches to text-classification as it is widely used for example in spam-filters or website-classification, and one of them is decision trees. A decision tree is a sequence of questions about features associated with the items (Kingsford & Salzberg, 2008). The questions thereby form a hierarchy, encoded as a tree. There are statistical means to design such hierarchical trees, for example to ensure that the data is divided
into groups with similar variances by each questions (Kingsford & Salzberg, 2008). However, for evaluating students’ responses, the hierarchy is derived from the goal of the assessment and has to be grounded in topic-specific mathematics education backgrounds.

<figure><img src="image_43.png" /><figcaption>Figure 3: Decision tree for the Item “Bottles” (from Figure 2)</figcaption></figure>

Moons (2023) reports that some teachers consider grading schemes to be more efficient even when in fact it does not accelerate their grading process. A grading scheme is a set of statements from which teachers can select those that match the students’ response (also called check-box grading). As the traditional, holistic grading in this study already had a very high interrater-reliability (κ > 0.8), a significant improvement by introducing the check-box grading could only be observed for one item.

For the Item “Bottles” (Figure 2), we designed a decision tree in order to evaluate students’ conceptual understanding of multiplication (Figure 3). In this paper, we analyze those answers that justify their (correct) decision that the word problem posed by a fictitious student does not match the multiplication 2 x 5. The first step in the decision tree was to identify “nonsense” answers that cannot be used for an assessment. The second question aimed at identifying if answers indicate students’ understanding that the posed word problem uses counted units of different size which does not correspond to multiplication unit structures (first 2 bottles, then 5 bottles instead of two times the same amount of bottles). This is the essential aspect of the item to capture if a student uses expressions referring to the bundle sizes. If not, the next question checks if the answer argues that the word problem would fit an addition (and (implicitly) argues that it thus cannot be a multiplication). All while being correct as well, this answer reveals insight into students’ conceptual understanding of the contrast between addition and multiplication. Differentiating between these nuances allows to make detailed diagnoses of individual students to enchain specific follow-up questions only to certain students. If this third question is answered negatively, the response-text is checked if the student correctly rejected the proposed multiplication but based his decision on incorrect reasons. As the posed word problem contains three numbers, a typical surface strategy is to blindly take and combine them. So, the third question tries to identify those answers that draw upon this surface strategy by
saying for example “the second 2 is missing in the calculation, it has to be 2 x 2 x 5 = 20”. All remaining response-texts are assumed to be atypical mistakes that cannot unambiguously be related to theoretical misconceptions.

# Empirical Study: Human and AI-coding with a decision tree

## Methods

To investigate if the decision tree-based approach is a suitable way to optimise the coding of open-ended items, 124 children' responses to the Item “Bottles” from a pilot study of the Online-Check were coded with the decision tree. Firstly, two trained raters coded all responses independently, discussed differences and decided on a final coding (“expert-rating”) that is used as base-line to compare the quality of other ratings. Secondly, 15 pre-service teachers in their master studies coded a subset of 50 responses according to the decision tree. Every pre-service teacher received 10 responses to code in a rotated design. Those 10 responses were compiled to be a representative set of answers in order to avoid systematic misunderstandings of the questions biasing the ratings. The sample was drawn from a stochastics course for second year pre-service teachers at TU Dortmund University, which did not relate to the topic of the item. There were no additional information provided for the raters other than those in Figure 3. This coding resulted in a dataset (“teacher-rating”) with three ratings per students’ response. The different codings were compared (a) within the group of pre-service teachers in order to estimate their agreement (using Fleiss Kappa) and (b) between the expert-rating and the teacher-rating. Thirdly, the AI ChatGPT was asked with the help of few-shot prompts to classify the students’ responses analogically. This coding was also compared to the expert-rating and the teacher-rating based on the accuracy (proportion of predictions that are correct), the precision (proportion of positive predictions that are correct) and the sensitivity (recall) (proportion of positive answers that are correctly predicted).

## Findings

The expert-rating revealed that even though the hierarchical structure is only developed with respect to the assessed content, every group of responses is represented (Table 1) and only 38 responses (31 % of all responses) belonged to the “other error” category. 21 responses (55 % of this category) showed atypical mistakes like misunderstanding the situation (“he gets two and five bottles and he does that two times, so it has to be two times seven”), the others gave incomplete justifications (“he goes two times in the basement”) or gave no reasons at all (“it does not fit”).

<table><caption>Table 1: Distribution of responses categories</caption><thead><tr><th></th><th>End of decision-tree branch</th><th>Continuation of decision-tree</th></tr></thead><tbody><tr><td>A</td><td>No: 5 responses (5 %)</td><td>Yes: 119 responses (96 %)</td></tr><tr><td>B</td><td>Yes: 33 responses (27 % of remaining responses)</td><td>No: 86 responses (73 % of remaining responses)</td></tr><tr><td>C</td><td>Yes: 39 responses (45 % of remaining responses)</td><td>No: 47 responses (55 % of remaining responses)</td></tr><tr><td>D</td><td>Yes: 9 responses (19 % of remaining responses)</td><td>No: 38 responses (81 % of remaining responses)</td></tr></tbody></table>

The “teacher-rating” conducted by master students showed an improvement in the interrater-reliability for the coding based on a decision tree compared to the classical category-based approach (see above), ranging between κ = .41 (B), κ = .63 (C) and κ = .58 (D). It is interesting to see that the
question that requires the most pedagogical content knowledge about unitizing or multiplication as
counting in groups is the category with the lowest agreement. This question had an insufficient
agreement between expert-rating and teacher-rating. The pre-service teachers only coded 59 % of the
responses like the expert-rating for question B, while 73 % of agreement was reached for question C.
Question D showed the lowest agreement with 39 %. It has to be kept in mind that the pre-service
teachers did not receive any examples or explanations for the different questions.

<figure><img src="image_44.png" /><figcaption>Figure 4: Prompt to Chat-GPT to answer question C in the decision-tree (Figure 3)</figcaption></figure>

In order to explore how a Large-Language-Model such as ChatGPT can evaluate the responses
without being a priori trained with labelled data, we formulated few-shot prompts, where we
described the item (classifying students' responses), the origin of the data (students' responses to the
item "Bottles") and explained the questions that ChatGPT had to answer for every response (Figure
4). To improve the quality of the coding, we also included a few examples for the decisions yes or no
respectively. Those examples were given both as general description and with a precise example. We
iterated the prompt design and revised the prompts when we could identify systematic
misunderstandings. We report here the statistics of the best fitting prompts. The identification of non-
rateable responses worked very well with an accuracy of 96 %. In two cases, ChatGPT found a
rateable response to be non-rateable, so the precision was a 100 % but the recall (sensitivity, how
well a yes-answer can be detected) was 98 %. This error would thus lead to the abort of the coding
process and the loss of diagnostic information. The identification of the structural element of the
multiplication was identified in 32 cases in the expert-rating. All of those cases have also been
identified by the AI, the recall was thus 100 %. However, 21 cases have been falsely diagnosed to
make reference to the structural element of multiplication (precision: 60 %). In total the accuracy was
82 %. The expert-rating revealed 56 cases where no reference to the structural element was made but
a reference to the addition or the result of the addition. 50 of these cases have also been detected by
ChatGPT (recall 89 %), 11 cases were falsely marked (precision 82 %). The accuracy was 80 %. For
Question D, that asks if students decided correctly but due to an incorrect surface-strategy, the expert-
rating identified nine cases of the remaining 31 responses. Six of them were found by the AI (recall
67 %), five answers were falsely accused of a surface strategy (precision 55 %) and the accuracy was
at 65 %. All questions showed that with the few-shots prompt, the recall (sensitivity) was higher than
the precision, meaning that the identification of true yes-answers works well with the price that there
are several false positive classifications. For Questions B and C, this would imply that the problem is
falsely classified as correct and possible problems are not detected. For Question D this implies that
the surface strategy is more often suspected than true. The balance of both error types is of course a challenge, but for the specific use of the formative assessment, we would prefer, especially for Question B, to have a higher precision.

<table><caption>Table 2: Accuracy, recall and precision of ChatGPT's classification of decision tree questions (Figure 3)</caption><thead><tr><th>Decision tree question</th><th>n</th><th>accuracy<br>(<sup>correct predictions</sup>/<sub>all predictions</sub>)</th><th>recall<br>(<sup>true positive predictions</sup>/<sub>true positives + false negatives</sub>)</th><th>precision<br>(<sup>true positive predictions</sup>/<sub>positive predictions</sub>)</th></tr></thead><tbody><tr><td>A rateable answer?</td><td>124</td><td>95.7%</td><td>98.3%</td><td>100%</td></tr><tr><td>B structural element of<br>multiplication?</td><td>118</td><td>82.2%</td><td>100%</td><td>60.4%</td></tr><tr><td>C addition?</td><td>87</td><td>80.5%</td><td>89.3%</td><td>82.0%</td></tr><tr><td>D surface strategy?</td><td>31</td><td>64.5%</td><td>66.7%</td><td>54.6%</td></tr></tbody></table>

## Discussion and conclusion

Open-ended items are challenging to use in any assessment, but they bring enormous advantages especially for formative assessments aiming at capturing students' conceptual understanding in details and nuances (Hankeln et al., submitted). This small, exploratory study gave insight into the challenges of the evaluation of open-ended items and proposed the use of decision trees to get a precise impression of the features of a response while on the same time improving the reliability of a scoring. The empirical findings show that interrater-reliability of pre-service teachers can indeed be improved by a question-based decision tree. However, in this non-representative sample, it did not reach a satisfactory level. This can of course be due to insufficient topic-specific epistemic background knowledge of the pre-service teachers who have not yet finished their studies, but this could also indicate the need for additional information on the expected coding, also when a decision tree is used. Such additional information could either be general descriptions that can be accessed on demand, or exemplary codings, like they were included in the few-shot prompt that was given to Chat-GPT. Our first results seem to confirm Çekiç and Bakla (2021), that AI-based coding is a promising approach for future development. In our case, however, we saw a tendency of ChatGPT to have a better recall than precision. This has to be investigated further, especially in contrast to other AI-based classifier like for example BERT.

## Acknowledgment

**Funding.** The Mastering Math Online-Check is developed and investigated within the projects SchuMaS (funded by the National Ministry of Education and Research, grant No. SchumaS-SMS2101L-01PR2101C to S. Prediger) and Mastering Math (funded by the Departments of Education, Hamburg and Hessia to S. Prediger, C. Hankeln, and U.Kroehne).

## References

* Çekiç, A., & Bakla, A. (2021). Review of digital formative assessment tools: Features and future directions. *International Online Journal of Education and Teaching, 8*(3), 1459–1485.

* Gasparetto A, Marcuzzo M, Zangari A, Albarelli A. (2022). A survey on text classification algorithms: from text to predictions. *Information. 13*(2), 83. https://doi.org/10.3390/info13020083
Götze, D. & Baiker, A. (2021). Language-responsive support for multiplicative thinking as unitizing: Results of an intervention study in the second grade. *ZDM – Mathematics Education, 53*(2), 263–275. https://doi.org/10.1007/s11858-020-01206-1

Hankeln, C., Kroehne, U., Voss, L., Gross, S. & Prediger, S. (submitted). Developing digital formative assessment for deep conceptual learning goals: Which topic-specific research gaps need to be closed? Submitted manuscript.

Hiebert, J., & Carpenter, T. P. (1992). Learning and teaching with understanding. In D. A. Grouws (Eds.), *Handbook of research on mathematics teaching and learning* (pp. 65–97). Macmillan.

Hoogland, K. & Tout, D. (2018). Computer-based assessment of mathematics into the twenty-first century: Pressures and tensions. *ZDM – Mathematics Education, 50*(4), 675–686. https://doi.org/10.1007/s11858-018-0944-2

Burkhardt, H. & Schoenfeld, A. (2018). Assessment in the service of learning: Challenges and opportunities or Plus ça Change, Plus c'est la même Chose. *ZDM – Mathematics Education, 50*(4), 571–585. https://doi.org/10.1007/s11858-018-0937-1

Kingsford, C., & Salzberg, S. L. (2008). What are decision trees? *Nature Biotechnology, 26*(9), 1011–1013. https://doi.org/10.1038/nbt0908-1011

Moons, F. (2023). Semi-automated assessment of handwritten mathematics tasks: Atomic, reusable feedback for assessing student tests by teachers and exams by a group of assessors. [Doctoral thesis 3 University of Antwerp]. https://hdl.handle.net/10067/1980770151162165141

Prediger, S. (2022). Enhancing language for developing conceptual understanding. In J. Hodgen, E. Geraniou, G. Bolondi, & F. Ferretti (Eds.), *Proceedings of 12th Twelfth Congress of the European Society for Research in Mathematics Education* (pp. 8–33). University of Bolzano / ERME.

Prediger, S., Dröse, J., Stahnke, R., & Ademmer, C. (2023). Teacher expertise for fostering at-risk students’ understanding of basic concepts: Conceptual model and evidence for growth. *Journal of Mathematics Teacher Education, 26*(4), 481–508. https://doi.org/10.1007/s10857-022-09538-3

Prediger, S., Fischer, C., Selter, C., & Schöber, C. (2019). Combining material- and community-based implementation strategies for scaling up: The case of supporting low-achieving middle school students. *Educational Studies in Mathematics, 102*(3), 361–378. https://doi.org/10.1007/s10649-018-9835-2

Siemon, D. (2019). Knowing and building on what students know: The case of multiplicative thinking. In D. Siemon, T. Barkatsas, & R. Seah (Eds.), *Researching and Using Progressions (Trajectories) in Mathematics Education* (pp. 6–31). Brill.

Stacey, K. Steinle, V., Price, B. & Gvozdenko, E. (2018). Specific mmathematics assessments that reveal thinking: An online tool to build teachers’ diagnostic competence and support teaching. In T. Leuders, K. Philipp, & J. Leuders (Eds.), *Diagnostic Competence of Mathematics Teachers* (pp. 241–261). Springer.
# Validity objections to comparative judgement

Ian Jones

Loughborough University, Department of Mathematics Education, UK; I.Jones@lboro.ac.uk

Comparative judgement approaches to assigning grades to students' work have received interest from mathematics education researchers over recent decades. These approaches involve assessors deciding which of two presented pieces of work is 'better', and the decisions are then converted into scores. Several objections have been raised to using comparative judgement for summative assessment and in this theoretical paper I respond to objections that such approaches are 'not valid'. These include objections that there is a lack of evidence supporting validity, and that researchers assert comparative judgement is 'intrinsically valid' in ways that are incomplete and inconsistent. I argue that most validity objections are addressed by published evidence, and that the validity of applying comparative judgement to mathematics education assessments is a special case.

Keywords: Comparative judgement; alternative assessment; summative assessment; validity.

## Comparative judgement

Comparative judgement approaches to generating scores or grades from students' exam scripts or test responses have been gaining attention in education over recent decades (Bartholomew & M. Jones, 2022; I. Jones & Davies, 2023). A key motivation for using comparative judgement approaches to assess mathematical learning is to encourage a shift away from the types of short, objective items that are rife in examinations and tests (Swan & Burkhardt, 2012). Short, objective items threaten consequential validity if we value mathematics students performing longer-form mathematical activities such as problem solving, sustained reasoning and explaining their understanding (NCTM Research Committee, 2013). Importantly, comparative judgement approaches can encourage the shift away from short, objective items because they readily reduce assessor inconsistency compared to marking for the case of open-ended and relatively unstructured assessment tasks across a range of mathematical topics and student age ranges (e.g. Davies et al., 2020; Hunter & I. Jones, 2018; I. Jones & Inglis, 2015).

The approach applies Thurstone's (1927) well-established Law of Comparative judgement which states that human beings are poor at making absolute judgements (e.g. determining the temperature in a room in Celsius) and good at making relative judgements (e.g. determining if it is colder in the room or outside). Thurstone's law can be applied to educational assessment by harnessing the collective judgement of a group of subject experts (Pollitt, 2012). In practice, an online platform presents pairs of students' written mathematical work ('responses') to expert assessors via an internet browser. Interested readers can try judging pairs of responses to the test prompt "What is an equation?" here: http://tinyurl.com/190523equation. Once many judgements have been collected from several assessors the binary decision data is converted into a unique score for each response. Due to space constraints I do not detail the mathematics or other statistical details here, but see Pollitt (2012) and I. Jones and Davies (2023) for technical details.
Comparative judgement approaches differ from marking because there are no rubrics, no scoring of short items, and no aggregation of scores into a single mark or grade. Instead, outcomes are grounded in several assessors' holistic judgement of direct evidence of student work. Validity has been conceptualised for comparative judgement assessments in terms of the "collective understanding of the construct by a relevant community of experts" (I. Jones & Inglis, 2015, p. 341). Validity has been empirically evaluated across mathematics education studies using a range of constructs including criterion, divergent and content validity (e.g. Bisson et al., 2019). Study foci have included undergraduate modules on calculus (I. Jones & Alcock, 2014), statistics (Bisson et al., 2016) and proof comprehension (Davies, Alcock & I. Jones. 2020), and secondary school topics including fractions (I. Jones et al., 2013), statistics (Marshall et al., 2020) and problem solving (I. Jones & Inglis, 2015). Studies have also investigated conceptual understanding across a range of topics at secondary level (I. Jones & Karadeniz, 2016) and primary level (Hunter & I. Jones, 2018). Across these studies and others the outcomes of using comparative judgement have been found to be robust in terms of assessor consistency (called 'reliability' here on; Verhavert et al., 2019), and in terms of validity (Bartholomew & M. Jones, 2022; I. Jones & Davies, 2023).

## The validity objections

Most of the objections that have I encountered have not been published in the scholarly literature but have arisen in other sources such as social media, blog posts, conversations at conferences or seminars, and reviewers' comments on submitted articles. I focus here on what I perceive to have been the most common, and I strive to present them coherently and fairly.

A common objection is that there is a lack of evidence in support of the validity of comparative judgement for summative assessment. For example, van Daal et al. (2022) wrote, in the context of education generally, "only a limited number of studies dig into the validity of comparative judgement" (p.2). Others go further, suggesting there is something inherently challenging about validity when it comes to comparative judgment (e.g. Bokhove, 2019). A particular concern is the opacity of assessment decisions lacking detailed criteria, which it has been alleged means that comparative judgement "drains from the assessment any considerations of *what* [sic.] is being assessed & the whole question of validity worth having" (Davis, 2017).

Perhaps the most substantive objections to the validity of comparative judgement approaches, including for mathematics education, came from a theoretical paper by Kelly, Richardson and Isaacs (2022). Before I present and respond to the key objections presented in their paper, it is worth clarifying that the vast majority of literature on using comparative judgement for educational assessment is authored by advocates, of which I am one. This has resulted in a body of literature overtly committed to demonstrating its virtues with few dissenting voices. Kelly et al.'s paper is therefore a needed and refreshing counter to the growing pro-comparative judgement corpus, to which the current paper adds, and it would be healthy for the discipline if further sceptical researchers publish scholarly critiques.

Kelly et al. critiqued what they called the "intrinsic validity" (p.2022) rationale for using comparative judgement. The intrinsic validity rationale is that validity can be defined in terms of what experts collectively deem 'good' answers to be and therefore, advocates of comparative judgement argue,
outcomes are inherently valid. Above, I quoted this as the “collective understanding of the construct by a relevant community of experts” (I. Jones & Inglis, 2015, p. 341). Kelly et al. offer four specific critiques of the intrinsic validity rationale: (i) there is no evidence relative judgements are superior to absolute judgements; (ii) researchers rarely define ‘expert’; (iii) non-experts, specifically students comparatively judging peers’ work, produce outcomes that correlate well with experts’ outcomes; (iv) researchers sometimes remove poorly performing (‘misfitting’) experts thereby undermining their very definition of validity. They argue that “it is inconsistent to justify the use of a method based on its psychological underpinnings, and also to contend that the details of this theory are irrelevant provided the method works in practice” (p.679).

Finally, an objection related to validity is that comparative judgment can only produced norm-referenced grades (e.g. Dolan, 2021). Norm-referenced grading involves allocating grades statistically, such as awarding the top 10% of scores a grade A. This means that grade A for a ‘weaker’ cohort is not equivalent to grade A for a ‘stronger’ cohort. In contrast, criterion-referenced grading involves allocating grades against agreed standards. This means that we would expect fewer grade A’s to be awarded to the ‘weaker’ cohort than the ‘stronger’ cohort (Lok, McNaught & Young, 2016). It has been asserted that comparative judgement can only be used for norm-referenced grading, presumably because each student’s performance is judged relative to the other students’ performances.

**Responding to the validity objections.**

It is the case that there could and should be research into the validity of comparative judgement for summative assessment, including across different mathematical topics, student age ranges, jurisdictions and so forth. However, most of the assertions of a lack of evidence seem not to acknowledge, let alone critique, the evidence that has been published over the past decade. In fact, and as mentioned above, numerous studies have evaluated constructs such as the convergent, divergent and face validity of comparative judgement assessments.

Convergent validity has been demonstrated by correlating or predicting outcomes with independent measures (standardised instruments, specially designed tests, achievement data, teacher estimates) across age ranges, learning contexts and mathematical topics (e.g. I. Jones et al., 2016; I. Jones et al. 2019; Marshall et al., 2020). For example, Bisson et al. (2019) conducted a randomised controlled trial using two outcome measures of students’ conceptual understanding of calculus – one based on comparative judgement and the other based on a traditional standardised test – and compared the results.

Divergent validity has been investigated using independent measures we would not expect to correlate with or predict comparative judgement-based mathematics outcomes. For example, Bisson et al. (2019) found that secondary students’ mathematics but not English GCSE grades predicted comparative judgement scores (GCSE refers to a terminal school qualification in parts of the United Kingdom). In another example, I. Jones et al. (2013) used comparative judgement to assess secondary students’ explanations of how to put fractions in order. They found that conceptual but not procedural measures of mathematical knowledge predicted comparative judgement scores, thereby demonstrating the potential of comparative judgement to reliably assess conceptual understanding
rather than procedural knowledge. Other studies have shown that non-expert judges (peers, lay people) produce comparative judgement outcomes that diverge from those of experts (e.g. I. Jones & Alcock, 2014; I. Jones & Wheadon, 2015).

Content validity has been demonstrated using various techniques including expert review (e.g. I. Jones & Inglis, 2015), thematic analysis and related coding methods (e.g. Davies et al. 2021), and interviewing or surveying judges (e.g. Hunter & I. Jones, 2018; I. Jones et al., 2014; Marshall et al., 2020). An increasingly common method is to qualitatively code students' responses using existing frameworks (e.g. I. Jones & Karadeniz, 2016) or grounded approaches (e.g. Davies et al., 2020), and then use multiple regression techniques to identify the features of high-scoring responses. For example, I. Jones and Karadeniz (2016) applied a published coding scheme (Hunsader et al., 2014) and found that quantity written, use of numbers, and use of graphics predicted comparative judgement scores, but other features such as use of letters or use of 'real-world' examples did not. Similarly, Davies et al. (2020) developed a grounded coded scheme for undergraduates' definitions of proof, and found that comparative judgement scores produced by research mathematician judges were consistent with typical characterisations of proof reported by philosophers of mathematics (Davies et al., 2020).

This body of evidence partly addresses Kelly et al.'s objection that researchers assume that comparative judgement is intrinsically validity. We have seen convergent, divergent and content validity has been investigated across a variety of contexts, and not merely assumed to be inherent, the case for summative assessment in mathematics education.

I now turn to Kelly et al.'s specific objections of the intrinsic validity rationale.

(i) Kelly et al. claimed that researchers have not established that relative judgements are superior to absolute judgements for producing reliable outcomes. However this is not the case, at least for the case of secondary school peer assessment. I. Jones and Wheadon (2015) showed that for absolute judgement outcomes, inter-rater reliabilities were effectively zero (mean r = -.02), whereas the relative (comparative judgement) outcomes, reliabilities were high (mean r = .86).

(ii) It is fair to claim that researchers tend not to provide a precise or universal definition of 'expert'. It is also fair to counter that expertise is clearly defined within the context of many published studies, such as "mathematics PhD students" (Davies et al., 2020, p. 188). Experts are also sometimes contrasted against non-experts such as peers or novices (e.g. I. Jones & Alcock, 2014). Nevertheless, these operationalisations and contrasts tend to be buried in methods sections, and researchers could and should be more upfront and explicit about the term 'expert'.

(iii) A related point is Kelly et al.'s objection that non-experts' outcomes sometimes "correlated well" (p. 682) with the outcomes of experts. This has indeed been the case in some studies (e.g. I. Jones & Alcock, 2014; I. Jones & Wheadon, 2015), and moreover has been used as an argument for peer judgements contributing to summative outcomes. However, there is an important caveat: studies typically use many times more peer judgements than expert judgements because the former are easier to obtain. Crucially, the number of judgements per student answer collected is correlated with reliability (see Verhavert et al., 2019, for a detailed explanation and demonstration), and this largely explains peers' robust outcomes. Moreover, when novices (e.g. participants who have received no
mathematics education beyond secondary school and have not studied calculus) make comparative judgements of undergraduate mathematicians' responses to an open-ended calculus question, correlations are lower still (e.g. I. Jones & Alcock, 2014).

(iv) Kelly et al.'s objection that removing 'misfitting' experts undermines claims that validity is intrinsic to collective expert judgement has merit. Opinions as to whether or not misfitting experts should be identified and removed varies across comparative judgement researchers in my experience. I am of the view that removing misfitting experts is generally not necessary or desirable (see I. Jones & Davies, 2023) bar rare occasions when an expert appears not to have judged in good faith, or with adequate attention (e.g. they completed their judgements suspiciously quickly). Therefore, Kelly et al. make a good theoretical point and researchers should consider carefully the rationale and validity of removing misfitting expert judges.

Finally, it is not the case as asserted by some (e.g. Dolan, 2021) that comparative judgment can only be used to produce norm-referenced grades. There are several methods are available for criterion-referencing comparative judgment scores when producing grade boundaries. For example, Marshall et al. (2020) included exemplar grade-boundary scripts in the judging pot of students' responses. The boundary scripts' scores were then used as cut-scores to assign grades. In fact comparing the standards of grades across different cohorts is a particular strength of comparative judgement approaches. For example, I. Jones et al. (2016) used comparative judgement to investigate changes over five decades of a terminal qualification (A-level Mathematics) in England. Conversely, albeit in the context of creative writing, Heldsinger and Humphry (2013) used comparative judgement to identify grade boundary scripts, which were then disseminated to teachers in order to establish consistent writing standards in Australia.

## Discussion.

I have presented objections to the validity of using comparative judgement to produce summative scores or grades. I have argued that in the main these objections are addressed by the published research evidence. That is not to say more validity evidence would be unwelcome. For example, recent developments have included eye-tracking equipment to investigate how experts make judgement decisions of argumentative writing (Gijsen et al., 2021), and such methods should be applied in the context of mathematics education. In addition, while criterion-referencing methods have been applied to comparative judgement outcomes (e.g. Marshall et al., 2020), these methods have not been widely published or, to the best of my knowledge, systematically investigated.

I agree with Kelly et al.'s conclusion that there is a need for a "comprehensive, systematic review of the evidence to explore to what extent the rationales for using comparative judgment have empirical support" (p. 684). Critical review is essential to scholarly progress and, based on the theoretical and detailed objections of Kelly et al., comparative judgement researchers should consider giving greater thought to and being clearer about their use of the term 'expert'. They should also reflect carefully on the implications for validity, including Kelly et al.'s critique of what they call intrinsic validity, when considering removing misfitting expert judges.
## Final comment

Objections to the validity of comparative judgement tend not to distinguish between different subjects. There is a motivation to use comparative judgement that is specific to mathematics education which I set out in the introduction to this paper: comparative judgement can reduce exam and test designers' dependence on short, objective test questions, and allow greater use of open-ended, relatively unstructured formats, without decreased assessor consistency (i.e. reliability). This is not the case for other subject areas. For example, researchers have investigated using comparative judgement to assess students' creative writing (e.g. Wheadon et al., 2020). Here the motivation, unlike for assessing mathematics, is not to enable different types of test questions, but to improve the reliability of assessing existing test questions (Pollitt, 2012).

To conclude, Kelly et al.'s "intrinsic validity" could be amended for the case of mathematics education to include the theoretical argument that comparative judgement enables the inclusion of relatively open and unstructured items in exams and tests, and therefore can increase their validity with no loss of reliability.

## References

* Bartholomew, S. R., & Jones, M. D. (2022). A systematized review of research with adaptive comparative judgment (ACJ) in higher education. *International Journal of Technology and Design Education, 32*, 1159–1190. https://doi.org/10.1007/s10798-020-09642-6

* Bisson, M.-J., Gilmore, C., Inglis, M., & Jones, I. (2016). Measuring conceptual understanding using comparative judgement. *International Journal of Research in Undergraduate Mathematics Education, 2*(2), 141–164. https://doi.org/10.1007/s40753-016-0024-3

* Bisson, M.-J., Gilmore, C., Inglis, M., & Jones, I. (2019). Teaching using contextualised and decontextualised representations: Examining the case of differential calculus through a comparative judgement technique. *Research in Mathematics Education, 22*(3), 284–303. https://doi.org/10.1080/14794802.2019.1692060

* Bokhove, C. ([@cbokhove]. (2019, June 27). *The time is absolutely not ripe for Comparative Judgement at a national scale in year 6 sats*. Validity, *total* time, accountability, logistics are challenges. Maybe just as additional moderation. I feel such desires mainly based on dislike of current system, not eval of CJ. Twitter. https://twitter.com/cbokhove/status/1144137828621680640?s=43&t=ohKl7Uu4ddr7VZQXzHd2sA*

* Davis, A. J. ([@ded6ajd]. (2017, April 8). *Comparative judgment drains from the assessment any considerations of *what* is being assessed & the whole question of validity worth having*. Twitter. https://twitter.com/ded6ajd/status/850639513399545856?s=43&t=ohKl7Uu4ddr7VZQXzHd2sA*

* Davies, B., Alcock, L., & Jones, I. (2020). Comparative judgement, proof summaries and proof comprehension. *Educational Studies in Mathematics, 105*(2), 181–197. https://doi.org/10.1007/s10649-020-09984-x
Dolan, T. [@timdolan]. (2021, February 2). *Maths Teachers: Has anyone considered using comparative judgement as part of Y11 or Y13 assessment? I wonder whether it's worth exploring, for data to help with valid inferences about mathematical ability. Would only be norm-referenced data, which may be of limited value? Twitter.* https://twitter.com/timdolan/status/1356532458771087360?s=43&t=ohKl7Uu4ddr7VZQXzHd2sA

Gijsen, M., van Daal, T., Lesterhuis, M., Gijbels, D., & de Maeyer, S. (2021). The complexity of comparative judgments in assessing argumentative writing: An eye tracking study. *Frontiers in Education, 314*. https://doi.org/10.3389/feduc.2020.582800

Heldsinger, S. A., & Humphry, S. M. (2013). Using calibrated exemplars in the teacher-assessment of writing: An empirical study. *Educational Research, 55*(3), 219–235.
https://doi.org/10.1080/00131881.2013.825159

Hunsader, P. D., Thompson, D. R., Zorin, B., Mohn, A. L., Zakrzewski, J., Karadeniz, I., Fisher, E. & MacDonald, G. (2014). Assessments accompanying published textbooks: the extent to which mathematical processes are evident. *ZDM, 46*, 797-813. https://doi.org/10.1007/s11858-014-0570-6

Hunter, J., & Jones, I. (2018). Free-response tasks in primary mathematics: A window on students’ thinking. *Proceedings of the 41st Annual Conference of the Mathematics Education Research Group of Australasia, 41*, 400–407.

Jones, I., & Alcock, L. (2014). Peer assessment without assessment criteria. *Studies in Higher Education, 39*(10), 1774–1787. https://doi.org/10.1080/03075079.2013.821974

Jones, I., & Davies, B. (2023). Comparative judgement in education research. *International Journal of Research & Method in Education*. https://doi.org/10.1080/1743727X.2023.2242273

Jones, I., & Inglis, M. (2015). The problem of assessing problem solving: Can comparative judgement help? *Educational Studies in Mathematics, 89*(3), 337–355.
https://doi.org/10.1007/s10649-015-9607-1

Jones, I., Inglis, M., Gilmore, C., & Hodgen, J. (2013). Measuring conceptual understanding: The case of fractions. In A. M. Lindmeier & A. Heinze (Eds.), *Proceedings of the 37th Conference of the International Group for the Psychology of Mathematics Education* (Vol. 3, pp. 113–120). IGPME.

Jones, I., & Karadeniz, I. (2016). An alternative approach to assessing achievement. In C. Csikos, A. Rausch, & J. Szitanyi (Eds.), *The 40th Conference of the International Group for the Psychology of Mathematics Education* (Vol. 3, pp. 51–58). IGPME.

Jones, I., Swan, M., & Pollitt, A. (2014). Assessing mathematical problem solving using comparative judgement. *International Journal of Science and Mathematics Education, 13*(1), 151–177. https://doi.org/10.1007/s10763-013-9497-6

Jones, I., & Wheadon, C. (2015). Peer assessment using comparative and absolute judgement. *Studies in Educational Evaluation, 47*, 93–101. https://doi.org/10.1016/j.stueduc.2015.09.004
Jones, I., Wheadon, C., Humphries, S., & Inglis, M. (2016). Fifty years of A-level mathematics: Have standards changed? *British Educational Research Journal*, *42*(4), 543–560.
https://doi.org/10.1002/berj.3224

Kelly, K. T., Richardson, M., & Isaacs, T. (2022). Critiquing the rationales for using comparative judgement: A call for clarity. *Assessment in Education: Principles, Policy & Practice, 29*(6), 674–688.
https://doi.org/10.1080/0969594X.2022.2147901

Lok, B., McNaught, C., & Young, K. (2016). Criterion-referenced and norm-referenced assessments: Compatibility and complementarity. *Assessment & Evaluation in Higher Education, 41*(3), 450–465.
https://doi.org/10.1080/02602938.2015.1022136

Marshall, N., Shaw, K., Hunter, J., & Jones, I. (2020). Assessment by comparative judgement: An application to secondary statistics and English in New Zealand. *New Zealand Journal of Educational Studies, 55*(1), 49–71.
https://doi.org/10.1007/s40841-020-00163-3

NCTM Research Committee. (2013). New Assessments for New Standards: The Potential Transformation of Mathematics Education and Its Research Implications. Journal for Research in Mathematics Education, 44, 340–352.
https://doi.org/10.5951/jresematheduc.44.2.0340

Pollitt, A. (2012). The method of Adaptive Comparative Judgement. *Assessment in Education: Principles, Policy & Practice, 19*(3), 281–300.
https://doi.org/10.1080/0969594X.2012.665354

Swan, M., & Burkhardt, H. (2012). Designing assessment of performance in mathematics. *Educational Designer, 2*(5), 1–41.
https://isdde.org/wp-content/uploads/2018/05/isdde09_burkhardt_swan.pdf

Thurstone, L. L. (1927). A law of comparative judgment. *Psychological Review, 34*, 273–286.
https://doi.org/10.1037/h0070288

van Daal, T., Lesterhuis, M., de Maeyer, S., & Bouwer, R. (2022). Validity, reliability and efficiency of comparative judgement to assess student work. *Frontiers in Education, 7*, 1100095.
https://doi.org/10.3389/feduc.2022.1100095

Verhavert, S., Bouwer, R., Donche, V., & Maeyer, S. D. (2019). A meta-analysis on the reliability of comparative judgement. *Assessment in Education: Principles, Policy & Practice, 26*(5), 541–562.
https://doi.org/10.1080/0969594X.2019.1602027

Wheadon, C., Barmby, P., Christodoulou, D., & Henderson, B. (2020). A comparative judgement approach to the large-scale assessment of primary writing in England. *Assessment in Education: Principles, Policy & Practice, 27*(1), 46–64.
https://doi.org/10.1080/0969594X.2019.1700212
# The students' use of visual representation for stimulating their metacognitive strategies: Teachers' perspectives

Amal Kadan-Tabaja

Haifa University, Faculty of Education, Israel; math33444@gmail.com

Visual representation is effective in enhancing mathematical learning and thinking processes. This study focuses on visual representations automatically provided by a formative assessment platform to describe students' mathematical strategies. We examined the teachers' perspectives on how the students' metacognitive strategies could be stimulated by visual representation of strategy (VRS) provided by a formative assessment platform in assignments for comparing fractions. Twenty-five teachers participated in this study. Based on different data resources, we were able to identify three categories where the teachers considered the VRS as a tool for stimulating students' use of metacognitive strategies: as part of class management, as part of task requirements, and as part of feedback information.

Keywords: Example-eliciting task, strategy, fraction, visual representation, metacognition.

## Introduction and theoretical background

Mathematical strategies refer to the methods used by students to solve problems with mathematical content, whether their answers are correct or not (Hegedus & Otálora, 2022). Researchers have reported that students use different strategies to successfully compare fractions, which serve as the mathematical content of this study. These strategies were included in the mathematical curriculum and taught in the classroom. The common strategies are using the same numerator, the same denominator, the benchmark of one whole or one half, the distance of each fraction from one whole, and finding equivalent fractions by expansion or reduction algorithms to get a common denominator or numerator. Ellis et al. (2019) argued that investigating mathematical features and properties of sets of examples may shed light on the students' strategies and thinking. In this study, we used STEP (Seeing the entire picture) as a formative assessment platform. STEP enables primarily: (a) example-eliciting tasks (EETs)—a task that includes an interactive diagram and asks students to generate examples by performing dragging under given constraints (Yerushalmy & Olsher, 2020); (b) assessing students' mathematical strategies based on automated analysis of the mathematical features of the students' examples; and (c) reflecting these strategies automatically and visually, which we call "visual representation of strategy" (VRS) (Kadan-Tabaja & Yerushalmy, 2023). Research shows that the technological platforms that provide an immediate picture of students' work may support the work of teachers and allow for better representation of the mathematical content and more effective student learning (Olsher, Yerushalmy, & Chazan, 2016). Research shows that students dealing with visual representation related to examples they had constructed may be more effective for their learning and further stimulate their thinking process (Robutti, 2010). In this study, we focused on the students' metacognitive strategies as the thinking process that refers to the learners' knowledge, planning, monitoring, and evaluating their strategies for learning and thinking in the cognitive process (Pintrich, 2002).
The novelty of this study lies in integrating the formative assessment platform with the students' answers to an EET, to represent their mathematical strategies automatically and visually. We examined the teachers' perspectives on how the students used the VRS that STEP provided to stimulate their metacognitive strategies. Our research question was: From the teachers' perspective, how can the students' use of VRS within a formative assessment platform stimulate their metacognitive strategies?

## Research setting

The interactive diagram of the task in this study was based on a similar representation as that mentioned in the literature (Figure 1) (Arnon et al., 2001), using the STEP platform. Arnon et al. (2001) studied and reported on students who learned with the Shemesh software, which was designed to promote conceptual learning of fractions, offering concrete representations of the fraction and the operations performed on it. Fractions are represented in the discrete Cartesian coordinate system by a point whose vertical coordinate is the numerator and its horizontal coordinate the denominator. All equivalent fractions are represented on a straight ray passing through the origin point. The origin and points on the vertical axis do not represent any fraction. All equivalent fractions are represented on a straight ray passing through the origin (e.g., $\frac{2}{5} = \frac{4}{10} = \frac{6}{15}$). Points that exist on a ray with a larger slope represent larger fractions (e.g., $\frac{6}{5} > \frac{2}{5}$). The red point (corresponding to an X in Figure 1) represents the given fraction (a fixed point), and the green point (corresponding to an empty circle in Figure 1) is the fraction that the student can drag freely to satisfy the requirement of the task.

The task on which this study was based, students were asked to construct a fraction larger than the given fraction $\frac{2}{5}$ represented by the red point by dragging the green point (Figure 2). The task required students to construct 10 examples of fractions that fit the requirement. Examples were submitted separately and captured in STEP. In response to the students' submissions of their examples, STEP visually represented each student's examples on a single screen using blue points and provided automated analysis both to teachers and students. Automated analysis for teachers represented the students' identified strategies of choosing examples both verbally and visually (Kadan-Tabaja & Yerushalmy, 2023). The automated analysis for students included a set of statements describing strategies of comparing fractions. Students were asked to read and activate each statement, then to mark which statement they used to choose their examples. Otherwise, to indicate the method they used to choose their examples. The statements marked by the students were automatically reflected as VRSs (Figure 3).

<figure><img src="image_45.png" /><figcaption>Figure 1: The interactive diagram</figcaption></figure>

<figure><img src="image_46.png" /><figcaption>Figure 2: The interactive diagram of the task</figcaption></figure>

<figure><img src="image_47.png" /><figcaption>Figure 3: The automated analysis for students</figcaption></figure>
The VRSs are based on automated analysis executed by mathematical algorithms that the researchers set up in STEP when they designed the task. In Table 1, based on Kadan-Tabaja and Yerushalmy (2023), we describe the strategies for comparing fractions (marked in underlined lowercase letters) and the automated VRSs that STEP provides. The teachers were asked to refer to these VRSs when they related to the students' use of the visual representations.

<table><caption>Table 1: Strategies for comparing fractions and the VRSs</caption><thead><tr><th>The strategies for comparing fractions</th><th>VRS</th></tr></thead><tbody><tr><td>S1. Comparing fractions by using a benchmark of one whole. The VRS takes the shape of a ray that separates between two regions. The green/red regions represent all the fractions that are larger/smaller than one, respectively. The ray between them represents fractions equivalent to one whole. The red fraction is less than one, and the submitted examples are equal to or larger than one. Thus, each blue fraction is larger than the red one.</td><td></td></tr><tr><td>S2. Comparing fractions by using a benchmark of one-half. The VRS takes the shape of a ray that separates between two regions. The green/red regions represent all fractions that are larger/smaller than one-half, respectively. The ray between them represents fractions equivalent to one-half. The red fraction is less than one-half, and the submitted examples are equal to or larger than one-half. Thus, each blue fraction is larger than the red one.</td><td></td></tr><tr><td>S3. Comparing fractions by using the same denominator. The VRS takes the shape of a ray that is parallel with the vertical axes in X= (the denominator of the red fraction). All submitted examples (blue) have the same denominator as the given red fraction.</td><td></td></tr><tr><td>S4. Comparing fractions by using the same numerator. The VRS takes the shape of a ray parallel with the horizontal axes in Y= (the numerator of the red fraction). All submitted examples (blue) have the same numerator as the given red fraction.</td><td></td></tr><tr><td>S5. Comparing fractions based on "the numerator and the denominator of the larger fraction are larger than the numerator and the denominator of the smaller fraction, respectively." The VRS takes the shape of a region that represents all fractions that have a larger numerator and denominator than the given red fraction. This is a misconception, and each blue fraction can be larger or smaller than the red one.</td><td></td></tr><tr><td>S6. The blue fraction is larger than the red one. This strategy is true when the student's example fulfills the requirements of the tasks. In the visual representation, each blue fraction is above the red ray which represents all the fractions that are equivalent to the red fraction, which means that each blue fraction is larger than the red one.</td><td></td></tr></tbody></table>
S7. Comparing fractions based on "the denominator of the larger fraction is a multiple of an integer of the denominator of the smaller fraction." The VRS takes the shape of rays that represent all fractions that are parallel with the vertical axes, X= a × (the denominator of the red fraction), where a is an integer that is not equal to zero. Thus, each blue fraction can be larger than the red one.

<figure><img src="image_48.png" /></figure>

## Methodology

Twenty-five elementary and secondary teachers volunteered to participate in the study. The teachers took part in 30 hours of teacher development workshops (10 sessions, 3 hours each) conducted by the first researcher, after which they conducted 4 activities on fractions that contained about 12 tasks. For this study, we focused on one task. Some of the sessions were held online, while others were conducted offline (Table 2).

<table><caption>Table 2: Teacher development workshops</caption><thead><tr><td>Online/offline sessions</td><td>Duration</td><td>The process of teachers' works in the session</td></tr></thead><tbody><tr><td>Online</td><td>12 hours</td><td>The teachers worked on the tasks in small groups (including the task in our study). They discussed the task requirements, the conceptions or misconceptions, the learning process, the different kinds of feedback, the use of the automated information that the platform enables. Each group documented the discussion in Google Slides and represented it in front of all development workshop participants.</td></tr><tr><td>Online</td><td>6 hours</td><td>The teachers were acquainted with the terms "students' self-regulation," "thinking process," and "metacognitive actions and reflection process."</td></tr><tr><td>Offline</td><td>12 hours</td><td>In the offline sessions of the workshops, each teacher was asked to observe one student engaging with the tasks through the automated VRS that STEP enabled, specifically with the task of this study. Then, the teachers were asked to answer a semi-structured questionnaire containing 10 questions.</td></tr></tbody></table>

## Data resources and analysis

To answer the research question, we made use of three resources: (a) video recordings of discussions in small groups (of five teachers each), with the teachers working on the task using the automated VRS in the online session. The transcripts of video recordings were analyzed to extract the statements showing the teachers' perceptions of how their students handle the task and the automated VRSs and how these representations may stimulate the students' metacognitive strategies. (b) While observing the group discussions, the first researcher took field notes. (c) Each teacher responded to a semi-structured questionnaire after observing one student working on the task using the automated VRS. The questionnaire contained ten questions regarding the examples and strategy the student used; challenges; difficulties faced by the student and the way the student handled them; and the insights that the teachers gained from the engagement with the task and the VRS to stimulate the students' thinking and learning. The analysis of the responses to the questionnaire and repeated reading of the
data from the transcripts allowed classifying the teachers' perceptions into categories. The final categorization of the data was checked by other researchers for consistency (the questions are listed in Table 3).

<table><caption>Table 3: List of questions in the questionnaire the teachers answered after observing one student working on the task using the automated VRSs</caption><tr><td>1.</td><td>Provide background information about the previous knowledge of the student.</td></tr><tr><td>2.</td><td>Is there a special reason for choosing this student?</td></tr><tr><td>3.</td><td>What were the characteristics of the fractions that the student chose?</td></tr><tr><td>4.</td><td>Which strategy did the student choose while constructing the examples?</td></tr><tr><td>5.</td><td>Were you able to identify difficulties, common mistakes, or misconceptions while the student was working on the task?</td></tr><tr><td>6.</td><td>Did the student change his/her choices while constructing his/her examples, and if yes, why?</td></tr><tr><td>7.</td><td>When the student's responses when he/she was exposed to the VRS?</td></tr><tr><td>8.</td><td>Do you have any suggestions for changes to the current task or the follow-up tasks? Explain.</td></tr><tr><td>9.</td><td>What are the insights that you gained from the task and the automated assessment? Explain.</td></tr><tr><td>10.</td><td>In your opinion, how can the use of the VRSs enhance the student's learning and thinking process?</td></tr></table>

We used a qualitative approach to analyze the data. In an open coding process, we examined excerpts from the teacher's responses and transcripts to describe the categories that would allow us to learn about how, in the teachers' opinion, the use of the VRS stimulated the students' metacognitive strategies. Based on Schoenfeld's (2013) metacognitive framework, we identified the following phrases and sentences that may reflect the metacognitive strategies (Table 4).

<table><caption>Table 4: Metacognitive action and statement</caption><thead><tr><td>Metacognitive strategy</td><td>When did it occur?</td><td>The following statement or action may be an example of a metacognitive strategies</td></tr></thead><tbody><tr><td>Planning</td><td>Before beginning a task</td><td>To understand what makes a correct answer...<br>To set specific strategies before beginning a task...<br>To make it easy... To reread the problem...</td></tr><tr><td>Monitoring</td><td>During the learning and feedback processes</td><td>To check the answers and the strategy while working on the task...<br>To ask questions...</td></tr><tr><td>Evaluating</td><td>After a learning episode</td><td>To summarize the learning or thinking after finishing...<br>To evaluate the conclusion that was reached...</td></tr></tbody></table>

## Results

The findings show that the teachers' statements according to which the students' use of the VRS may stimulate their metacognitive strategies can be classified into three categories. Below we describe these categories, giving examples from the teachers' statements, and point out the metacognitive strategies that may be stimulated.

a. The VRS is part of classroom management. This category is related to the use of the VRS when students work on the task individually, in pairs, in small groups, or whole-class discussion. For
example, one teacher said in the course of an online session: "I think that working on different VRSs in small groups or pairs may help students rethink their answer and compare it with those of others." Another teacher answered in the questionnaire: "It will be very interesting to have a discussion about the different visual representations and to connect them with the strategies for comparing fractions in the classroom." These two examples show that monitoring is a metacognitive strategy that may be stimulated by using the VRSs.

b. The VRS is part of task requirements (or task design), as when using the automated VRS as part of self-reflection before submitting the examples or when the task requirement is specifically related to the visual representation. One teacher stated: "When the student is asked to reflect on the VRS as part of the task requirements, this may be a clue for the student in the choice of examples." Another teacher said in the online session discussion: "We can use the visual representations in tasks to help students rethink and ask questions about their strategies for comparing fractions." According to the first example, planning is the metacognitive strategy that may be stimulated by using the VRS. The second example suggests that monitoring is the metacognitive strategy that may be stimulated by using the VRSs.

c. The VRS is part of feedback information. This category is related to the use of the automated VRS when it is related to the correctness, characteristics, and strategy chosen, and to misconceptions or common mistakes in their examples. For example, one teacher stated that "when the student was exposed to the VRS, he wondered which VRS he might have received had he used another strategy. So, the feedback visual representation made the student think again about the task and about his examples." Another teacher claimed: "The visual representations helped the student check whether his examples were correct or not; he was also able to check the strategy he used for comparing fractions visually." The first example demonstrates that monitoring is the metacognitive strategy that may be stimulated by using the VRSs. The second example demonstrates that evaluating is the metacognitive strategy that may be stimulated by using the VRSs.

Table 5 shows categorization of the metacognitive strategies described by the teachers.

<table><caption>Table 5. Categorization of the metacognitive strategies described by the teachers</caption><thead><tr><td>The metacognitive strategy</td><td>The visual representation of the strategy is part of class management</td><td>The visual representation of the strategy is part of task requirements</td><td>The visual representation of the strategy is part of feedback information</td></tr></thead><tbody><tr><td>Planning</td><td></td><td>The students' use of the VRS may help them understand what makes a correct answer for the task; it may encourage them to look at a broader range of rich strategies and to compare the information across different strategies; and it may help them visualize the sequence of steps of their answer.</td><td></td></tr></tbody></table>
<table><tr><td>Monitoring</td><td>Clustering the students in pairs or small groups for work on tasks using the VRS may help them present, compare, and discuss their strategies with other students. This may enhance the student's mathematical discourse.<br/>The students' use of VRSs in the classroom discussion may help them rethink their answers and see the problem from different perspectives.</td><td>The students' use of the VRS as part of the task requirement may help them compare, rethink, and adjust their strategy.<br/>It may help expose students to strategies and initiate an inquiry process to comply with the task requirements; it may encourage students to generate new strategies based on reasoning and exploration.<br/>It may support students in assuming ownership of their learning process and responsibility for it.<br/>It may help students enrich their strategic example space.<br/>It may encourage students to think in various modes and understand the concept from multiple perspectives.</td><td>The students' use of the VRS as part of the feedback information may lead them to assume responsibility for their learning and become independent thinkers.<br/>It may help students rethink their examples and ask questions.</td></tr><tr><td>Evaluating</td><td>The students' use of the VRS in the classroom discussion may expose them to new strategies or interesting answers, which may be used for clarification of their questions or comments.</td><td>The students' use of the VRS may help them assemble the lists of evaluation criteria of different strategies by which to assess their examples.<br/>It may encourage students to reconsider their answers before submitting them.</td><td>The students' use of the VRS as part of feedback information may help them evaluate the correctness and characteristics of their answers, the strategy they used to choose their examples, and the misconceptions or common mistakes in them.</td></tr></table>

## Discussion and conclusion

In this study, we used Schoenfeld's (2013) metacognitive framework to describe, from the teachers' perspectives, how the visual representation of strategy in an automated assessment platform may stimulate the students' metacognitive strategies. Based on the teachers' responses, we were able to identify three categories that related to the students' use of VRSs included in the formative assessment platform and the way each category may stimulate the students' metacognitive strategies—the use of the visual representation when it was part of classroom management, part of task requirements, and part of the feedback information. In each category, teachers stated that monitoring and evaluating actions may stimulate metacognitive strategies. Planning was identified in the students' use of the visual representation when it is part of the task requirements. The findings are consistent with the literature, which reported that visual representation may stimulate the students' mathematical thinking
(Boonen, 2016) and play a central role in the formative assessment process, as perceived by the teachers (Kadan-Tabaja, & Yerushalmy, 2023).

From the teachers' perspective, integrating EETs that automatically and visually reflect students' mathematical thinking into the formative assessment platform appears to stimulate students' metacognitive strategies. The results of this study open new possibilities for using such VRSs in the automated assessment processes in the mathematics classroom.

## References

Arnon, I., Nesher, P., & Nirenburg, R. (2001). Where do Fractions Encounter their Equivalents? Can this Encounter Take Place in Elementary-School?. International Journal of Computers for Mathematical Learning, 6(2), 167–214. https://doi.org/10.1023/a:1017998922475

Ellis, A. B., Ozgur, Z., Vinsonhaler, R., Dogan, M. F., Carolan, T., Lockwood, E., ... & Zaslavsky, O. (2019). Student thinking with examples: The criteria-affordances-purposes-strategies framework. The Journal of Mathematical Behavior, 53, 263–283. https://doi.org/10.1016/j.jmathb.2017.06.003

Hegedus, S. J., & Otálora, Y. (2022). Mathematical strategies and emergence of socially mediated metacognition within a multi-touch Dynamic Geometry Environment. Educational Studies in Mathematics, 1–19. https://doi.org/10.1007/s10649-022-10170-4

Kadan-Tabaja, A., & Yerushalmy, M. (2023). The role of automated assessment in representing students' strategies for comparing fractions. In Thirteenth Congress of the European Society for Research in Mathematics Education (CERME13).

Olsher, S., Yerushalmy, M., & Chazan, D. (2016). How might the use of technology in formative assessment support changes in mathematics teaching?. For the learning of mathematics, 36(3), 11-18.

Pintrich, P. R. (2002). The role of metacognitive knowledge in learning, teaching, and assessment. Theory into Practice, 41(4), 219-25.

Robutti, O. (2010). Graphic calculators and connectivity software to be a community of mathematics practitioners. ZDM – Mathematics Education, 42, 77-89.

Schoenfeld, A.H. (2013). Reflections on problem solving theory and practice. The Mathematics Enthusiast, 10(1–2), 9–34

Yerushalmy, M., & Olsher, S. (2020). Online assessment of students’ reasoning when solving example-elicitimg tasks: Using conjunction and disjunction to increase the power of examples. ZDM, 52(5), 1033–1049. https://doi.org/10.1007/s11858-020-01134-0
# Using e-assessment for interactive example-generation tasks

George Kinnear<fnref n="1" /> and Igor' Kontorovich<fnref n="2" />

The paper concerns the use of e-assessment systems to advance students' example spaces of mathematical concepts. We report on a pilot study with seven first-year students of linear algebra. Three students engaged with the static version of the e-task, which asked them to generate three examples that were as different as possible. The remaining students were prompted by an interactive e-task that assessed the provided examples and asked for another one with different properties. The analysis of a single task about matrices showed that the students who worked with the interactive e-task were more successful than those working with the static e-task, in terms of the number and the range of generated examples. These preliminary findings open the doors to further study of the design and use of interactive example-generation tasks in university mathematics education.

Keywords: Assessment, concepts, feedback, learner-generated examples.

## Introduction

Example-generation tasks have been suggested as an effective way to promote students' learning about concepts (Watson & Mason, 2005), and as a way for researchers (and teachers) to gain insight into students' understanding of concepts (Zazkis & Leikin, 2007). E-assessment offers the potential to provide large undergraduate classes with formative example-generation tasks, giving automated feedback on a scale that would not be feasible for teachers to do manually (Sangwin, 2003). Such formative tasks can serve a dual purpose: on the one hand, students' responses provide the teacher with assessment information about their students' knowledge of concepts; on the other hand, the tasks can also prompt students to consider examples that they might not otherwise think about, thereby promoting further learning about the concepts. The design of e-assessment tasks that expand students' understanding of concepts and the capability to generate concept examples has been identified as an open question that is particularly of interest at the undergraduate level (Kinnear et al., 2022).

Fahlgren and Brunström (2023) note the potential of e-assessment tasks that give feedback in the form of prompts for further examples, with the prompts depending on the examples given so far by the student. Such an approach would automate the recommended approach in clinical interview settings, of “asking for ‘another and another’ [example] and for ‘something different’” (Zazkis & Leikin, 2007, p. 19). Prompting in this way may both stimulate the learners to consider examples beyond the most immediately obvious ones and provide richer data about the learners’ knowledge.

Here we describe the design and evaluation of interactive example-generation tasks, that prompt students for further examples based on the examples they have given so far, implemented as prototypes in an e-assessment system. The tasks address topics in linear algebra, and were devised and piloted in collaboration with a group of fourth-year students as part of their undergraduate research project. Our overarching aim was to develop e-assessment tasks that prompt students to generate a rich range of examples. We were particularly interested in the effect of interactivity on students’ example-generation activity, so we developed static versions of the tasks to serve as a

<footnote n="1">¹The University of Edinburgh, UK; G.Kinnear@ed.ac.uk</footnote>

<footnote n="2">²The University of Auckland, New Zealand; i.kontorovich@auckland.ac.nz</footnote>
comparison. The research question guiding this study was: *how do the examples produced by students compare between static and interactive e-assessment tasks?*

**Theoretical background**

A central theoretical notion of this study is an *example space* (Watson & Mason, 2005), which is a collection of possible examples of a mathematical concept. Watson and Mason define a conventional example space for a given concept as that “generally understood by mathematicians” (p. 62). Each individual has their own personal example space, based on their past experience, and will access it in different ways depending on the situation (e.g., in response to particular cues in a task). The structure of example spaces can be described by the *dimensions of possible variation* (DofPV), which are the features of examples that can vary, and their associated *range of permissible change* (RofPCh). For instance, if asked for a quadratic polynomial, one DofPV is the coefficient of $x^2$, for which the RofPCh is any non-zero real number. Fahlgren and Brunström (2023) used these notions to analyse three example-generation tasks, and we similarly used them to guide the design and analysis of our tasks.

Zazkis and Leikin (2007) proposed a framework for characterizing students’ example spaces, that includes a focus on *accessibility* of the examples: what are the most obvious concept examples, and how readily can students generate examples beyond those. Watson and Mason (2005) offer the metaphor of “example space as larder”, where finding an example can be thought of as “either immediately picking out something familiar or having to look for it for a while” (p. 162). According to the *principle of intellectual parsimony*, “when solving a problem, one intends not to make more intellectual effort than the minimum needed” (Koichu, 2010, p. 217), so it may be that students will not consider less accessible examples without explicit prompting.

Watson and Mason (2005) give advice about designing example-generation tasks that prompt students for a range of examples. They suggest asking for a sequence of examples satisfying additional constraints, as the “increasing constraints extend awareness of what is possible” (p. 132). In this way, example-generation tasks can help to draw learners’ attention to DofPV (or to the extent of the associated RofPCh) that they were not previously aware of. Arzarello et al. (2011) further argue that helping students to see structure in example spaces through this sort of prompting is a key role for teachers. Our study begins to explore how to prompt students in this way using e-assessment.

**Method**

To investigate the potential for interactive example-generation tasks to probe students’ example spaces, we developed three prototype tasks for use in clinical interviews with undergraduate students.

**Participants and protocol**

We invited students taking a first-year linear algebra course to participate in the interviews, near the end of the semester while they were revising for the final assessment. The course is compulsory for students on mathematics and computer science degree programmes, and an option for students on many other degree programmes. The course operates using a flipped classroom design, with preparatory activities that include weekly e-assessment quizzes (for further details about the course, see Docherty, 2023). Out of 581 students on the course, 10 students volunteered to participate in the interviews. During the interview, each student was asked to complete three tasks, with each task in a
different format: on paper, as a static e-assessment task, and as an interactive e-assessment task. The task-format combinations were permuted across the 10 interviews, with each task-format combination occurring 3 or 4 times in total. After the students had completed each task, the interviewer asked them to explain how they produced their answers, and whether they could give any other types of examples that had not been covered so far. The interviews were audio-recorded and transcripts were produced for the relevant episodes. However, our main analysis is based on the students' concept examples as recorded by the e-assessment system.

## Materials: design of the static and interactive tasks

Three tasks on linear algebra were created for the study, addressing concepts from the course: Eigenvalues, Span, and Reduced row-echelon form. There were two e-assessment versions of each task: static and interactive. The static e-assessment version asked students to provide three examples, and to “try to make each example as different as you can.” The interactive e-assessment version asked for a single example; depending on the answers given, students were then prompted for further examples with different properties. Full details for all the tasks can be found at https://osf.io/7wrgz. In this paper, we focus on the Eigenvalues task, which asked students for examples of matrices with eigenvalues 1 and 5 (as shown in Figure 1).

<figure><img src="image_49.png" /><figcaption>Figure 1: The Eigenvalues task: the static version (A) asked for three different examples, while the interactive version (B) asked for one initial example before prompting for further examples (C).</figcaption></figure>

For the Eigenvalues task, we anticipated that the most accessible example for students would be the 2 × 2 diagonal matrix with entries 1 and 5 on the diagonal. We designed the interactive version to
prompt for further examples by exploring the DofPV that we identified while devising the task. One DofPV is the type of matrix: in addition to diagonal matrices, upper/lower triangular and non-triangular matrices are possible. According to the principle of intellectual parsimony (Koichu, 2010), we expected that students would first try a triangular example (since that would avoid the need for any calculation of eigenvalues) and would only resort to constructing a non-triangular example if specifically prompted to do so. Another DofPV is the size of the matrix, which can be $n \times n$ for any $n \ge 2$ (i.e., the range of permissible change is $n \ge 2$). This dimension may not be immediately obvious to students (despite the course dealing with matrices of different sizes), since the two eigenvalues provided in the task may cue students to think of $2 \times 2$ examples. Moreover, the task is deliberately vague in not specifying that 1 and 5 are the *only* eigenvalues, and in not specifying their multiplicity.

We implemented the interactive tasks in the STACK e-assessment system, using the feedback messages displayed after a student submitted an answer. If the answer was incorrect, a feedback message was displayed (e.g., "This matrix does not have eigenvalues 1 and 5. Try again!"); the student could then change their answer and re-submit. If the answer was correct, the feedback message included hidden JavaScript code that revealed the next prompt and input box (the code is available at https://osf.io/7wrgz). We opted to use this relatively simple design, even though it imposed a constraint on the decisions about which prompt to show next: these decisions could only be based on the first and last submitted answers, rather than the full sequence of examples provided so far. In line with this constraint, we developed a flowchart for deciding which prompt to present next. The flowchart for the interactive Eigenvalues task is shown in Figure 2.

<figure><img src="image_50.png" /><figcaption>Figure 2: Flowchart showing the properties that were checked to determine the next prompt in the interactive Eigenvalues task</figcaption></figure>
of the leading diagonal. If you take the characteristic polynomial it ends up $(x-1)(x-1)(x-5)$. They were not confident in their final example ("it's possibly wrong") but reasoned using the cofactor expansion for computing determinants, where "I think the zeros would cancel out." S6 also reasoned using the cofactor expansion, but gave an incorrect $3 \times 3$ example. When asked how they produced this example, they explained that when choosing the matrix entries, "as long as I have one row or one column of zeros, I can fill in any random values" (i.e., they appeared to overlook the need to consider the determinant of the bottom-right $2 \times 2$ sub-matrix). S6 was able to produce a correct non-diagonal example: an upper-triangular $2 \times 2$ matrix that they described as "a variation of" their first example.

<table><caption>Table 2: Student responses to the static Eigenvalues task.</caption><thead><tr><th>Student 5</th><th>Student 6</th><th>Student 7</th><th>Student 9</th></tr></thead><tbody><tr><td>[1 0]<br>0 5]</td><td>[1 0]<br>0 5]<br><br>[1 0 1]<br>0 5 2<br>0 3 1]<br><br>[1 2]<br>0 5]</td><td>[1 0]<br>0 5]<br><br>[1 3 6]<br>0 1 4<br>0 0 5]<br><br>[5 0 0]<br>2 1 9<br>5 0 5]</td><td>[1 0]<br>0 5]</td></tr></tbody></table>

A summary of the results is shown in Table 3, where each student's examples are classified according to the two DofPV that we identified for this task: the type of matrix and size of matrix. This enables comparison between students (and between the two task modalities), in terms of which DofPV they showed awareness of through their examples (e.g., providing both $2 \times 2$ and $3 \times 3$ examples shows awareness of the "size of matrix" DofPV). For the static version, S5 and S9 did not show awareness of either DofPV since they provided only one example, while S6 and S7 showed awareness of both DofPV. For the interactive version, all three students showed awareness of both DofPV, although only S1 provided a triangular example within the "type of matrix" DofPV.

<table><caption>Table 3: Summary of student responses to the Eigenvalues task showing awareness of DofPV as evidenced by correct (filled disc) or incorrect (hollow disc) examples.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">Type of matrix</th><th colspan="2">Size of matrix</th></tr><tr><th>Diagonal</th><th>Triangular</th><th>General</th><th>2 × 2</th><th>3 × 3</th></tr></thead><tbody><tr><td rowspan="4">Static</td><td>S5</td><td>●</td><td></td><td></td><td>●</td><td></td></tr><tr><td>S9</td><td>●</td><td></td><td></td><td>●</td><td></td></tr><tr><td>S6</td><td>●</td><td>●</td><td>○</td><td>●</td><td>○</td></tr><tr><td>S7</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td rowspan="3">Interactive</td><td>S2</td><td>●</td><td></td><td>○</td><td>●</td><td>●</td></tr><tr><td>S10</td><td>●</td><td></td><td>●</td><td>●</td><td>●</td></tr><tr><td>S1</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr></tbody></table>
Discussion

We developed prototypes of interactive example-generation tasks and tested them with students in a laboratory setting, alongside static versions of the same tasks. The analysis presented here focused on one task, about matrices with given eigenvalues. Overall, the interactive version of the task appears to have been successful in prompting students to consider both DofPV that we had targeted in the design of the task. This stands in contrast to the static version of the task, where two of the four students provided only a single example and therefore did not demonstrate awareness of any DofPV (though they were later able to do this with explicit prompting from the interviewer).

Interactive example-generation tasks have the advantage of providing a controlled approach for prompting learners to generate examples, which is important when seeking to “make inferences about participants’ knowledge from the examples they generate” (Zazkis & Leikin, 2007, p. 19). Since all the prompts in the e-assessment task need to be decided in advance, careful thought should be given to the sequence of prompts; and where the e-assessment system has powerful mathematical capabilities, properties can be checked more quickly and reliably than an interviewer or a teacher could manage to do on the spot. However, the pre-designed prompts may turn out to be sub-optimal when learners respond in unanticipated ways. For instance, with the Eigenvalues task, S2 and S10 were not prompted to produce a triangular example since they skipped over that step in the anticipated sequence of “diagonal → triangular → non-triangular” (i.e., the triangular examples were less accessible than was anticipated). This demonstrates the value of pilot studies in the development of interactive example-generation tasks, so that the design can be refined in light of students’ responses.

The results of this pilot study of the Eigenvalues task suggest that the flowchart that we developed (Figure 2) could be refined to ensure that students are prompted to consider the full range of anticipated DofPV. We have already begun to work on a more sophisticated implementation, that could make decisions about which prompt to give based on details of the entire sequence of examples generated so far: for instance, “both of your examples are 2 × 2, can you give one that is 3 × 3?”, or “could you think of a simpler example, like a triangular matrix?”. This approach could be combined with asking for two or three different examples at the outset (similar to Fahlgren & Brunström, 2023), to get some indication of the most accessible examples and the DofPV that the student is aware of, and thus determine which DofPV would be fruitful ones to explore next.

Two further improvements could be considered for this task, which may also apply to other tasks. First, it could be worthwhile to offer students hints about how to proceed if they struggle to generate an example, even for types of example that are expected to be the most accessible. For instance, S2 was unsure how to proceed after their second example was incorrect; the interviewer intervened to move them on, and without this intervention they may not have had the chance to consider other DofPV. This sort of intervention could perhaps be formalized, by giving the student the option of asking for a hint. Second, the task could include prompts for examples that are not possible, to probe the students’ understanding of the RofPCh for particular DofPV. For instance, students could be asked to “give an example of a 2 × 2 matrix with eigenvalues 1, 2 and 3 (or enter none if this is not possible)”.

This was a small-scale pilot study of these tasks, so we do not seek to make any strong claims about
the comparison between the interactive and static task formats. However, our findings do suggest that
the interactive example-generation task was able to stimulate learners to consider a broad range of examples, and that this range was broader than the one demonstrated by the static task group. Indeed, students completing the static versions of the tasks admitted to giving up on the instruction to produce examples that were "as different as possible". We believe that the interactive example-generation task format therefore warrants further development and study.

## Acknowledgment

We would like to thank Molly Berrick, Abbie Whitehouse, Eilidh Millin and Álvaro Beattie Eizaguirre who collected data for this study as part of their fourth-year project at the University of Edinburgh.

## References

Arzarello, F., Ascari, M., & Sabena, C. (2011). A model for developing students’ example space: The key role of the teacher. *ZDM, 43*(2), 295–306. https://doi.org/10.1007/s11858-011-0312-y

Docherty, P. (2023). Case study 3: An introductory linear algebra course. In A. Wood (Ed.), *Effective Teaching in Large STEM Classes* (pp. 9-1–9-9). IOP Publishing. https://doi.org/10.1088/978-0-7503-5231-4ch9

Fahlgren, M., & Brunström, M. (2023). Designing example-generating tasks for a technology-rich mathematical environment. *International Journal of Mathematical Education in Science and Technology*, 1–17. https://doi.org/10.1080/0020739X.2023.2255188

Kinnear, G., Jones, I., Sangwin, C., Alarfaj, M., Davies, B., Fearn, S., Foster, C., Heck, A., Henderson, K., Hunt, T., Iannone, P., Kontorovich, I., Larson, N., Lowe, T., Meyer, J. C., O’Shea, A., Rowlett, P., Sikurajapathi, I., & Wong, T. (2022). A collaboratively-derived research agenda for E-assessment in undergraduate mathematics. *International Journal of Research in Undergraduate Mathematics Education*. https://doi.org/10.1007/s40753-022-00189-6

Koichu, B. (2010). On the relationships between (relatively) advanced mathematical knowledge and (relatively) advanced problem-solving behaviours. *International Journal of Mathematical Education in Science and Technology, 41*(2), 257–275. https://doi.org/10.1080/00207390903399653

Sangwin, C. J. (2003). New opportunities for encouraging higher level mathematical learning by creative use of emerging computer aided assessment. *International Journal of Mathematical Education in Science and Technology, 34*(6), 813–829. https://doi.org/10.1080/00207390310001595474

Watson, A., & Mason, J. (2005). Mathematics as a Constructive Activity. Routledge. https://doi.org/10.4324/9781410613714

Zazkis, R., & Leikin, R. (2007). Generating examples: From pedagogical tool to a research tool. For the Learning of Mathematics, 27(2), 15–21.
# Leveraging formative assessment to develop students' mathematical reasoning through images and student generated language

Richard Kitchen<fnref n="1" /> and Janet Lear<fnref n="2" />

A sixth-grade teacher who taught at a highly diverse school in the United States participated in professional development activities during the 2022-23 school year to learn about and implement an instructional protocol designed for use during problem-solving lessons. The protocol is intended to guide teachers to simultaneously attend to developing their students' mathematical reasoning and learning of the mathematics register. In a case study undertaken to examine teachers' use of the protocol, an example emerged of how formative assessment can be leveraged to support the development of students' mathematical reasoning through student generated language and images. This study contributes to the research literature by demonstrating how the use of formative assessment can promote the use of language to “carry” a mathematical concept in the sense that language can help students create a mental image of that concept.

Keywords: Formative assessment, instructional innovation, mathematical reasoning.

## Introduction

During the 2022-23 school year, the lead author collaborated with “Ms. Diaz,” a sixth-grade teacher who taught at a diverse school district in northern New Mexico, USA. Ms. Diaz volunteered to learn about and use an instructional protocol designed specifically for use during problem-solving lessons. The protocol, referred to as the “Discursive Mathematics Protocol” (DMP), is intended to be used as a guide by teachers to support students to develop both their mathematical reasoning and learn the mathematics register. Additional information about the DMP can be found in Matute (2022). In this paper, a vignette is shared from one of the problem-solving lessons in which Ms. Diaz implemented the DMP. The vignette demonstrates how formative assessment can be leveraged to support the development of students’ mathematical reasoning through student generated language and images. The following research question is addressed in this study: What is an example of how formative assessment can be used to support students’ mathematical reasoning through student generated language? Brief reviews of the research literature on formative assessment and the role of language and images in the learning of mathematical concepts are now provided.

## Formative assessment in mathematics

Classroom assessments are used to inform teachers, students and parents about student knowledge and understanding of mathematical concepts, processes and skills (Wiggins, 1993). There are two categories of classroom assessments; summative and formative. Summative assessment formats focus on what students know at a given time (Guskey & Bailey, 2001). Formative assessments differ from summative assessment in that the focus is not just on summarizing students' learning, but on using student learning data to inform instruction. After examining 250 research studies on classroom assessments, Black and William (1998) found that when teachers focus on formative assessment,

<footnote n="1">¹University of Wyoming, School of Teacher Education, Laramie, WY, USA; rkitchen@uwyo.edu</footnote>

<footnote n="2">²University of Wyoming, School of Teacher Education, Laramie, WY, USA; jlear2@uwyo.edu</footnote>
student achievement gains are among the largest ever reported for educational interventions. Formative assessment can include any of the following: classroom observation, inquiry, group work, whole class discussions, peer assessment, written work, individual interviews, student self-assessment, and portfolio assessment (Gearhart & Saxe, 2004). The vignette provided below resulted from the interplay of classroom observation, inquiry, group work, and a whole class discussion.

## The role of language and images in the learning of mathematics

Words are referents to mental images (Arnheim, 1969) that teachers can leverage to support their students' mathematical reasoning and learning (Gonzales, 2004). As one example, Kieren (1988) described how students used visual images associated with cutting, symmetry, and numerical halving to make sense of and express their ideas about the notion of something being a half of a whole. The deliberate use of images and language is particularly relevant in problem-solving based lessons given that representations are used to understand problems and devise solutions to those problems (Pólya, 1945/1986). In addition, students engage in language-rich activities to problem-solve by making conjectures, conceiving arguments, and formulating and carrying out proofs (Schoenfeld, 2013). The study's theoretical perspective is introduced next followed by an explanation of the research methods.

## Theoretical framework

Given the considerable attention given to cognition, sense making, and social discourse in the DMP, our theoretical framework is social-constructivism (Shepard, 2000). In this study, we viewed cognition through a measurement lens as students solve a task introduced below, examined students' mathematical reasoning as they made sense of images and used language to devise solution strategies and test those strategies (Shepard, 2000), and situated student learning as occurring through their engagement in discourse in communities of practice (Erath et al., 2021).

## Methods and data sources

This study was part of a case study research project that examined the DMP with teachers such as Ms. Diaz. In case studies, a real-life, bounded system or entity is selected to study within a particular setting (Yin, 2014). Throughout the 2022-23 school year, the following data was collected: observation data, videos of teachers and students who provided consent, student work and interviews with the two participating teachers. The data presented here were collected during one problem-solving lesson led by Ms. Diaz with her sixth-graders. A co-teaching approach (Cook & Friend, 1995) was used to plan and deliver the task. During the implementation of the task "Height Requirements" (See Figure 1), the lead author facilitated whole class discourse with Ms. Diaz as shown below.

At Sea World San Diego, kids are only allowed into the Air Bounce if they are between 37 and 61 inches tall. They are only allowed on the Tide Pool Climb if they are 39 inches tall or under:

1. Represent the height requirements of each ride in words.

3. Show the allowable heights for the rides on separate number lines.

4. Using inequalities and a number line, describe the height of kids who can go on both the Air Bounce and the Tide Pool Climb. Explain how you figured this out.
2. Represent the height requirements of each ride with inequalities. Explain the meaning of the terms you used for each of your inequalities.

Adapted from IM: https://tasks.illustrativemathematics.org/content-standards/6/EE/B/8/tasks/2010

<caption>Figure 1: Height Requirements Task</caption>

We used social-constructivism (Shepard, 2000) to examine students' mathematical reasoning in conjunction with how language was generated by Ms. Diaz and her students to communicate ideas. As will be shown, a vignette emerged as students were attempting to address the task's fourth question which involved determining the range of kids' heights that would allow them to go on the two rides.

## Results

In a video vignette described below, the phrase “in between” became a focus of instruction. What inspired the use of this phrase was that “Elena” used it when describing the shaded portion of her number line between 37 and 39 in a small group discussion with her peers and the first author. She shared, “It could just go in between 37 and 39.” After a short discussion in which Elena explained further, another girl in her small group explained why the answer must be in between 37 and 39: “Because 37 is how far you have to be for the Air Bounce and 39 is how tall you have to be for the Tide Pool.” In a whole class discussion, students in the class were asked to revoice Elena’s idea. The following discussion ensued between two students and the lead author:

<table><tr><td>1</td><td>S1:</td><td>What she's basically saying is, like, you know how the Tide Pool ends at 39 and the Air Bounce starts at 37?</td></tr><tr><td>2</td><td>RK:</td><td>Yeah.</td></tr><tr><td>3</td><td>S1:</td><td>You say 37, 38, and 39 inches could ride both the rides.</td></tr><tr><td>4</td><td>RK:</td><td>Kids who are 37, 38 or 39 inches tall?</td></tr><tr><td>5</td><td>S1:</td><td>Yeah.</td></tr><tr><td>6</td><td>RK:</td><td>What if the kid is 37 and a half inches tall?</td></tr><tr><td>7</td><td>S2:</td><td>They can still ride it.</td></tr><tr><td>8</td><td>RK:</td><td>Why? Are they still in between 37 and 39 inches tall?</td></tr><tr><td>9</td><td>S1:</td><td>Yeah.</td></tr></table>

At this point, Ms. Diaz encouraged “Luna” to come forward to share her ideas at the front board. Luna described her double number line answer that shows an overlap of heights that would allow children to take both rides. Luna then proceeded to modify her written response to the task’s fourth question displayed on the board by including the phrase “in between” in her response, observed by the entire class. Others leveraged this phrase as well in their written solutions to the task’s fourth question, helping them derive the range of kids’ heights that would allow them to go on both rides.

## Discussion

The use of formative assessment, specifically an observation made of student generated ideas, provided the means to engage students in a whole class discussion on the language they produced to solve the task. In the given vignette, a student's use of the phrase “in between” led to students making
sense of the meaning of the overlap of the two number lines. The overlap of two shaded sections of a number line allowed students a visual means to derive the solution to the task. The vignette shows how words are referents to mental images (Arnheim, 1969) and how teachers can leverage both images and language to help students learn mathematics (Gonzales, 2004). Moreover, the vignette reveals how language can “carry” a mathematical concept in the sense that language can help students create a mental image of that concept. Leveraging a simple phrase such as “in between” demonstrates how student generated language can support mathematical understanding. This study contributes to the research literature by giving an example of how the use of formative assessment can promote the use of language to “carry” a mathematical concept to help students create a mental image of concepts.

## References

* Arnheim, R. (1969). *Visual thinking*. Berkeley, CA: University of California Press.

* Black, P., & William, D. (1998). Inside the black box: Raising standards through classroom assessment. *Phi Delta Kappan, 80*(2), 139-148.

* Cook, L., & Friend, M. (1995). Co-teaching: Guidelines for creating effective practices. *Focus on Exceptional Children, 28*(3), 1-16.

* Erath, K., Ingram, J., Moschkovich, J., & Prediger, S. (2021). Designing and enacting instruction that enhances language for mathematics learning: A review of the state of development and research. *ZDM – The International Journal on Mathematics Education, 53*, 245–262.
https://doi.org/10.1007/s11858-020-01213-2

* Gearhart, M. & Saxe, G. B. (2004). When teachers know what students know: Integrating mathematics assessment. *Theory Into Practice, 43*(4), 304–313.

* Gonzales, L. (2004). *Building personal knowledge of rational numbers through mental images, concepts, language, facts, and procedures* (Publication No. 3155976) [Doctoral dissertation, New Mexico State University]. ProQuest Dissertations & Theses Global.

* Guskey, T. R., & Bailey, J. M. (2001). *Developing grading and reporting systems for student learning*. Corwin Press.

* Kieren, T. E. (1988). *Personal knowledge of rational numbers: Its intuitive and formal development*. *Number concepts and operations in the middle grades* (Vol. 2). Reston, VA: National Council of Teachers of Mathematics.

* Matute, K. (2022). *Discursive Mathematics Protocol for supporting English learner students to develop the mathematics register: A case study* (Order No. 29321247). Available from Dissertations & Theses @ University of Wyoming. (2718164761).

* Pólya, G. (1986). *How to solve it: A new aspect of mathematical method*. Princeton, NJ: University Press.

* Schoenfeld, A. H. (2013). *Reflections on problem solving theory and practice*. *The Mathematics Enthusiast, 10*, (1–2), 9–34.

* Shepard, L.A. (2000). The role of assessment in a learning culture. *Educational Researcher, 29*(7), 4–14.

* Wiggins, G.P. (1993). *Assessing student performance: Exploring the purpose and limits of testing*. San Francisco, CA: Jossey-Bass.
# Unravelling (Mis)conceptions about Algebraic Letters: Exploring Response Patterns in the 'Meaning of Letters' SMART-test using Latent Class Analysis

Katrin Klingbeil<fnref n="1" /> and Filip Moons<fnref n="2" />

Identifying typical hurdles, common errors and misconceptions in a certain domain is crucial to deepen our understanding of students' learning. In this paper, we explore response patterns of 2051 German grade 7 and 8 students shown in the SMART-test "Meaning of Letters", designed to assess (mis)conceptions regarding variables, more precisely, the letter-as-object misconception. Using Latent Class Analysis, we were able to identify six response patterns. These patterns are described and thoroughly analysed. They urge us to think more deeply about the interplay between (mis)conceptions and contexts and can help build valid assessment tools to diagnose students' current understanding.

Keywords: Online formative assessment, algebra, variables, letter-as-object misconception, latent class analysis.

## Introduction

Assessing students' (mis)conceptions is a challenging task. SMART ("Specific Mathematics Assessments that Reveal Thinking") online tests, that have been developed at the University of Melbourne since 2008, offer a solution by facilitating easy provision and processing of diagnostic tasks on students' conceptual understanding and potential misconceptions. SMART's extended analysis detects patterns between diagnostic tasks, revealing insights into students' understanding and misconceptions. In addition to this automatic diagnosis, it also provides teachers with explanations, tasks, and suggestions for targeted interventions (Steinle et al., 2009).

The test investigated here, *Meaning of Letters*, aims to assess the *letter-as-object* misconception and its subtypes based on students' responses to six multiple-choice tasks. Despite known challenges of multiple-choice tasks, developers argue that well-designed tasks can effectively unveil students' thinking: Klingbeil et al. (2024) showed that students' explanations aligned well with their shown (mis)understandings in their multiple-choice responses.

In a comprehensive intervention study spanning six federal states in Germany, 2051 7th- and 8th-grade students undertook the *Meaning of Letters* test after a few algebra lessons. This paper investigates the response patterns of these students using *Latent Class Analysis* (LCA) (Brandenburger & Schwichow, 2023). LCA is a statistical method used to identify unobservable subgroups (latent classes) within a heterogeneous population based on patterns of responses. This analysis can unravel the (mis)conceptions in understanding algebraic letters and how they interact.

In the following paragraphs, we introduce the theoretical background behind the *Meaning of Letters*-test and the six tasks; next we pose the research question.

<footnote n="1">¹University of Duisburg-Essen, Germany; katrin.klingbeil@uni-due.de</footnote>

<footnote n="2">²Freudenthal Institute, Utrecht University, The Netherlands; f.moons@uu.nl</footnote>
## Struggling to understand algebraic letters: the letter-as-object misconception and its subtypes

Arcavi, Drijvers and Stacey (2017) "distinguish five facets of the concept of variable: a placeholder for a number, an unknown number, a varying quantity, a generalised number, and a parameter" (p. 12). Across these facets, variables stand for or refer to one or more numerical values. Yet, algebra learners often struggle with this numerical interpretation, and various typical errors and misconceptions have been identified. One of them, the *letter-as-object* (LO) misconception, has been described by Küchemann in 1981 as the letter being "regarded as a shorthand for an object or as an object in its own right" (p. 104) and extensively documented over decades (e.g., Akhtar & Steinle, 2017). As part of the foundational Concepts in Secondary Mathematics and Science (CSMS) study on the mathematical understanding of secondary school students in the United Kingdom, Küchemann (1981) utilised the following task: "Blue pencils cost 5 pence each, and red pencils cost 6 pence each. I buy some blue and some red pencils and altogether it costs me 90 pence. If *b* is the number of blue pencils bought and if *r* is the number of red pencils bought, what can you write down about *b* and *r*?" (p. 107).

While only 10% of tested 14-year-old students provided the correct equation $5b + 6r = 90$, 17% gave $b + r = 90$ as an answer, which might have been read as "blue pencils and red pencils together cost 90 pence" (LO). Interpreting $b$ as "the number of blue pencils" is a possibility here, too; however, this would still imply a wrong understanding of equations with a number of pencils on one side of the equation and the price of all pencils on the other. Interestingly, 6% of the students came up with another kind of equation: $6b + 10r = 90$ or $12b + 5r = 90$. These students had figured out a possible solution to the problem first and then used these values as coefficients in their equation. Since the letters are used as abbreviations for the involved objects ("12 blue pencils and 5 red pencils together cost 90 pence"), this is regarded as a special form of LO, which we will refer to as the *solution-as-coefficient* (SAC) misconception in the following. Another special form of LO is called *letter-as-unit* (LU) when the algebraic letter is interpreted as an abbreviation for a unit (Akhtar & Steinle, 2017), e.g., in a task about 8 trucks weighing 24 tonnes, the *t* in the equation $8t = 24$ would be misinterpreted as standing for tonnes (not realising that this would not be a correct equality).

## Research question

The *Meaning of Letters* SMART-test polls the understanding of variables and detects the presence of the letter-as-object misconception, leading to the following research question: *Which response patterns regarding the letter-as-object misconception can be identified among German grade 7 and 8 students based on their responses to the six multiple-choice tasks of the SMART-test Meaning of Letters?*

## Methods and Materials

### SMART-test *Meaning of Letters*

For the diagnosis of students, two parallel versions of the SMART-test *Meaning of Letters* were used with the A or B version randomised by class. Here, we describe only the A version of the German translation of the test (see Figure 1; COR indicating the correct response option). The first task type (Meaning tasks), originating from the work of MacGregor and Stacey (1997), uses only one algebraic letter and asks students to decide on the meaning of this letter in a linear equation in a given context.
While the *Ducks* item uses the initial letter of the involved objects and units, the *Bricks* item uses the letter y. Apart from the correct response (cost/height), MC options include the involved objects (singular and plural; LO) as well as the corresponding unit (LU).

<table><caption>Figure 1: Tasks of Meaning of Letters test (German A version) translated back into English</caption><thead><tr><th>Meaning tasks</th><th>Additive tasks</th><th>Proportional tasks</th></tr></thead><tbody><tr><td><img src="ducks_icon.png" alt="Ducks icon"><br>Ducks<br>Lucy bought 6 ducks for $12. She wrote the equation 6d = 12. What does d in Lucy's equation stand for?<br>☐ the cost of one duck (COR)<br>☐ one duck (LO)<br>☐ ducks (LO)<br>☐ dollars (LU)</td><td><img src="garden_icon.png" alt="Garden icon"><br>Garden<br>Payam bought r red rose bushes and l lilac lavender bushes. The roses cost €4 each. The lavenders cost €5 each. Which equation says that the total cost was €70?<br>☐ 4r + 5l = 70 (COR)<br>☐ r + l = 70 (LO)<br>☐ 10r + 6l = 70 (SAC)</td><td><img src="biros_icon.png" alt="Biros icon"><br>Biros are sold in packs of 3. Sam bought p packs and got b biros altogether.<br>Choose the correct equation:<br>☐ 3p = b (COR)<br>☐ p = 3b (LO)<br>☐ p = 3 (LO)<br>☐ b + p = 4 (LO)<br>☐ 30b = 10p (SAC)</td></tr><tr><td><img src="bricks_icon.png" alt="Bricks icon"><br>Bricks<br>Tina stacked 9 identically sized bricks on top of each other making a 99 mm high tower. She wrote the equation 9y = 99. What does y in Tina's equation stand for?<br>☐ the height of one brick (COR)<br>☐ one brick (LO)<br>☐ the bricks in the tower (LO)<br>☐ millimetres (LU)</td><td><img src="wheels_icon.png" alt="Wheels icon"><br>At a bike shop there are b bikes (2 wheels) and t trikes (3 wheels). Which equation says that there is a total of 100 wheels in the shop?<br>☐ 2b + 3t = 100 (COR)<br>☐ b + t = 100 (LO)<br>☐ 35b + 10t = 100 (SAC)</td><td><img src="racetrack_icon.png" alt="Racetrack icon"><br>A car takes 12 minutes to drive round this racetrack. A driver drives r times around the racetrack in m minutes. Choose the correct equation:<br>☐ 12r = m (COR)<br>☐ 12m = r (LO)<br>☐ r = 12 (LO)<br>☐ 5r = 60m (SAC)</td></tr></tbody></table>

The second type of task (Additive tasks) is based on Küchemann (1981). It uses two algebraic letters (corresponding to the initial letters of involved objects), which are additively connected and restricted by the given situation. Students are supposed to choose the correct linear equation (in standard form) for the described context. In the correct equation, the letters represent the number of objects and the coefficients for the price per object (*Garden*) and the number of components per object (*Wheels*), respectively. The first alternative response option simply adds the variables without any coefficients, making it possible to interpret the letters as abbreviations for the involved objects (e.g., “Bikes and trikes have 100 wheels altogether.”; LO). In the equation of the other alternative option, coefficients equal a possible solution to the problem (that has not been posed) so that the equation can be read as some solution sentence (e.g., “35 bikes (with 2 wheels each) plus 10 trikes (with 3 wheels each) have 100 wheels altogether.”; SAC). Also, in this case, the letters are read as abbreviations for the involved objects.

The third task type (Proportional tasks) is derived from the famous “Students and Professors” problem (Clement et al., 1981):

*“There are six times as many students as professors at this university.” Write an equation using S for the number of students and P for the number of professors.*

This is often answered with $6S = P$ instead of $6P = S$. The proportional tasks in the SMART-test have the same algebraic structure: the two variables are directly proportional to each other. Students are again asked to choose the equation matching the given situation. In the correct equation, the letters (matching the initial letters of involved objects/units) stand for the number of objects for both involved objects (*Biros*) or for the number of racetrack rounds and the number of minutes (*Racetrack*). In both items, the coefficient is the proportionality constant (number of biros per pack
or number of minutes per round). The first alternative response option is the reverse of the correct equation, which allows for a LO interpretation (e.g., “A pack contains 3 biros.” or “1 round equals 12 minutes”; LO). For the *Racetrack* task, the first alternative can also be seen as an LU interpretation (e.g., “12 minutes equals one round”). However, since it is unclear how exactly students interpret the letter here, we opted for the more general LO interpretation. The LO interpretation also applies to the second alternative response although the second variable is missing (e.g., as “A pack has 3.”; LO). In the equation of the third alternative option, the coefficients correspond to a possible solution (to the question that has not been asked), which can be interpreted as a kind of solution sentence (e.g., “Sam bought 10 packs and has 30 biros now.”), indicating the SAC misconception. The *Biros* task offers one more response option that features the addition of the two variables without coefficients. Again, the letters can be interpreted as abbreviations (e.g., as “One biro plus a pack of biros is 4 altogether.”; LO). This response type does not make sense for the *Racetrack* task since no different objects but rounds and minutes would be added.

## Participants

In total, 2051 grade 7 and 8 students (aged 12–14) from six federal states of Germany (78% attending grammar schools, 22% attending non-grammar schools) completed the SMART test. These students were taught by 103 mathematics teachers, leading to a nested data structure (data are analysed at the student level, but the students are clustered in classes).

Teachers were asked to administer the SMART online test 1–2 weeks into their teaching sequence about variables, algebraic expressions and/or equations. Thus, students in grade 7 should have been familiar with a basic concept of variables and be able to use and manipulate them in easy algebraic expressions before taking the test. In grade 8, students probably have started focussing on (solving) equations.

## Data analysis

The response patterns of the test were analysed using Latent Class Analysis (LCA) (Brandenburger & Schwichow, 2023). LCA is a form of structural equation modelling useful for identifying patterns/groups within categorical responses. These patterns/groups are called *latent classes*. Intuitively, one can think of the 2051 participating students as ‘latent classes’. Of course, a description of 2051 ‘latent classes’ will be hard to interpret. LCA considerably reduces the complexity of the data by grouping students with similar patterns of responses in one class, bringing down the 2051 ‘latent classes’ to a comprehensible, clearly distinguishable number of latent classes.

We used SAS Enterprise Guide 8.3 with the PROC LCA for the LCA analysis, considering the nested data structure. As the students were not required to answer all questions, 49 students of the 2051 (2.3%) left some tasks unanswered. Hot-deck imputation was used to impute these missing values.

## Results

### Overall response rates to the SMART-test *Meaning of Letters*

The overall response rates of the participants are shown in Figure 2. Note the low number of correct answers as well as that LU answers are only possible in task 1 and 2, while SAC answers are only possible in tasks 3 to 6. The LO misconception was omnipresent in the responses to the meaning and
proportional tasks; while SAC was present in most responses to the additive tasks.

<figure><img src="image_51.png" /><figcaption>Figure 2: Overall response rates to the *Meaning of Letters* test</figcaption></figure>

## Model selection and model fit of LCA

We iteratively built several models with different numbers of classes to choose the appropriate number of classes for our LCA model. Information criteria (AIC/BIC) and the possibility of giving meaningful labels to the classes were used to decide the number of classes. The 6-class model had the best information criteria, the most interpretable latent classes, and no particularly infrequent class. Table 1 presents the given labels to each class and the prevalence (indicating overall class membership probability) of each class. There is no order or hierarchy of the classes.

<table><caption>Table 1: Mean classification probability; hit rate in the diagonal (bold)</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Classes with labels</th><th rowspan="2">Prevalence</th><th colspan="6">Mean classification probability</th></tr><tr><th>Class 1</th><th>Class 2</th><th>Class 3</th><th>Class 4</th><th>Class 5</th><th>Class 6</th></tr></thead><tbody><tr><td>Class 1</td><td>LO predominant</td><td>23%</td><td><strong>85%</strong></td><td>2%</td><td>5%</td><td>1%</td><td>6%</td><td>1%</td></tr><tr><td>Class 2</td><td>LO with SAC</td><td>11%</td><td>4%</td><td><strong>91%</strong></td><td>1%</td><td>3%</td><td>2%</td><td>0%</td></tr><tr><td>Class 3</td><td>LO with SAC only in additive tasks</td><td>38%</td><td>6%</td><td>1%</td><td><strong>91%</strong></td><td>2%</td><td>1%</td><td>0%</td></tr><tr><td>Class 4</td><td>Correct meaning tasks with LO/SAC elsewhere</td><td>8%</td><td>4%</td><td>3%</td><td>9%</td><td><strong>77%</strong></td><td>3%</td><td>3%</td></tr><tr><td>Class 5</td><td>LO apart from additive tasks</td><td>15%</td><td>8%</td><td>0%</td><td>15%</td><td>2%</td><td><strong>70%</strong></td><td>4%</td></tr><tr><td>Class 6</td><td>Mostly correct</td><td>5%</td><td>3%</td><td>1%</td><td>1%</td><td>6%</td><td>8%</td><td><strong>81%</strong></td></tr></tbody></table>

LCA allows the calculation of the likelihood that a student belongs to each class by analysing their test responses. A robust LCA model, characterised by high homogeneity among latent classes and clear class separation, yields for most students a class where the probability of classification is high for that best-fitting class and low for the others. By calculating the mean classification probability for all students aggregated along their best-fitting class, we get a grip on the homogeneity and class separation in our 6-class LCA model. These mean classification probabilities are shown in Table 1: for example, a student with class 2 as best has a probability of 91% to be in this class (we call this the ‘hit rate’, shown in the diagonal of the table) and only a probability of 4% to be in the first.

## Description of the latent classes

In Figure 3, the item-response probabilities for each task are shown for the six classes. In the following paragraphs, we give a description of each of the classes.

Class 1 (prevalence: 23%) is characterised by LO responses being most likely in all tasks. Even when the initial letter of the involved object is not used (*Bricks*), LO is more likely than a correct answer. The subtype SAC is also possible in additive tasks, but less likely than LO, indicating a relatively
consistent interpretation of letters as abbreviations for objects. We label this class "LO predominant"

<figure><img src="image_52.png" /><figcaption>Figure 3: The six latent classes with their item-response probabilities for every task</figcaption></figure>

Class 2 (prevalence: 11%) is characterised by high probabilities for SAC responses. However, for *Biroš* a LO response is more likely than a SAC response. In the meaning tasks that do not offer a SAC option, LO is most likely; a correct response in *Bricks* is possible. Since this is the class with the highest probability for SAC in proportional tasks, students in this class seem to be quite convinced that coefficients stand for (possible) solutions also in different equation types. We label this class "LO with SAC".

Class 3 (prevalence: 38%) is characterised by very high probabilities for SAC responses in additive tasks and high probabilities for LO responses in all other tasks. This indicates a rather consistent interpretation of letters as abbreviations for involved objects in combination with an interpretation of coefficients as solutions in additive equations. We label this class "LO with SAC only in additive tasks".

Class 4 (prevalence: 8%) is characterised by high probabilities for correct responses in meaning tasks, high probabilities for SAC in additive tasks, and high probabilities for LO in proportional tasks. Students in this class seem to be able to identify the correct meaning of an algebraic letter when directly being asked for it. However, when choosing equations they still fall into the trap of interpreting letters as abbreviations (and coefficients as solutions in additive equations). We label this class "Correct meaning tasks with LO/SAC elsewhere".

Class 5 (prevalence: 15%) is characterised by medium to high probabilities for LO in all tasks other than additive tasks. While in *Garden* the correct response is most likely, in *Wheels* SAC is slightly
more likely than the correct response. For this response pattern, it is impossible to identify one reason (see Discussion). We label this class "LO *apart from additive tasks*".

Class 6 (prevalence: 5%) is characterised by correct responses being most likely in almost all tasks. Only in *Wheels*, SAC is double as likely as the correct response. In *Ducks* and *Racetrack*, LO is also possible but half as likely as the correct response. This indicates at least a partial understanding that algebraic letters do not stand for abbreviations. We call this class "*Mostly correct*".

## Discussion and Outlook

Utilising LCA, six distinct response pattern classes were identified, offering detailed insights into the relationship between students' comprehension, misconceptions, and test tasks. These classes play a crucial role in enhancing our understanding of how students interpret algebraic letters across various contexts. It is important to note that a comprehensive understanding of the implications is an ongoing research process, and this discussion marks our initial attempt at exploring these insights.

Starting with classes that exhibit at least some correct answers, a notable discovery is that Class 6, characterised by mostly correct answers, has a low prevalence of 5% and still shows many SAC responses to the *Wheels* task. The absence of a class labelled ‘All answers correct’ is not surprising, as only 21 students (1%) would belong to this class, which contradicts the principle of a good LCA model that avoids very rare classes. Class 4 (8%) is intriguing, displaying high probabilities for correct meaning tasks, but struggles when translating this understanding into equations. These students seem to possess a superficial knowledge of variable meanings, adequate for direct inquiries about meaning with one variable but insufficient when dealing with equations involving two variables. This underscores the importance of recognising that merely asking about the meaning of letters in simple contexts does not necessarily imply a deep and accurate understanding. Class 5 (15%) is characterised by a very high probability of a correct answer on the *Garden* task and LO/SAC in most other tasks. Since these students do not seem to grasp the meaning of letters, it is likely that these correct responses are not a result of (partial) understanding but of a strategy of combining given letters and numbers according to the described situation without proper understanding.

Regarding the LO misconception and its subtypes, it is crucial to highlight that the LU misconception had a minimal occurrence in the meaning tasks. Some students consistently show LO (Class 1, 23%); however, even in this class, the subtype SAC has a probability of 33% in *Wheels*. This might indicate that this task especially fosters students' urge to come up with a numerical solution. In general, the subtype SAC is often clearly present in additive tasks only (especially Classes 3 and 4). Such additive equations can probably be read more intuitively as a solution sentence such as “35 bikes plus 10 trikes have 100 wheels altogether” compared to proportional tasks that would have to be read as something like “Sam bought 30 biros in 10 packs”. It is also possible that the additive tasks more easily trigger some students' desire to give a solution than proportional tasks. In this respect, Class 2 is exceptional: the probability for SAC is very high in *Racetrack*, but only 37% in *Biros*, while they are both proportional tasks (with similar response rates, see Figure 2). This might indicate that a SAC interpretation in proportional equations is more likely when the letters involved refer to non-physical objects (e.g., rounds in *Racetrack*) or can be interpreted as units (e.g., minutes in *Racetrack*).
The analysis of the identified classes underlines how important the task type, complexity, and context – including the realness of involved entities and the underlying structure of the equation – seems to be for correctly interpreting algebraic letters. These are aspects that need to be taken into account for teaching as well as assessment. For example, problems that focus on a numerical interpretation (like “Think of a number”) or require operations on both sides of the equation might help to support a correct interpretation of the equal sign as well as of algebraic letters. Two further questions remain for future research: How do students transition from these classes after a lesson series about algebraic letters? And: How can this analysis improve the SMART-test *Meaning of Letters* diagnosis?

## Acknowledgements

This research was supported by a Travel Award from the Interdisciplinary Centre for Educational Research (IZfB) of the University of Duisburg-Essen granted to Katrin Klingbeil.

## References

Akhtar, Z., & Steinle, V. (2017). The prevalence of the ‘letter as object’ misconception in junior secondary students. In A. Downton, S. Livy & J. Hall (Eds.), *Proceedings of the 40th annual conference of the Mathematics Education Research Group of Australasia* (pp. 77–84). MERGA.

Arcavi, A., Drijvers, P., & Stacey, K. (2017). *The learning and teaching of algebra: Ideas, insights, and activities*. Routledge.

Brandenburger, M., & Schwichow, M. (2023). Utilizing Latent Class Analysis (LCA) to Analyze Response Patterns in Categorical Data. In X. Liu, & W. J. Boone (Eds.), *Advances in Applications of Rasch Measurement in Science Education. Contemporary Trends and Issues in Science Education*, vol 57. (pp. 123–154). Springer, Cham. https://doi.org/10.1007/978-3-031-28776-3_6

Clement, J., Lockhead, J., & Monk, G. (1981). Translation difficulties in learning mathematics. *American Mathematical Monthly*, 88(4), 286–290.

Klingbeil, K., Rösken, F., Barzel, B., Schacht, F., Stacey, K., Steinle, V., & Thurm, D. (2024). Validity of multiple-choice digital formative assessment for assessing students’ (mis)conceptions: Evidence from a mixed-methods study in algebra. *ZDM – Mathematics Education*. https://doi.org/10.1007/s11858-024-01556-0

Küchemann, D. (1981). Algebra. In K. M. Hart, M. L. Brown, D. E. Küchemann, D. Kerslake, G. Ruddock, & M. McCartney (Eds.), *Children's understanding of mathematics: 11-16* (pp. 102–119). John Murray.

MacGregor, M., & Stacey, K. (1997). Students’ understanding of algebraic notation: 11-15. *Educational Studies in Mathematics*, 33(1), 1–19.

Steinle, V., Gvozdenko, E., Price, B., Stacey, K., & Pierce, R. (2009). Investigating students’ numerical misconceptions in algebra. In R. Hunter, B. Bicknell & T. Burgess (Eds.), *Proceedings of the 32nd annual conference of the Mathematics Education Research Group of Australasia* (Vol. 2, pp. 491–498). MERGA.
# The potential of AI: generating answers for multiple-choice questions using ChatGPT

Laura Kuusemets¹

¹University of Tartu, Faculty of Science and Technology, Tartu, Estonia; laura.kuusemets.1@ut.ee

Computer-based multiple-choice tests reduce teachers' workloads and enable students to receive immediate feedback on their performance, which is why they are widely used in the education system. To create a high-quality multiple-choice test, the author has to develop plausible distractors to be included among the provided answers. Therefore, creating a high-quality test is time-consuming. The purpose of this paper is to determine the suitability of ChatGPT for generating answer options for arithmetic and textual multiple-choice questions using Estonian and English prompts as examples, based on typical mistakes made by students and their similarities and differences with the answer options created by a human expert. This paper uses ChatGPT3.5 to test if it is possible to generate answers for multiple-choice questions that are based on given parameters. The results show that it is possible to generate answer options for multiple-choice questions using ChatGPT.

Keywords: Multiple-choice questions, artificial intelligence, mathematics education, distractors.

## Introduction

The capacity of computers to execute cognitive tasks commonly associated with human intelligence, especially in learning and problem-solving, is defined as artificial intelligence (AI). AI-based systems are used to support teachers, reduce their workload, and automate assessment (Baker & Smith, 2019). In addition, AI-based assessment systems make the assessment process easier and faster for teachers (Kersting et al., 2014). Chat Generative Pre-Trained Transformer (ChatGPT) is an AI technology that generates conversational interactions based on user prompts (OpenAI et al., 2023). Large language models (LLM), such as ChatGPT have been pre-trained on huge volumes of textual data and are therefore able to answer questions with high accuracy, generate text and perform other language-related tasks (Kasneci et al., 2023). The ChatGPT model has proven its potential in various domains, including education (Liu et al., 2023).

Automated assessment and multiple-choice questions (MCQs) tools have been used for a long time. Because a trained language model can answer questions, it can be used to generate multiple-choice answers for tests. However, experience shows that the generation of a high-quality MCQ depends on the quality of the prompts (Kıyak, 2023). While it should not be overlooked, a study conducted by Liu et al., (2023) showed that ChatGPT has rather limited mathematical ability.

In the case of MCQs, the content of the question should be as concise, clear, precise, and unambiguous as possible, (Kelly, 1916); it should refer to the substance of the problem and the statement followed by the question to be answered. One answer option is always correct – it is called 'the key' – and one or more answer options are always false, known as 'decoy responses' or 'distractors' (Gierl et al., 2017; McNichols et al., 2023). The answer options must not be partially true/false (Kelly, 1916). Distractors must be plausible and linked to the mistakes made by students and they should be misleading for students but not entirely false, which would make them easy to eliminate (Gierl et al.,
2017; McNichols et al., 2023). If many distractors need be generated, this becomes burdensome for the test writer (Gierl et al., 2017).

## Method

The main objective in using computer-based environments and MCQs, is to reduce teachers' workload through automated testing. Automated tests with multiple-choice answers are not widespread in the Estonian education practice, as preparing tasks and multiple-choice answers is time-consuming. Due to the aging teaching staff in Estonia, mathematics teachers are reluctant to use computer-based programs because they lack of digital competencies, and their command of English is relatively weak. If ChatGPT could generate answer options for MCQs in Estonian, the teacher would only have to enter the questions and answers for the test. As a result, teachers' workload would be significantly reduced, and teachers would have more time to support the students who need additional tutoring.

The purpose of this paper is to determine the suitability of ChatGPT for generating answer options for arithmetic and textual MCQs using Estonian and English prompts as examples, based on typical mistakes made by students and their similarities and differences with the answer options created by a human expert. In this study, both arithmetical and textual tasks are discussed, both arithmetic skills and knowledge of rules and theorems are checked in Estonian school mathematics. The topics and questions for the study have been taken from the textbook "Testid koolimatemaatikas I" ("Tests in school mathematics I") by Lea Lepmann, a mathematics didactician at the University of Tartu. Following the textbook, we compare arithmetic and textual answer options generated by ChatGPT with those created by a human expert, a mathematics didactician, in terms of their accuracy and the quality of distractors and keys. When creating the prompt for ChatGPT, the recommendations for creating multiple-choice answers were taken into account, based on typical mistakes made by students. The ChatGPT prompt was entered in two languages – Estonian and English. To answer the research question, the answer options generated by ChatGPT and their correctness and the comparability of the false answers were analysed and compared with the answer options created by the mathematics didactician. ChatGPT3.5 has been used to carry out the study, and three research questions have been formulated:

1) How appropriate are ChatGPT-generated answer options, based on students' typical mistakes, for multiple-choice questions in an arithmetic task compared to human expert-generated answer options?

In order to answer the research question, four different types of arithmetic tasks were chosen for one subtopic of Lepmann's (1991) textbook – addition fractions with same denominators, subtraction of fraction with mixed numbers, addition and subtraction of fractions with different denominator.

2) How appropriate are ChatGPT-generated answer options, based on students' typical mistakes, for multiple-choice questions in a textual task compared to human expert-generated answer options?

In order to answer the research question, two theory-based textual tasks were chosen from Lepmann's (1991) textbook – natural and prime numbers. The aim was to check whether ChatGPT understood the prompt and whether the output was formulated using correct mathematical vocabulary.
### 3) What are the differences in answer options between the English and Estonian prompts in ChatGPT?

ChatGPT prompt can also be in Estonian and can be used for understanding and constructing textual tasks in the local language, Estonian. To evaluate if ChatGPT can provide didactically correct answer options in the Estonian language in arithmetic and textual tasks, research questions 1 and 2 must be analysed.

## Results

The results of each prompt are presented as a table, where each answer option is accompanied by its type: key (k), distractor (d), or incorrect option (i).

### 1) How appropriate are ChatGPT-generated answer options, based on students' typical mistakes, for multiple-choice questions in an arithmetic task compared to human expert-generated answer options?

Prompts 1.1-1.4: Prepare four answer options for the actions ______, where one answer option is correct, and three are incorrect. When preparing incorrect answer options, rely on typical mistakes made by students.

Prompt 1.1 – After receiving prompts in English and Estonian, ChatGPT prepared four answer options, one correct and three false (Table 1). There was a correct answer, in both instance and the generated distractor responses were plausible. The prompt given in Estonian, resulted in a non-reduced answer ($\frac{8}{34}$). For some of the answer options created by Lepmann, a partial calculation step has been added. There is also a non-reduced fraction ($\frac{8}{34}$).

<table><caption>Table 2. Four answers proposed by ChatGPT for action $\frac{5}{17} + \frac{3}{17}$</caption><thead><tr><th rowspan="2">Input</th><th colspan="2">Output</th><th rowspan="2">Lepmann, 1991</th></tr><tr><th>In Estonian</th><th>In English</th></tr></thead><tbody><tr><td rowspan="4">$\frac{5}{17} + \frac{3}{17}$</td><td>$\frac{8}{17}$ (k)</td><td>$\frac{8}{17}$ (k)</td><td>$\frac{8}{17 \cdot 17} = \frac{8}{289}$ (d)</td></tr><tr><td>$\frac{15}{34}$ (d)</td><td>$\frac{15}{17}$ (d)</td><td>$\frac{5 \cdot 17 + 3 \cdot 17}{17} = \frac{136}{17}$ (d)</td></tr><tr><td>$\frac{1}{10}$ (d)</td><td>$\frac{2}{17}$ (d)</td><td>$\frac{8}{17}$ (k)</td></tr><tr><td>$\frac{8}{34}$ (d)</td><td>$\frac{35}{17}$ (d)</td><td>$\frac{8}{34}$ (d)</td></tr></tbody></table>

Prompt 1.2 – After receiving a prompt in Estonian, ChatGPT generated four answer options, one correct and three false (Table 2), i.e., they match the given parameters. Two of the generated distractor answers were negative, and one option is in a non-reduced form. ChatGPT generated four answer options for the English prompt, all of which were false. One incorrect answer option was marked as the key. The wrong answer options were all in a non-reduced form. If the same prompt was given a second time, ChatGPT created new answer options and indicated the correct answer, but it was in a non-reduced form ($1\frac{2}{6}$). One option was also in a non-reduced form ($2\frac{2}{6}$), two other answer options
were not simplified correctly and were not reduced ($1\frac{8}{6}$ and $2\frac{8}{6}$). The answer options created by Lepmann show a partial calculation step. One of the answer options is in a non-reduced form ($2\frac{4}{6}$).

<table><caption>Table 3. Four answers proposed by ChatGPT for action $4\frac{1}{6} - 2\frac{5}{6}$</caption><thead><tr><th rowspan="2">Input</th><th colspan="2">Output</th><th rowspan="2">Lepmann, 1991</th></tr><tr><th>In Estonian</th><th>In English</th></tr></thead><tbody><tr><td rowspan="4">$4\frac{1}{6} - 2\frac{5}{6}$</td><td>$1\frac{1}{3}$(k)</td><td>$1\frac{2}{3}$(k)</td><td>$(4 - 2) + (\frac{1}{6} - \frac{5}{6}) = 2 - \frac{4}{6} = 1\frac{1}{3}$(k)$</td></tr><tr><td>-$1\frac{1}{3}$(i)</td><td>$1\frac{4}{6}$(d)</td><td>$$(4 - 2) + (\frac{5}{6} - \frac{1}{6}) = 2 + \frac{4}{6} = 2\frac{2}{3}$(d)$$</td></tr><tr><td>-$\frac{4}{3}$(i)</td><td>$1\frac{4}{36}$(d)</td><td>$(4 - 2) - (\frac{1}{6} - \frac{5}{6}) = 2 - \frac{1-5}{6} = 2\frac{4}{6}$(d)$</td></tr><tr><td>$3\frac{2}{3}$(d)</td><td>$3\frac{3}{12}$(d)</td><td>$(4 - 2) - (\frac{1}{6} + \frac{5}{6}) = 2 - \frac{6}{6} = 1$(d)$</td></tr></tbody></table>

Prompt 1.3 – In response to a prompt in Estonian, ChatGPT generated three answer options, one correct and three false, i.e., matching the given parameters (Table 3). If the prompt was given in English, ChatGPT did not produce any answer options that matched the parameters. A wrong answer was marked as the key, and there was no correct answer option. When the same prompt was given a second time, the results were partially different; again, a wrong answer was marked as the key, and a correct answer option was missing. In both cases, there was no correct answer among the outputs. If ChatGPT was prompted to solve an action, it returned the correct answer with solution steps. Only then, it generated answer options corresponding to the parameters, and the correct answer option was in a reduced form. Three of the four answer options created by Lepmann show a partial calculation step. Only one out of the four answer options was in a non-reduced form ($\frac{7}{21}$).

<table><caption>Table 4. Four answers proposed by ChatGPT for action $\frac{2}{3} + \frac{5}{7}$</caption><thead><tr><th rowspan="2">Input</th><th colspan="3">Output</th><th rowspan="2">Lepmann, 1991</th></tr><tr><th>In Estonian</th><th>In English</th><th></th></tr></thead><tbody><tr><td rowspan="6">$\frac{2}{3} + \frac{5}{7}$</td><td>$\frac{29}{21}$(k)</td><td>$\frac{31}{21}$(d)</td><td>$\frac{31}{21}$(k)</td><td>$\frac{27+3-5}{3+7} = \frac{29}{10}$(d)$</td></tr><tr><td>$\frac{7}{10}$(d)</td><td>$\frac{1}{7}$(d)</td><td>$\frac{7}{10}$(d)</td><td>$\frac{27+3-5}{21} = \frac{29}{21}$(k)$</td></tr><tr><td>$\frac{10}{21}$(d)</td><td>$\frac{10}{21}$(k)</td><td>$\frac{15}{21}$(d)</td><td>$\frac{27+5}{21} = \frac{7}{21}$(d)$</td></tr><tr><td>$\frac{34}{21}$(d)</td><td>$\frac{7}{10}$(d)</td><td>$\frac{2}{10}$(d)</td><td>$\frac{7}{10}$(d)</td></tr></tbody></table>

Prompt 1.4 – Having received a prompt in Estonian, ChatGPT suggested four answer options, one correct and three false (Table 4). The output matched the parameters. The distractors are all in a reduced form. When providing a prompt in English, the output was incorrect the first time. The output was similar to the output given in Estonian and to the answer options produced by Lepmann. However, a wrong answer option was marked as correct and it was in a non-reduced form. In response to a prompt requiring ChatGPT to solve the given operation first and then generate the answer options, it produced the correct answer option in a reduced form. Two of the generated false answers were
equal – one in a reduced and the other in a non-reduced form ($\frac{2}{6}$ and $\frac{1}{3}$). Another distractor was also in a non-reduce form ($\frac{3}{12}$). All of the answer options created by Lepmann include calculation steps.

<table><caption>Table 5. Four answers proposed by ChatGPT for action $\frac{2}{3} - \frac{1}{6}$</caption><thead><tr><th rowspan="2">Input</th><th colspan="3">Output</th><th rowspan="2">Lepmann, 1991</th></tr><tr><th>In Estonian</th><th colspan="2">In English</th></tr></thead><tbody><tr><td rowspan="4">$\frac{2}{3} - \frac{1}{6}$</td><td>$\frac{1}{2}$(k)</td><td>$\frac{1}{6}$(d)</td><td>$\frac{1}{2}$(k)</td><td>$\frac{2-1}{6} = \frac{1}{6}$(d)</td></tr><tr><td>$\frac{1}{3}$(d)</td><td>$\frac{1}{3}$(k)</td><td>$\frac{2}{6}$(d)</td><td>$\frac{4-1}{3} = \frac{3}{3} = 1$(d)</td></tr><tr><td>$\frac{1}{6}$(d)</td><td>$\frac{3}{6}$(d)</td><td>$\frac{3}{12}$(d)</td><td>$\frac{4-1}{6} = \frac{3}{6} = \frac{1}{2}$(k)</td></tr><tr><td>$\frac{3}{4}$(d)</td><td>$\frac{2}{9}$(d)</td><td>$\frac{1}{3}$(d)</td><td>$\frac{12-3}{6} = \frac{9}{6} = \frac{3}{2}$(d)</td></tr></tbody></table>

## 2) How appropriate are ChatGPT-generated answer options, based on students' typical mistakes, for multiple-choice questions in a textual task compared to human expert-generated answer options?

Prompts 2.1-2.4: Prepare four answer options for the assertion "______", where one is correct and three are incorrect. When preparing incorrect answer options, rely on typical mistakes made by students.

Prompt 2.1 – After receiving a prompt in Estonian, ChatGPT produced an output partially corresponding to the parameters (Table 5). The marked correct answer is partially true because the set of natural numbers is closed under addition and multiplication but not under subtraction and division. For the English prompt, ChatGPT gave an output that partially matches the parameters, indicating addition as the correct answer while noting subtraction and division as incorrect.

<table><caption>Table 6. Options proposed for the assertion "The set of natural numbers is closed ..."</caption><thead><tr><th rowspan="2">Input</th><th colspan="2">Output</th><th rowspan="2">Lepmann, 1991</th></tr><tr><th>In Estonian</th><th>In English</th></tr></thead><tbody><tr><td>"Naturaalarvude hulk on kinnine ..."<br/>„The set of natural numbers is closed ..."</td><td><ul><li>Liitmise ja lahutamise suhtes (Under addition and subtraction); (k)</li><li>Jagamise suhtes (Under division); (d)</li><li>Murdude suhtes (Under fractions); (d)</li><li>Korrutamise suhtes (Under multiplication). (d)</li></ul></td><td><ul><li>Under addition; (k)</li><li>Under subtraction; (d)</li><li>Under multiplication; (d)</li><li>Under division. (d)</li></ul></td><td><ul><li>Ainult liitmise suhtes (Only under addition); (d)</li><li>Ainult liitmise, lahutamise ja korrutamise suhtes (Only under addition, subtraction); (d)</li><li>Ainult liitmise ja korrutamise suhtes (Only under addition, multiplication); (k)</li><li>Kõigi nelja aritmeetilise tehte suhtes (Under all four arithmetical operation). (d)</li></ul></td></tr></tbody></table>

Prompt 2.2 – In response to a prompt in Estonian, ChatGPT generated an output corresponding to the parameters (Table 6). For English prompts, however, ChatGPT solved the task completely differently by generating true/false statements. Each answer option explains why the statement is true/false and the reasoning is relevant.

<caption>Table 7. Options proposed for the assertion "Of the following numbers, 1, 2, 3, 4, 5, 6, 7, 8, 9 are prime numbers"</caption>
<table><thead><tr><th rowspan="2">Input</th><th colspan="2">Output</th><th rowspan="2">Lepmann, 1991</th></tr><tr><th>In Estonian</th><th>In English</th></tr></thead><tbody><tr><td>"Järgmistest arvudest 1, 2, 3, 4, 5, 6, 7, 8, 9 on algarvud"<br>"Of the following numbers 1, 2, 3, 4, 5, 6, 7, 8, 9 are prime numbers"</td><td><ul><li>2, 3, 5 ja 7; (k)</li><li>1, 2, 3, 5, ja 7; (d)</li><li>2, 4, 6, ja 8; (d)</li><li>2, 3, 4, ja 7. (d)</li></ul></td><td><ul><li>False. 1, 4, 6, 8 and 9 are not prime numbers (k)</li><li>True. All the numbers are prime; (d)</li><li>False. Only 2, 3, 5, 7 are prime numbers; (d)</li><li>True. Any single-digit number is a prime number. (d)</li></ul></td><td><ul><li>Ainult 1, 3, 5, 7 (only 1, 3, 5, 7); (d)</li><li>Ainult 2, 3, 5, 7 (only 2, 3, 5, 7); (k)</li><li>Ainult 1, 3, 5, 7, 9 (only 1, 3, 5, 7, 9); (d)</li><li>Kõik nimetatud arvud (All given numbers). (d)</li></ul></td></tr></tbody></table>

## Discussion

The study results showed that when constructing multiple answers to given operations, the wording of the parameters in the prompt requires careful consideration. ChatGPT generated multiple-choice answers based on the parameters.

From the answer options generated by ChatGPT it is evident that they still require human verification. The generated multiple-choice answers included many mistakes, such as marking a wrong answer option as the correct one, especially with the English prompt. With the Estonian prompt, ChatGPT generated the correct answer option and three false options for all the given tasks on the first run. Most of the false responses were also plausible, which means that ChatGPT can generate suitable answer options in response to an Estonian prompt. Comparing the answer options generated by ChatGPT with the options generated by the mathematic didactician Lea Lepmann, there are some similarities for some tasks but no complete overlaps. For some tasks, no lure responses were the same for the Estonian prompt, the English prompt, and the response options of the human expert (Table 2). However, for some tasks, there were response options that occurred in all three (Table 3, 4). With an English prompt, ChatGPT marked an incorrect answer option as the key 75% of the time on the first run. Most of the false responses were also plausible, which means that in the case of an English prompt, the answer options generated by ChatGPT may not be suitable and need to be checked by teacher. ChatGPT provides suitable answer options if it first solves the action. An important difference is that the answer options generated by the human expert are structurally different – they include calculation steps that represent the mistake made in the calculation of fraction. None of the answer options generated by ChatGPT included calculation steps.

Care must be taken to ensure that the answer options and the task statement are consistent with each other and the curriculum. In the topic of common fractions, the problem statement should indicate the expected form of the answer (reduced or non-reduced). Estonian school mathematics requires the answer to be given in a reduced form. In this case, this should also be described in the stem. In Estonian school mathematics, fractions are taught in the 6th grade, while calculation with negative numbers is taught in the 7th grade. Therefore, two answer options can be eliminated from the Estonian output for Prompt 1.2. If an answer option contains an answer in a form that the students have not yet learned, they are likely to treat this option as incorrect. According to Girel et al., (2017) an answer option must not be completely wrong because in this case, students will be able to eliminate the answer immediately.
In Prompt 2.1, one can see that ChatGPT makes mathematical mistakes and seems that ChatGPT is aware of the mistakes but still marks the wrong answer as the key. The testing conducted in this study showed that ChatGPT can understand the content of the text and generate a textual response in both Estonian and English. The answers are plausible in Estonian but require linguistic editing. When comparing the answer options in Estonian and English and the one prepared by the didactician, they are similar, or only one of the answer options is different. The most significant difference was for Prompt 2.2, where ChatGPT produced true/false statements for the English prompt, which is undesirable for MCQs because it intentionally complicates the test.

Nevertheless, there were errors or omissions in compiling the multiple-choice answers. For several of the statements, ChatGPT gave a wrong or incomplete answer. For both English and Estonian prompts, there were errors in the mathematical terms or rules in the output. When generating answer options for arithmetical tasks, ChatGPT performed better with Estonian prompts than with English prompts. There were many errors in the outputs in response to English prompts, and the correct answer option often needed to be corrected. Thus, ChatGPT is good at generating multiple-choice answers based on Estonian prompts. Furthermore, it was found that ChatGPT can understand Estonian prompts and produce verbal responses. Based on the results, Estonian teachers can use ChatGPT to create multiple-choice answers, but it does not necessarily result in efficiency gain for teachers, as all the multiple-choice answers still have to be checked by the teacher.

## Conclusion and future work

In this work, ChatGPT was used to see if it could make it easier and faster for teachers to create multiple-choice answers. The most important findings are that, first, the variability of the distractors generated by ChatGPT is smaller than the variability of the distractors generated by mathematical didactician. Second, the keys generated by ChatGPT are incorrect in some cases incorrect. In the case of Estonian prompts, ChatGPT failed to give a correct answer only in a textual task where it marked a partially false statement as the key. With English prompts for arithmetic tasks, ChatGPT marked the correct answer option as the key only 25% of the time in the first attempt. In the case of textual tasks, it indicated the wrong answer option as the correct one in some of the outputs. The third and most important result for Estonian mathematic teachers was that ChatGPT generates more evenly matched responses for Estonian prompts than for English prompts. In some cases, the distractors generated in Estonian are more plausible than those generated in English, because there were more answer-options that were non-reduced fractions.

The present work's limitations lie in using an older version of ChatGPT, 3.5, when the newer version 4.0 is more advanced, has newer data, and is better at processing textual information. In this work, version 3.5 is used because it is available free of charge to everyone, while 4.0 is a paid version. Even though ChatGPT 4.0 is not free of charge, would be advisable to use the newer version.

In future work, ChatGPT should be given more details on the desired learning outcomes for a given topic are on students' common misconceptions in that topic. It should then be investigated whether ChatGPT can generate more appropriate distractor responses based on the given parameters and, for each false response, generate feedback that supports the student based on the type of mistake.
## References

Baker, T., & Smith, L. (2019). *Educ-AI-tion Rebooted? Exploring the future of artificial intelligence in schools and colleges*. https://media.nesta.org.uk/documents/Future_of_AI_and_education_v5_WEB.pdf

Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, Analyzing, and Using Distractors for Multiple-Choice Tests in Education: A Comprehensive Review. *Review of Educational Research*, 87(6), 1082–1116. https://doi.org/10.3102/0034654317726529

Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., ... Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. *Learning and Individual Differences*, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274

Kelly, F. J. (1916). The Kansas Silent Reading Tests. *Journal of Educational Psychology*, 7(2), 63–80. https://doi.org/10.1037/h0073542

Kersting, N. B., Sherin, B. L., & Stigler, J. W. (2014). Automated Scoring of Teachers’ Open-Ended Responses to Video Prompts: Bringing the Classroom-Video-Analysis Assessment to Scale. *Educational and Psychological Measurement*, 74(6), 950–974. https://doi.org/10.1177/0013164414521634

Kıyak, Y. S. (2023). A ChatGPT Prompt for Writing Case-Based Multiple-Choice Questions. *Revista Española de Educación Médica*, 4(3). https://doi.org/10.6018/edumed.587451

Lepmann, L. (1991). *Testid koolimatemaatikast I*. Eesti õppekirjanduse keskus.

Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., Wu, Z., Zhao, L., Zhu, D., Li, X., Qiang, N., Shen, D., Liu, T., & Ge, B. (2023). Summary of ChatGPT-Related research and perspective towards the future of large language models. *Meta-Radiology*, 1(2), 100017. https://doi.org/10.1016/j.metrad.2023.100017

McNichols, H., Feng, W., Lee, J., Scarlatos, A., Smith, D., Woodhead, S., & Lan, A. (2023). *Exploring Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning* (arXiv:2308.03234). arXiv. http://arxiv.org/abs/2308.03234

OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., ... Zoph, B. (2023). *GPT-4 Technical Report*. https://doi.org/10.48550/ARXIV.2303.08774
# Secondary mathematics teachers' experiences of using ChatGPT to design probability and statistics assessment items

Minsung Kwon<fnref n="1" /> and Inah Ko<fnref n="2" />

The purpose of this study was to explore secondary mathematics teachers' experiences of using ChatGPT to design probability and statistics assessment items. For this purpose, we analyzed 22 secondary mathematics teachers' conversations with ChatGPT and their survey responses in terms of their overall experiences with ChatGPT, their intentions to use ChatGPT-generated assessment items, and affordances and challenges of using ChatGPT to design assessment items. The results showed that most teachers did not specify the purpose of assessment and only one teacher identified mathematical errors in the ChatGPT's responses. The teachers employed a wide range of follow-up questions in responding to the ChatGPT's suggestions. The survey results showed that their intentions to use ChatGPT were polarized. The teachers perceived that ChatGPT provided affordances such as creativity and efficiency but shared their concerns about mathematical errors, inaccuracy, ethical issues, and its security.

Keywords: Summative assessment, probability and statistics, Artificial Intelligence (AI), ChatGPT, secondary mathematics teachers

## Introduction

Since Shulman's seminal work on the pedagogical content knowledge (PCK) as a special amalgam of content and pedagogy needed for teaching (Shulman, 1986), the scholarship in teacher education has made a major shift from identifying teacher characteristics toward conceptualizing PCK, developing instruments to measure PCK, and designing teacher education programs to develop PCK. Besides the efforts made in the subject-matter specific PCK, researchers have identified the knowledge needed for teaching with technology in response to the emergence of new digital technology and the importance of technology competence. This specialized knowledge that teachers need to teach with technology is conceptualized as Technological, Pedagogical, and Content Knowledge (TPACK) and it is further specified into sub-domains (Misha & Koehler, 2006). After the publication of the *Second Handbook of Technological Pedagogical Content Knowledge (TPACK) for Educators* in 2016, we have experienced a rapid change and social demands of incorporating technology in teaching and learning over the past few years. Especially, the emergence of generative Artificial Intelligence (AI) and the public release of ChatGPT on November 30, 2022 have attracted more attention from educators, both for the benefits of using AI and concerns about using AI in teaching and learning.

Given these rapid changes using technology, we aimed to explore the experiences of secondary mathematics teachers to design probability and statistics assessment items using ChatGPT in Korea. Because of the nature of grading on a curve and its importance for the college admission in Korea, many secondary mathematics teachers have experienced challenges in writing assessment items that can result in a desired item difficulty and item discrimination but can be differentiated from commercially available workbooks. The advancement of ChatGPT made us to wonder how secondary

<footnote n="1">¹California State University Northridge, USA; minsung.kwon@csun.edu</footnote>

<footnote n="2">²University of Michigan, USA; inahko@umich.edu</footnote>
mathematics teachers might use ChatGPT to design assessment items, what experiences they have with ChatGPT, and whether ChatGPT might resolve their persistent challenges or create new issues to write assessment items. More specifically, this paper examines the following research questions:

1. What prompts do secondary mathematics teachers use to design probability and statistics assessment items in ChatGPT?

2. What probability and statistics assessment items do ChatGPT generate? To what extent do teachers have intentions to use these ChatGPT-generated assessment items?

3. What affordances and challenges do secondary mathematics teachers perceive in using ChatGPT to design probability and statistics assessment items?

## Methods

Using a convenience sampling method (Pattern, 1990), we collected the data from 22 secondary mathematics teachers who enrolled in a three-credit graduate course in mathematics education in Korea. After the instructor's short introduction of ChatGPT, the teachers were instructed to have conversations with ChatGPT 3.5 approximately for 10-15 minutes to design probability and statistics assessment items for Grade 11. We asked teachers to design an assessment item for probability and statistics, a topic that tends to require less use of advanced mathematical expressions or graphical representations. After conversations with ChatGPT, teachers were asked to complete a short survey including a URL link for their ChatGPT conversations, one final ChatGPT-generated probability and statistics assessment item, their overall experiences with ChatGPT, and whether ChatGPT understood their questions, intentions, and feedback. Additionally, we asked whether they had any mathematical or pedagogical issues with ChatGPT, the benefits and challenges of using ChatGPT for assessment, and whether they had intentions to use ChatGPT-generated assessment items using a five-point Likert scale (1: strongly disagree and 5: strongly agree). Lastly, the teachers were asked to explain their perceived item difficulty generated by ChatGPT and to evaluate its appropriateness for their students.

First, we analyzed the prompts that teachers used in ChatGPT to design probability and statistics assessment items. We asked the teachers to share a URL link for their ChatGPT conversations. At the time of analyzing the data, three links were invalid which resulted in 19 links for the analysis. In analyzing the teachers' ChatGPT conversations, it became clear that the strategy and skill of providing effective prompts are crucial for obtaining desirable responses. This is because ChatGPT generates responses based on the user's inputs or prompts. Some reports suggest that ChatGPT has often provided unvalidated and incorrect information (Einarsson et al., 2023). Considering that ChatGPT's mathematical skills are not as strong as in other areas (Frieder et al., 2023), it is important for the users, in this case secondary mathematics teachers, to employ the strategy of using effective prompts to get the desirable outcomes and assess the validity, accuracy, and credibility of ChatGPT's responses. In this paper, we categorized the teachers' prompts into four categories: 1) Specificity of the prompts to design an assessment item; 2) Identification of errors in ChatGPT's responses; 3) Quality of follow-up questions; 4) Evaluation of ChatGPT's responses. Table 1 illustrates our coding rubric to analyze teachers' prompts to ChatGPT.
<table><caption>Table 1. Coding rubric for teachers' prompts to ChatGPT in designing assessment items</caption><thead><tr><td>Category</td><td>Description</td><td>Scoring Rubric</td></tr></thead><tbody><tr><td>1.Specificity of prompts</td><td>Does a teacher's prompt identify...<ul><li>a topic?</li><li>a difficulty level?</li><li>a target grade-level?</li><li>a purpose?</li></ul></td><td><ul><li>1: Address one of the sub-categories</li><li>2: Address two of the sub-categories</li><li>3: Address three of the sub-categories</li><li>4: Address four of the sub-categories</li></ul></td></tr><tr><td>2.Identifying an error in ChatGPT's responses</td><td>Does a teacher identify...<ul><li>a contextual error?</li><li>a mathematical error?</li></ul></td><td><ul><li>0: neither identified contextual nor mathematical error</li><li>1: identified either contextual or mathematical error</li><li>2: identified both contextual and mathematical error</li></ul></td></tr><tr><td>3.Quality of follow-up questions</td><td><ul><li>Is a teacher's follow-up question based on interpretation or evaluation of the ChatGPT's responses?</li><li>Does a teacher's follow-up question further specify or challenge the ChatGPT-generated item?</li></ul></td><td><ul><li>0: accept ChatGPT's responses without posing any follow-up questions or asking unrelated questions to ChatGPT's responses</li><li>1: ask a follow-up question to clarify ChatGPT's responses or request simple modification (e.g., different format)</li><li>2: ask a follow-up question to further specify ChatGPT's responses by adding conditions or related concepts</li><li>3: ask a follow-up question to challenge the ChatGPT's responses or to provide a specific feedback to ChatGPT</li></ul></td></tr><tr><td>4.Evaluating ChatGPT's responses</td><td><ul><li>Does a teacher evaluate ChatGPT's responses?</li></ul></td><td><ul><li>0: no</li><li>1: yes</li></ul></td></tr></tbody></table>

To analyze the teachers' intentions to use ChatGPT-generated assessment items, we recoded disagree responses (1:strongly disagree and 2:somewhat disagree) to negative and recoded agree responses (4:somewhat agree and 5:strongly agree) to positive. For the affordances and challenges of using ChatGPT to design assessment items, we repeatedly read the teachers' responses and found themes emerged from their open responses using an inductive coding.

# Results

## RQ1. Teachers' Prompts Used in ChatGPT

The first analysis focused on determining whether the prompts included key details such as a topic, difficulty level, target grade-level, and purpose (Category 1). The analysis revealed that 14 out of 19 teachers (74%) received a score of 2 for this category. This score indicates that they provided two components of their desired assessment items, such as a broad topic and target grade-level, without specifying the purpose for using the assessment item (e.g., "Please provide a problem about probability and statistics for high school students"). Only one of the 19 teachers provided all the four
components, including the goal of an assessment item ("Please rephrase the problem conditions to assess if a student can derive the solution by using a normalization.")

Second, the analysis of identifying a contextual or mathematical error (Category 2) suggested that only one out of 19 teachers explicitly mentioned an error in ChatGPT's responses. For example, the teacher identified a mathematical error in ChatGPT's explanation of the suggested problem. Specifically, the teacher stated, "You mentioned that this problem will help students understand the concept of conditional probability, but your suggested problem is not related to the concept."

Third, an analysis of the quality of follow-up questions (Category 3) indicated that five teachers (26%) either accepted the ChatGPT's suggestions (e.g., "Good. Thank you!") or asked unrelated questions to the ChatGPT's responses (e.g., "Please give me another question"). Additionally, five teachers (26%) asked a simple parallel question (e.g., "Please give me a more difficult question"); five teachers (26%) further specified the ChatGPT's responses (e.g., "Please add a given that uses the concept of conditional probability"); while four teachers (21%) either provided specific revisions (e.g., "Please use a lower value for the number of plays to simplify the calculation") or challenge the ChatGPT's responses (e.g., "Isn't it possible for people with assigned seats to move among them?"). Figure 1 illustrates an example of one teacher's ChatGPT conversation. As shown in Figure 1, the teacher provides specific feedback on ChatGPT-generated assessment item. In response to this feedback, ChatGPT revised the initial assessment item, using a smaller number for matches and different probability values. However, solving the problem would still require a calculator.

<figure><img src="image_53.png" /><figcaption>Figure 1. An example of a teacher's ChatGPT conversation to design a probability item</figcaption></figure>

This teacher also received high scores in other categories. Specifically, the teacher clarified the purpose of an assessment item and evaluated ChatGPT-generated assessment item in terms of its applicability in real-life situations and item difficulty.

Finally, we investigated if there was clear evidence that teachers evaluated responses from ChatGPT. The analysis indicated that seven out of 19 teachers (37%) assessed the correctness and validity of ChatGPT-generated assessment item. For example, one of the prompts stated that "Your problem necessitates overly complex calculations to determine the standard deviation."
## RQ2. ChatGPT-generated assessment items and teachers' intentions to use them

For the second research question, we analyzed ChatGPT-generated assessment items that teachers came up with and their intentions to use these assessment items. As discussed in the previous section, teachers employed a wide range of prompts to design an assessment item which resulted in a wide range of assessment items. Table 1 illustrates four teachers' ChatGPT-generated probability and statistics assessment items with their intentions to use these items for the assessment (1: strongly disagree and 5: strongly agree).

<table><caption>Table 1. Examples of ChatGPT-generated assessment items and teachers' intention to use them</caption><thead><tr><td>Assessment Item</td><td>Intention</td></tr></thead><tbody><tr><td>Teacher A's ChatGPT-generated assessment item:<br/>A class of 10 students sits at a round table with 12 chairs. Two specific students, A and B, must sit next to each other, and two other students, C and D, must sit facing each other. The remaining six students can sit at random. Find the number of cases in which the students are seated at the round table.</td><td>4</td></tr><tr><td>Teacher B's ChatGPT-generated assessment item:<br/>Jimin and Junho are playing a coin flip game. The coin is assumed to be a fair coin, and the probability of getting heads (H) and tails (T) is 1/2 each. The rules of the game are as follows:<br/>1. Jimin and Junho each flip a coin.<br/>2. If it lands on heads (H), they win; if it lands on tails (T), they lose.<br/>3. When both friends start flipping at the same time, Jimin flips first.<br/><br/>Answer the following questions:<br/>1. What is the probability that Jimin wins and what is the probability that Junho wins?<br/>2. What is the probability that Jimin and Junho both win the first two coin tosses?<br/>3. What is the probability that Jimin and Junho flip a total of 5 coins and win exactly 3 of them?</td><td>2</td></tr><tr><td>Teacher C's ChatGPT-generated assessment item:<br/>A soccer team has played 15 games in a league. The team has a 60% chance of winning a game and a 10% chance of drawing a game. The rest of the games are considered losses. Answer the following questions:<br/>1. What is the probability that the soccer team will win exactly 10 out of 15 games?<br/>2. What is the probability that this soccer team will win at least 12 out of 15 games?<br/>3. What is the probability that this soccer team will lose or tie at least 5 out of 15 games?</td><td>4</td></tr><tr><td>Teacher D's ChatGPT-generated assessment item:<br/>You have a store that sells goods and you have five different types of goods. You need to display these items in a row, but the store's shelves are circular, so the first and last items are next to each other, i.e., they are arranged in a circle. Find the number of cases in which the store displays the five products in a circle.</td><td>2</td></tr></tbody></table>

Teacher A had an intention to use the ChatGPT-generated assessment item (rated 4 in a five-point Likert scale) but would like to revise the item by changing the number of chairs from 12 chairs to 10 chairs. However, Teacher A did not further explain the justification for this revision. Teacher B did not have an intention to use the ChatGPT-generated assessment item (rated 2 in a five-point Likert scale) because the ChatGPT-generated assessment item is simple, easy, and different from the expected item difficulty. Teacher B also pointed out that the assessment item has some inaccurate statements. Teacher C had an intention to use the ChatGPT-generated assessment item (rated 4 in a