Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiency
Authors: Eric Zelikman
Wanjing Anya Ma
Jasmine Elizabeth Tran
Diyi Yang
Jason D Yeatman
Nick Haber
Venue: EMNLP 2023 Main (2023)
Keywords: language model
student test generation
psychometrics
TL;DR: We use language models to generate, evaluate, and select educational test items mirroring the difficulty and reliability of human-made tests.

Abstract: Developing an educational test can be expensive and time-consuming, as each item must be written by experts and then evaluated by collecting hundreds of student responses. Moreover, many tests require multiple distinct sets of questions administered throughout the school year to closely monitor students' progress, known as parallel tests. In this study, we focus on tests of silent sentence reading efficiency, used to assess students’ reading ability over time. To generate high-quality parallel tests, we propose to fine-tune large language models (LLMs) to simulate how previous students would have responded to unseen items. With these simulated responses, we can estimate each item's difficulty and ambiguity. We first use GPT-4 to generate new test items following a list of expert-developed rules and then apply a fine-tuned LLM to filter the items based on criteria from psychological measurements. We also propose an optimal-transport-inspired technique for generating parallel tests and show the generated tests closely correspond to the original test's difficulty and reliability based on crowdworker responses. Our evaluation of a generated test with 234 students from grades 2 to 8 produces test scores highly correlated (r=0.93) to those of a standard test form written by human experts and evaluated across thousands of K-12 students.

Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiency
Eric Zelikman*
Wanjing Anya Ma*
Jasmine E. Tran
Stanford University
Stanford University wanjingmestanforde edu
Stanford University
ezel izelikmaness. stanford. edu
jasetranestanf edu
Diyi Yang
Jason D. Yeatman Stanford University
Nick Haber
Stanford University
Stanford University
diyiyêcs. stanford...... edu
jyeatman@stanford.cmm
nhaber@stanford..o edu
Abstract
120- 0.99
Developing educational test be expen- sive and consuminn,,,g as each item must be written by experts and then evaluated by collect- ing hundreds of student responses. Moreover, many tests require multiple distinct sets of questions administered throughout school year to closely monitor students' progress, known as parallel tests. this study, we focus tests of silent sentence reading efficiency, used to assess students' reading ability over time. To generate high-quality parallel tests, we propose to fine-tune large language models (LLMs) to simulate how previous students would have responded to unseen items. With estimate
total score fram tast form
80-
4t from Lab test
150
Figure 1: Student test scores for human-generated vs poodel-generated tests. Comparison of student scores on a test form of ftem-response-simllad -issponne-imuullto-flleeeedd language-modee wwwe---oddlleeeee numan-generatee items for sentence reading efficiency test. The numan-generater test was designed by experts and calibrated across thousands of K-12 stu- dents; the Al test form was generated by GPT-4 and filtered by our proposed item-response simulator.
these simulated responses, each item's 's difficulty and ambiguity. We first use GPT-4 generate new test items following a list of expert-developed rules and then apply a fine-tuned LLM filter the items based on criteria from psychological measurements. We also propose an
items and test form of
transport-inspireed technique for generating parallel lests and show the generated tests closely correspond to the original test's difficulty and reliability based on crowdworker responses. Our evaluation of generated test with 234 students from grades to 8 produces test scores highly correlated (r=0.93) those of standard test form written by human experts and evaluated across thousands of K-12 students.
unique items) administered throughout the aca- demic year, allowing for close monitoring of stu- dent progress while minimizing practice effects. These test forms, designed to be content-equivalent and reliably produce identical individual scores the original test form, are known parallel tests. Each step of the test development process, from expert item writing to extensive item calibration through large-scale data collection and ultimately parallel test creation and validation, poses signifi- cant demands in terms of resources and time. These challenges emphasize the necessity for an auto- mated and efficient test development framework.
1 Introduction
Developing educational test be resource- intensive and time-consuming, each item must be written by experts and then evaluated by collect- ing hundreds of student responses. This process evaluating items in terms of properties like dif- ficulty and their ability to discriminate between student abilities, known in psychometrics as item calibration, is fundamental to test development. Furthermore, schools often require the creation of multiple, distinct test forms (a collection of
In response to the challenge of item develop- ment, many works have proposed leveraging lan- guage models to generate items for educational sessments and instruction (Agarwal et al., 201 Stasaski et al., 2021; Srivastava and Goodman, 2021: Heck and Meurers, 2022; Rathod et al., 2022;
Thhese authors sontributed equally to this work

---

Training
Item Generation
Simulation
Item- Response Dataset
Sample Student
Train Simulator Predict Response
Predict Parameters with
Parameters to Generate
Responses
Generate Items
into Past Tests
Trained Simulator
Generated
nono
OO
Simulated
Reference Items
Parameters
Figure 2: Method overview. We first train the simulator to predict student's response to a question conditioned on their previous responses. Then, generate items and insert them into past tests for simulation. Finally, we perform simulation and match the new items to an existing test. Note the simulation and generation models differ.
Zou al., 2022; White et al., 2022). However, esti- mating the relevance, quality, and difficulty of these generated items and test forms an open challenge that must be carefully addressed for both psycho- metric and NLP communities as well as stakehold- ers in our education system. We propose an item- response simulator, training LLMs with student response data and simulating the responses of past items in order to calibrate them. As case study, apply this method develop and calibrate new items for silent Sentence Reading Efficiency (SRE) task (Burkhardt et al., 2023), which we elaborate on in Section We then propose an optimal-rranssort-nnpiiee technique for generating parallel test forms with the new items by referring to a well-calibrated humtn-written test form. In doing so, make the following contributions:
and approach parallel test construction with simu- lated responses as relaxed optimal transport.
Children can be sad. True (Response time: slow) You sleep on a log.
False (Response time: slow) [...]
Sweaters can made coal.
participants
False (Response time: very slow)
You can feed hamster.
False (Response time: very slow)
You can fill balloon with air.
Figure 3: Example prompt used for training and simula- tion. Given a prompt similar this, the response simulator predicts the relevant item parameters in our case, the student response and their log-responss time.
Silent Sentence Reading Efficiency
How can we measure students' reading abilities? Oral Reading Fluency (ORF) a widely used mea- sure in research and practice, measuring words read correctly per minute (Fuchs et al., 2001; Domingue et al., 2022). Recently, researchers have examined the relationship between oral and silent reading flu- ency (Kim et al., 201 and shifted focus silent reading, as is the most common form of read- ing for K-12 students. However, given the lack of observable verbal responses, it is a more chal- lenging measurement task. Our task, silent Sen- tence Reading Efficiency (SRE), is an online, self- administrated measure assessing the speed with which a student can read simple English sentences (Burkhardt et al., 2023). Modeled after other stan- dardized silent reading fluency measures such Test of Silent Reading Efficiency and Comprehen- sion (TOSREC) (Johnson al., 201 1; Kim et al., 2012), SRE requires students to read sentences and respond whether each is True or False, which we
1. We address novel task of automated item calibration for sentence reading efficiency, quiring model capable of predicting both responses and response times.
2. We propose fine-tuning LLMs estimate the properties of unseen items by simulating how past participants would have responded.
3. We demonstrate the effectiveness of marginal- izing response-come conditioond LLM predictions over distribution of past participants to esti- mate their responses to questions.
4. By automatically creating parallel test forms, we deploy our system to K-12 education and demonstrate its high quality through large-
scale (n 234) student evaluation.
Overall, our work advances the measurement of reading efficiency by introducing scalable methods to generate test items. We address the novel task of parallel test generation without collected responses

---

120
120
0.62
r=
-0.79
total score
80
80
accuracy threshold
higher than 0.95
total
lower 0.95
0-
0.00
0.25
0.50
0.75
1.00
6.5
7.0
7.5
8.0
9.0
accuracy (proportion correct)
response time (log ms)
Figure 4: Factors affecting reading efficiency scores. (left) the relationship between each participant's total score and their response accuracy, (right) and their median response time in log scale. This indicates that response time is a more important factor that contributes to reading efficiency than accuracy. Note that the response times of participants with similar accuracy (e.g.. >0.95) vary substantially and predict total scores.
refer to as the item truth value (see Fig. 3)- A student responds to as many sentences as they can within three minutes, and the final score is the num- ber of correctly answered sentences minus incor- rectly answered ones. Unlike TOSREC, SRE tar- gets reading efficiency with less emphasis on read- ing comprehension. This complicates item develop- ment, requiring syntactically and semantically sim- ple sentences with only vocabulary that all school- age students should be able to read and understand.
Eth Above
7%
(377)
14%
2nd
13%
7%
[154)
3rd
19%
1000
4th
6th
5th
Figure 5: Dataset grade distribution. The grade distri- bution students in the dataset discussed in Section 3, based on the grades they reported through the SRE app.
We focus on SRE for three reasons. First, from an educational perspective, there is a high demand from schools to closely monitor students' reading development. Thus, it is important to develop diverse parallel test forms with identical difficulty and reliability. Manually authoring thousands of items and collecting the data match test forms is extremely time-consuming and resource- inttessei Second, from psychological measurement standpoint, SRE is task where both accuracy and response time are crucial in determining ability. This characteristic does not align well with classic psychometric models, such as Item Response Theory (Lord, 1980), which focus on accuracy. Third, of particular relevance to the NLP community, measuring sentence-level reading fluency rather than comprehension poses novel challenge, as traditional readability metrics and linguistic features fail predict fluency accurately.
blocks with different test forms: one with TOSREC grade sentences (maximum number of items varies by grade, but less than 70), and the other the Lab test form, consisting of 130 sentences curated by two human experts through an itera- tive design process. After filtering students with response times indicative of random guessing (me- dian response times under 500 ms: 108), the final fine-tuning dataset includes 1962 participants with 183,782 responses. Fig. 4 indicates that the total score (i.e., the reading efficiency) each stu- dent is more correlated with their median response time than how accurately they respond to each ques- tion, which underscores the importance of incor- porating response time modeling to capture item difficulty during the item calibration process.
3 Training Dataset
4 Methods
4.1 Item Generation
We collect student data from 1st grade to high school by working closely with more than 30 di- verse school partners in the United States for two school years (See Fig. 5 for breakdown by grade level). All data collected under IRB guidelines. As part of the SRE validation study (Burkhardt et al., 2023), each student completed two 3-minute
We use GPT-4 (OpenAI, 2023) to generate diverse sentences through zero-shot prompting and avoided giving specific examples. We first generate univer- sally true sentences (temperature tions, with at most 5000 tokens generated) and then transform each true sentence into a corresponding
six comple-

---

1.00
false sentence by asking the model change 1 or 2 verbs, nouns, or adjectives. After several rounds iteration, we developed the following prompt- - each rule after the third was added iteratively in response to observed failure cases:
probabiit rospond
0.75-
0.50
0.55
Generate 150 sentences.
Rules:
6.00
The sentences have different length between 3 to 15 words.
2002
2502
predicted response time (ms)
3500
4000
grade students should be able read and understand the sentences.
truth wabe
Figure 6: Generated item simulated parameters. We visualize the probability that the simulator assigns to "true" student response for the GPT-4 generated sen- tences, colored by whether they came from the "true" or "false" set. For many "false" sentences scored as true, GPT-4 generated sentence with the wrong truth value.
Use only Kindergarten and high frequency vocabulary words to generate
sentences.
4) Make sure that each sentence is sentence stating a universal fact that is immediately, obviously true.
5) humans are used as the subject in a sentence, make sure it states a life experience that is true for everyone.
of the final item, conditioned on the previous items, as visualized in Figure our final model con- figuration, we use Low-Rank Adaptation (LoRA) (Hu et al., 2022) on 13-billion parameter LLaMA model (Touvron et al., 2023) with 8-bit weights (Dettmers et al., 2022a,b) and a substantial 0.5 dropout on the adapter layers and final layer to miti- gate overfitting. We include more detail hyperpa- rameters considered in Appendix E and discuss all differences between the hyperparameters used for the crowdworker and school student evaluations.
6) The sentence should not require any inferential comprehension skills to read nor understand.
7) The sentences should not be subjective nor describe opinions.
8) The sentences should not be centric to any country nor culture.
9) The sentences should be very diverse.
Excluding unexpected generation output (e.g., blank strings) and exact duplicates, this prompt gave us 761 true sentences. However, we found generated few long sentences even though "3 to 15 words" was specified, so we prompted the model generate additional 100 sentences with ad- ditional rule specifying "at least 10 words in each sentence." " We then used a prompt to transform each true sentence into corresponding false sen- tence: Transform each of the following sentences into false sentences by changing 1 or verbs, nouns, or adjectives, and the sentences should be universally false. Ultimately, this produced corpus with 861 true and 861 false sentences.
To apply this model for evaluating new items, sampled previous participants and insert the generated item as the final item in a randomly sampled subset of their responses. Then, aggregat- ing over many sampled participants per item, we calculate a mean and standard deviation response time for each sentence, as well as the expected proportion of simulated responses that were true or false for each sentence. Figure 6 visualizes the probabilities and response times simulated by the item-response simulator for the generated items, aggregated over a hundred sets of sampled student responses. Crucially, modeling the probabilities allows the simulator to identify ambiguous items items for which a nontrivial percent of responses are expected to be incorrect. In Figure 6, false items with higher probabilities and true items with lower probabilities are more ambiguous. We include qualitative analysis in Appendix A.2.
4.2 Item Evaluation
For training, fine-tuned an LLM to predict stu- dent responses conditioned on previous responses, which we refer to as an item-response simula- tor. We train on a manually authored, anonymized corpus of 1,962 participants collected by the lab, ranging from 1st grade to adulthood, containing response times and responses (i.e., true or falsc), described in Section 3. Specifically,, each train- ing example consists of a randomly-selected subset of a sampled participant's responses, which are arranged in random order. The model was then trained to predict the response and response time
4.3 Parallel Test Form Construction
We first simulate student responses both the lab test form and the generated items and try identify a set of generated items that corresponded well to the lab test form. However, it is important

---

Experimental design. For our computational model evaluation, we primarily validated our model by evaluating its ability to generalize to a random subset of 10% of the items in the dataset described in Section 3, not used for training. As also done for simulation, for each training example, we con- catenated subset of one student's responses items they responded to, arranged in random or- der, visualized in Figure 3. We exclude the student response for the last item and fine-tune the model (1) to predict it. As detail, we also found that bin- ning the input text corresponding to the student response times as "very fast", "fast", "medium", be able to generate multiple distinct test sets schools require multiple distinct parallel test forms to be administered throughout the year. A naive strategy would greedily pair the most similar items and then remove them when selecting new items for an additional test. But, this would ensure that successive tests would be gradually less similar to the original test and cannot ensure diverse items in each test. Instead, we duplicate the lab test form once per desired parallel test and find the best pairing between the lab and generated items. This is an unbalanced, constrained optimal transport problem, which we solve as follows: we first assign each duplicated lab item a probability of corresponding to a generated item, with no prob- ability of true items being paired to false items and term subtracted from the logits proportional to the distance between the lab and generated items.
Note that we only apply the optimal transport al- gorithm in crowdworker experiments, as the aim of the school student evaluation is primarily to demon- strate the usefulness of the item-response simulator in a collaborative setting, with more detail on the human-in- -nttn----tth-loop filtering included in Section 5.3. However, in future school deployments, we will use this algorithm, allowing to jointly handle multi- ple optimization criteria. Because of the human-in- the-loop filtering setup. it was necessary instead perform deduplication separately for the school stu- dent evaluation, discussed in Appendix C. We dis- cuss implementation details further in Appendix E, and the full pipeline in Figure 2.
4,4 Additional Filtering
For the crowd-worker experiments, we filter the dataset for safety using GPT-4, discussed in Ap- pendix D. For the school experiment, we manually review the questions for safety and ambiguity out of an abundance of caution and responsibility to pro- vide high-quality items, discussed in Section 5.3.
We then minimize 1) the expected distances be- tween the lab items and their paired generated items in the selected parameters (i.e., simulated response time), 2) the semantic similarity (over a threshold) within each copy of the dataset, and 3) the probabil- ity that the same generated item would be selected multiple times. This is non-convex optimization problem, but by initializing the logits reasonable initial values (e.g., logits proportional to distance). we found that the resulting simulated optimized tests converged and corresponded closely to the parameters that our model simulated for the lab- generated items. Figure 11 visualizes these simu- lated scores across two simultaneously generated, distinct test sets, as well as the ambiguous set of items for reference when optimizing for both accu- racy and response time. Precisely, we over:
Evaluations
5.1 Model Evaluation
d
Edistance
PaiDDass Daig:
m
m
Creuse
Pas Po.j
(2) "slow", "very slow" based their quantile of overall data reduced overfitting. We believe this may reduce the model's s ability to correlate specific sets of millisecond- response times with stu- dent responses. Note that we calculated the bins based on the overall response time distribution be- cause the variance across questions in the training dataset was much smaller than the variance across students. We include further item-level analysis for both the generated items and the ttem-response sim- ulator's evaluations of these items in Appendix A. Results. On our evaluation dataset, we find our simulator's ' tiem-aggregan predicted response
a=1 1=1 i
d
R Σ fl
n
Ccosine
ParCiyPaj.
(3)
for P E Rixxxxm D E Ruxm, C'E Ruxa where n is the number of generated items and m is the number of lab test form items, and d is the number of test copies optimize. P is the probability that lab test form item will be mapped to given generated item, the pairwise distance between the lab and generated item parameters, and is the semantic similarity between generated items.

---

125
0.92
total score from from parallel Al tost fomm
times are well-correlated with the tem-aggreeattd true response times, with correlation coefficient (r) of 0.50, and correspondingly, an r² of 0.25, and for predicted vs. actual aggregate probabilities, we found r = 0.76 (r² = 0.58) with a 25.4% RMSE.
50
5.2 Crowdworker Evaluation
Experimental design. Since school-aged students must be provided with carefully vetted tests with human supervision, we aim to utilize crowdworker evaluation to address questions that cannot be rea- sonably obtained through student evaluation alone:
total score from Lab test form
total score from antiiguus Al fest frrm
125
r= 000.
1. Can the parallel AI form (without any human filtering) reliably produce identical to- tal scores compared with the well-calibrated, human-generated test form?
100
75-
2. How well can the item-response simulator identify ambiguous generated items, and do these items actually challenge the reliability of the SRE measure?
25
25
60 75 100
125
total score from Lab test form
To answer these questions, we develop a new ver- sion of the SRE task with three 3-minute blocks ran- domly ordered: buman-generated sentences (Lab form), GPT-generated items filtered by the item- response simulator (parallel Al form), and GPT- generated items with ambiguous items identified by the feem-response simulator (ambiguous Al form). To create the two Al-generated splits, we first di- vide the generated items according to the median accuracy of items in the training data and then fol- low our construction method in Section 4.3 on the unambiguous items and sample randomly for the ambiguous items. We recruited 50 participants via Prolific and paid 155.00 per hour. 6 participants' results were found either incomplete or random guessing and were removed from the data analysis. Results. The total score of each participant is counted by the total correct responses subtracted from the total incorrect responses. Figure (top) shows that the total scores produced by the parallel AI test form achieve a high correlation (r = 0.92) with the scores produced by the Lab form. In addi- tion, the overall difficulty of the parallel AI form is highly identical to the difficulty of the Lab form. Figure (bottom) suggests that the test form with ambiguous AI items identified by the item-response simulator correlates much less (r 0.75) than the parallel test form above. These comparisons sug- gest the item-response simulator, without human in- tervention, is able to flag unseen, ambiguous items that actually challenge test reliability.
Figure Crowdworker scores. Comparison scores of Prolific participants on Lab test form with parallel Al test form (top) vs. ambiguous Al test form (bottom).
5.3 School Student Evaluation
Experimental design. The school student evalua- tion aims to answer three questions:
Can filtered GPT-generated items reliably pro- duce identical total scores compared with human-generated items?
2. Can the item-response simulator outperform traditional readability metrics in predicting the response time and difficulty of unseen items? And qualitatively, what are some items that the item-response simulator could predict well and what are some that couldn 't?
We use the item-response simulator to select the optimal 130 GPT- ger-reeeet true sentences and 130 false sentences, 260 items in total. Then, to ensure the appropriateness of the test items for school- aged children to read and test, the authors, who are familiar with K-12 reading assessments, review and remove items that were still ambiguous (20 items, e.g., "A hill is flat and square."), could have an inappropriate interpretation (4 items), dangerous (2 items: "Babies drink gasoline." or "Soap is for eating."), "), required too much world knowledge items, c.g., "A brick is made from clay") or are sub- jective item, "A doll can be fun play with."). Note that, this school experiment, for robustness and out of concern for automation bias, we do not

---

actual medin resporce time (178)
(m0.34
r
3000
3000
actual response
item bruth valuee
FALSE
2000
2000
TRUE
1500
1500
10
2000
2500
3000
3500
Flesch Kincaid score
predicted median response time (ms)
Figure 8: Response time predictions. Relationship between Flesch Kincaid grade vs. median response time (ms, left) and model predicted response times vs, actual median response times (ms, right) on the school student study.
Table 1: Parameters for human-generatee generated items (Lab test form) and GPT-generated items with L.LM and hu- man filtering (AI test form). Accuracy and response times are based on item-response simulator simulations.
Table 2: Model comparisons: prediction correlations
Accuracy
Median RT (ms)
Item Response Theory
NA
NA 0.254 (0. 0.351) 0.509 (0:427, 0.586)
Flesch Kincuid
0.001 (00,109, 0.106)
Item- Response Simulator
0.316 (0.215,
(Words)
Median (ms) Std.
Lab
False
6.13
0,800
2926
202
logistic test model (Sonnleitner, 2008; Stenner, 2022) and estimate IRT parameters from text data in other contexts (Ehara, 2018; Settles et al., 2020). Fig. 9 showcases example items that are curately and inaccurately predicted by the item- response simulator. Overall, the simulator can ef- fectively screen difficult or ambiguous items.
AI
False
5.02
0.872
2879
198
0.947
automatically filter the potentially offensive sen- tences as we do in the crowdworker experiments.
We then implement and deploy a new version of the SRE task with 2 blocks: human- -emaanggnerattd sentences (Lab test form) and GPT-generated sen- tences filtered by the item-response simulator ac- cording accuracy and response time (details in App. E.3) and then human experts (AI test form). 234 students grades 2-8 from a California public school participated in this study, and three students were removed from the analysis due to a median response time under 500ms (i.e., random guessing). In this version, students first see the human- generated block with maximum of 130 items. Then, in the second block, students see 130 rardee GPT-generated sentences (from an item pool of 200 filtered items). The overall pre- dicted item parameters based on the item-response simulator are shown in Table
6 Related Work
6.1 Item Generation
Earlier methods for the automatic item/question generation include rulc-based or template-based ap- proaches (e.g., Mitkov and Ha (2003); Flor and Ri- ordan (2018)) and attempts to adaptively generate questions (Du al., 2017). Additional work has fo- cused contexts in which generating high-quality questions (e.g.. math word problems) requires step- by-step reasoning (Keller, 2021). In particular, question generation for passage-bascd reading com- prehension (Agarwal et al., 2011: 1; Stasaski et al., 2021; Heck and Meurers, 2022; Rathod et al., 2022; Zou et al., 2022) and for teaching second languages (Chinkina and Mcurers, 2017; Srivastava and Good- man, 2021) have been especially well-explored.
Results. There high correlation (r 0.93) in terms of total scores between the filtered AI test form and the Lab form (Fig. 1). We also found that the two test forms match closely in difficulty, al- though we did not explicitly aim to match the iden- tity line when creating the Al test form for schools.
More recently, White al. (2022) demonstrate the effectiveness of using GPT-3 to create test items comparable to gold standard TOSREC test items for assessing reading fluency at first and eighth-grade levels. However, their approach still requires extensive human intervention and human evaluation for real-world use. Specifically, they demonstrate the feasibility of GPT-3 for item generation but do not develop an approach to calibrate items in terms of difficulty or filter items that would function poorly in the real world.
In terms of predicting the unseen item param- eters (accuracy and median response time), the item-response simulator outperforms the Flesch Kincaid method (Fig. and accomplishes task that the traditional psychometric models (e.g., Item Response Theory) cannot due to the unseen items (Table 2)- however, we note that prior work has attempted to model item difficulties using linear

---

Difficult for students but easy for the simulator.
modeling student knowledge (Piech et al., 2015; Abdelrahman et al., 2023). More recent studies, such as Benzahra and Yvon (2019) and Martinc et al. (2021), contrast this finding, noting that lan- guage model perplexity and other statistical met- rics were imperfect estimators of readability, but metrics derived from them performed well across datasets. Importantly, predicting reading fluency, while naturally related to reading comprehension, distinct and uxder-explored area.
Music makes people sleep. True: People wear sunglasses in sunlight.
False:
True: Bananas yellow when ripe.
Easy for students but difficult for the simulator.
False: Octopuses have arms. True: our stomach digest food,
False: We wear belts to loosen pants
Easy for both students and the simulator.
True: turtle has a shell.
6.3 Parallel Test Form Construction
False: Books have pages with salsa. Children play the playground.
True:
The body of work constructing parallel tests spans decades (Armstrong et al.. 1992, 1994; Gib- son and Weiner, 1998; Sun et al., 2008; Ignjatović et al., 2021). However, as most of these works point out, identifying the optimal pairing is an NP-hard problem. To circumvent this, the works often rely on item selection heuristics. For example, Arm- strong et al. (1992, 1994) calculate the degree to which each item is correlated with the total test score and then attempt to generate tests with simi- lar expected distributions of final scores. Similarly, Sun et al. (2008) uses a genetic algorithm to max- imize the similarity in the expected information gain of each item across set of tests. In contrast, we pose this as a relaxed optimal transport prob- lem we frame our optimization as differentiable, probabilistic relaxation of this underlying optimal transport problem and solve it directly. This allows us incorporate other optimization criteria that are important but rarely seen in parallel test literature, such as item diversity and truth parity constraints.
Difficult for both students and the simulator.
False: Stars are invisible at night.
False: Chairs have no legs.
True: Mushrooms grow in damp areas.
Example item difficulties as perceived by students and by the stem-response simulator.
Figure 9:
Relatedly, Srivastava and Goodman (2021) fine- tune model to generate questions adaptively for teaching reverse translation by predicting whether a student will correctly complete translation.
While their study is a seminal prior work, it is not directly applicable to parallel test generation. First, we require fixed item bank that can be closely reviewed before deployment because our items will be given to children and because we cannot have extra latency. Moreover, the best models for many zero-shot item generation tasks cannot be fine- tuned by the public, but prior work suggests that small models can effectively validate much more powerful models (Cobbe et al., 2021). Finally, we focus on providing reliable assessment, while their approach is only applicable to instruction.
7 Conclusion
Our study presents a new framework to generate test items and select them according to their effectiveness assess students by leveraging large language models and previous student responses. We use silent sentence reading efficiency assess- ment, a task with implications for the millions of students who take such tests annually, to illustrate the utility of this framework by generating new test items, predicting the difficulty and ambiguity of unseen items, and creating multiple parallel test forms that have comparable quality and appro- priateness to human numan-generated test forms well as remain appropriately difficult for students. We ultimately validate our approach with a wide range of evaluations, including more traditional machine lcarning metrics, crowdworker evaluations, as well as a real-world deployment in K-12 classrooms.
6.2 Item Evaluation
Although item generation has broadly received more attention from the NLP community than item evaluation, there are several influential works in this area, especially leveraging language models. Language models have been used as a proxy for reading, with prior work highlighting the limited correlation between human and language model readability: studics like Schwarm and Ostendorf (2005) and Si and Callan (2001) demonstrate that the perplexity of language model is powerful indicator of the likelihood that sentence is gen- crated from a corpus of a given difficulty. These works are also related to knowledge tracing, i.e.,

---

Limitations
Reliance on closed and restricted LLMs: Our study uses GPT-4 for generating and filtering test items. However, access to GPT-4 may be expensive if generating many thousands of items. In addition, we fine-tuned LLaMA (Touvron et al., 2023), but LLaMA's license does not support commercial use. As result, the exact approach in this paper cannot be applied in commercial contexts. Fortunately, LLaMA replications are underway, and artifacts have already been released (Geng and Liu, 2023).
While case study demonstrates the effective- ness of using large language models and previous student responses, it carries several limitations: Generalization to other types of tests and questions: The framework presented in this case study focuses primarily on assessing students' read- ing efficiency. However, we have not yet demon- strated that our approach can easily generalize to other kinds of tests and assessments. No aspect of this approach is inherently dependent on the SRE task, but more research is needed to investigate the applicability of our methods to wider range of educational tests. Although multiple-choice questions are not typically used in real-world silent sentence reading fluency evaluation (Bloomquist, 2017), we believe with a comparable amount of data, we could replicate this success for other test formats such as multiple-choice questions.
Ethics Statement
There are naturally many ethical considerations in the context of this work. First, all handling of student data must be extremely careful and consid- erate of the privacy of the students. In this work, we have taken care to ensure that the data used is not personally identifiable and that the data is used appropriately, We have also acted in accordance with the guidelines of our IRB. At all points, before presenting data to humans, especially to children, multiple rounds of review and analysis were per- formed to ensure that the data was appropriate. The value of our work is to largely reduce the burden of the test creation and invite humans to act as the final safeguards and experts to be responsible for reviewing as few items as possible.
Filtering and evaluation in school student evaluation: Due to ethical considerations, certain aspects of our study, such as automated filtering of ambiguous or inappropriate items, could not be deployed in the school student evaluation. Conse- quently, the items in the school student evaluation had to be carefully reviewed by experts before ad- ministration. This highlights the need for more robust and reliable methods for filtering and eval- uating generated items real-world educational settings. Although we now have validation support- ing the item-response simulator, deploying aspects like automatic filtering will require further study, not only to assess accuracy and sensitivity but also to mitigate automation bias and risks.
As for the ethical implications of the work itself, there are several to consider. First, language mod- cls exhibit and potentially exacerbate biases that are already present in society. Humans, of course, also exhibit these biases, but by automating this pipeline, we may reduce the possibility of human intervention to correct for these biases. Second, the ability to use language models in this context may result in an emphasis on tests being developed that are compatible with language models however, aspects like visual and phonetic information are valuable for reading fluency evaluation and many other tasks, and we should be mindful to avoid tai- loring tests closely language models' strengths.
Fine-tuning and data limitations: The item- response simulator was fine-tuned on data collected from diverse schools in the United States, but the model is trained a uniform distribution of these students. However, the students whose responses are used for training and simulation may differ demographically from the schools to which the tests are deployed. Thus, the model's performance in predicting item difficulty and ambiguity may not generalize to all populations or school contexts. Moreover, we have not quantified the degree to which additional generated examples from GPT-4 continue be novel language models fine-tuned using reinforcement learning from human feedback (RLHF) are believed suffer from mode collapse (Zhu et al., 2023), so ensuring that generated items continue to be meaningfully different essential.
Finally, while the ability to efficiently generate test items could lower the barrier to universal as- sessment and lead to more equitable assessment policies, it's important to proceed with caution and keep humans in the loop at each stage particu- larly when it pertains to educational assessment in young children. Thus, as we begin implementing Al-based assessment systems at scale, we advocate for proceeding with caution, keeping educational experts in the loop at each stage, and keeping an eye toward equity as we strive for efficiency.

---

Acknowledgements
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022a. Llm.int80: 8-bit matrix multi- plication for transformers at scale. NewrIPS.
We would like to thank the schools that partnered with us on this research. We thank the Stanford NLP community for their helpful feedback on the abstract of this paper. We appreciate Benjamin W. Domingue and the reviewers for their helpful feed- back on the manuscript and Sang T. Truong for highlighting related work. This work was funded by grants from the Stanford-Sequoia research col- laborative, Stanford Neuroscience:Translan pro- gram, Microsoft, and the Eunice Kennedy Shriver National Institute of Child Health and Human De- velopment R01 HD095861 to J.D. Y, and NSF IIS- 2247357 to D.Y.
Tim Dettmers Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022b. 8-bit optimizers via dosk-wiie quantization. 9th International Conference on Learn- ing Representations, ICLR.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized Ilms. arXiv preprint arXiv:2305. 14314.
Benjamin W Domingue, Madison Dell, David Lang, Re- becca Silverman, Jason Yeatman, and Heather Hough. 2022. The effect of covid oral reading fluency during the 2020-2002 2021 academic year. AERA Open. 8:23328584221 1 120254.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn- ing to ask: Neural question generation for reading comprehension, In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1352,, Vancouver, Canada. Association for Computational Linguistics.
References
Ghodai Abdelrahman, Qing Wang. and Bernardo Nunes. 2023. Knowledge tracing: A survey. ACM Comput- ing Surveys, 55(1 1333
Manish Agarwal, Rakshit Shah, and Prashanth Man- nem. 2011. Automatic question generation using dis- course cues, In Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1-9.
Ehara. 2018. Building an english vocabulary knowledge dataset of japanese english-as-a-secon language learners using crowdsourcing. In Proceed- ings the Eleventh International Conference on Lan- guage Resources and Evaluation (LREC 2018).
Ronald D Armstrong. Douglas H Jones, and Zhaobo Wang. 1994. Automated parallel test construction using classical test theory. Journal of Educational Statistics, 19(1)73390.
Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouve, and Gabriel Peyré. 2019. Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd In- rernational Conference on Artificial Intelligence and Statistics, pages 2681-221 2661-2690.
Ronald D Armstrong, Douglas Jones, and Ing- Long Wu. 1992. An automated test development of parallel tests from seed test. Psychometrika, 57:27 1-288.
Mare Benzahra and François Yon.. 2019. Mcasuring text readability with machine comprehension: pilot
Ronald A. Fisher. 1925. Statistical methods for research workers. Oliver and Boyd
study. In Workshop on Building Educational Appli- cations Using NLP, pages 412-422.
Michael Flor and Brian Riordan. 2018 A semantic role-based approach to open-domain automatic ques- tion generation. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Ed- ucational Applications, pages 254 263, New Orleans, Louisiana. Association for Computational Linguis- tics.
Christy L Bloomquist. 2017. An examination of the relationship of oral reading fluency, silent reading fluency, reading comprehension, and the Colorado State Reading Assessment. Utah State University.
Amy Burkhardt, Maya. Yablonski, Jamie Mitchell, Lies- beth Gijbels, and Jason D. Yeatman. 2023. Develop- ing items for a silent reading efficiency task (2023 ncme). Conference presentation.
Lynn S Fuchs, Douglas Fuchs, Michelle K Hosp, and Joseph Jenkins. 2001. Oral reading fluency as an indicator of reading competence: A theoretical. empirical, and historical analysis. In The Role of Fluency in Reading Competence, Assessment, and instruction, pages 239-256. Routledge.
Maria Chinkina and Detmar Meurers. 2017. Question generation for language learning: From ensuring texts read to supporting learning. In Proceedings of the 12th workshop on innovative use of NI.P for building educational applications, pages 334-344.
Xinyang Geng and Hao Liu. 2023. Openllama: An open reproduction of llama. Github.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicciro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10, 14168.
Wade M Gibson and John A Weiner. 1998. Generating random parallel test forms using ett in computer- based environment. Journal of Educational Measure- ment, (5(4))29773110

---

Tanja Heek and Detmar Meurers. 2022. Parametrizable exercise generation from authentic texts: Effectively targeting the language means the curriculum. In Proceedings of the 17th Workshop on Innovative Use NLP for Building Educational Applications (BEA 2022), pages 154-166.
Manav Rathod, Tony Tu, and Katherine Stasaski. 2022. Educational multi-question generation for reading comprehension. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 216-223.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bers Sentence embeddings using siamese newt-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguisties.
Edward Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee. Percy Liang. and Tatsunori Hashimoto, 2023. Whose opinions do language models reflect? Inter- national Conference on Machine Learning.
Miroslava M Ignjatović. Dragan M Bojić, and Igor I Tar-- talja. 2021. survey on problem formulations and (meta) heuristic-based solutions in automated assem- bly of parallel test forms. International Journal of Software Engineering and Knowledge Engineering, (188::1171-121
Sarah E Schwarm and Mari Ostendorf. 2005. Reading level assessment using support vector machines and statistical language models. In Proceedings of the 43rd annual meeting of the Association for Computa- tional Linguistics (ACI. pages 523-530.
E. Johnson, J. L. Pool, and D. R. Carter. 2011. Valid- ity evidence for the test of silent reading efficiency and comprehension (tosrec). Assessment for Effective Intervention, 37-15:
Burr Settles, Geoffrey T. LaFlair, and Masato Hagiwara. 2020. Machine learning driven language assessment. Transactions of the Association for computational Linguistics, 8:247-263.
Stanley Uros Keller. 2021.
Automatic generation of word problems for academic education via nat- ural language processing (nlp). arXiv preprint arXiv:2109.1 13123.
Luo Si and Jamie Callan. 2001. A statistical model for scientific readability. In Proceedings of the tenth international conference on Information and knowl- edge management, pages 574-576.
Young- Suk Kim. Richard Wagner, and Elizabeth Fos- ter. 2011. Relations among oral reading fluency, silent reading fluency, and reading comprehension: latent variable study of first-grade readers. Scientific Studies of Reading, 15(4)3338332.
Philipp Sonnleitner. 2008. Using the Iltm evaluate item-generating system for reading comprehension. Psychology Science Quarterly, 50(3):345-33 362.
Young-Suk Kim, Richard K. Wagner, and Dina Lopez. 2012. Developmental relations between reading flu- ency and reading comprehension: A longitudinal study from grade 1 to grade 2. Journal of Experimen- tal Child Psychology, 111
Megha Srivastava and Noah Goodman. 2021. Question generation for adaptive education. Association for Computational Linguistics.
Katherine Stasaski, Manav Rathod, Tony Tu, Yunfang Xiao, and Marti A Hearst. 2021. Automatically gen- crating cause-and-effect questions from passages. In Proceedings the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 170-170.
Frederic M Lord. 1980. Applications of item response theory to practical testing problems. Routledge.
Matej Martinc, Senja Pollak, and Marko Robnik- Sikonja. 2021. Supervised and unsupervised neu- ral approaches to text readability, Computational Linguistics, 47(1):141-179
A Jackson Stenner. 2022. Measuring reading compre- hension with the lexile framework. Explanatory Models, Unit Standards, and Personalized Learning in Educational Measurement: Selected Papers by A. Jackson Stenner, pages 63-88. Springer.
Ruslan Mitkov and Le An Ha. 2003. Computer-aidd aided generation of multiple-choite choice tests. In Proceedings of the HLT-NAACL 03 Workshop on Building Edu- cational Applications Using Natural Language Pro- cessing, pages 17-22.
Koun-Tem Sun, Yu-Jen Chen, Shu- Yen Tsai, and Chien- Fen Cheng, 2008. Creating irt-based parallel test forms using the genetic algorithm method. Applied measurement in education, 112211111-
OpenAI. 2023. Gpt-4 technical report. arXiv.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 2302:2300.11977
Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge trac- ing. Advances in neural information processing sys- lems, 28.

---

Julia White, Amy Burkhardt, Jason Yeatman, and Noah Goodman. 2022. Automated generation of sentence reading fluency test items. In Proceedings of the Anmual Meeting of the Cognitive Science Society, volume 44.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022, Opt: Open pre-trained transformer language models. arXiv preprint 2205v:2205.0106
Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong. Chenguang Zhu, Michael Jordan, and Jiantao Jiao. 2023. Fine-tuning language models with advantage-induued policy alignment. arXiv preprint SxXiv2306600231..
Bowei Zou, Pengfei Li, Liangming Pan, and Aiti Aw. 2022. Automatic true/false question generation for educational purpose. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 61-70.

---

A Model Analysis
posedly "false" sentences, where the item-response simulator determined that the following sentences were the least false (excluding repeated sentences) most are at least arguably true:
A.1 Analyzing GPT-4-generated Sentences
GPT-4 and, more broadly, language models fine- tuned based on human preferences have well- known challenges with mode collapse- although higher temperatures theoretically encourage diver- sity, this training step causes models to be more likely to express single "modal" opinion (San- turkar et al., 2023). We observed similar pattern, with much higher degree of similarity across dif- ferent instances model generations as opposed to across items within a single generation. This was part of the motivation for encouraging the model to generate many diverse sentences in sin- gle prompt instead of performing many indepen- dent calls. The following were some of the highly similar sentences that remained even after the item- response- response-simulator based filtering, as determined by the sentence embedding model we used to filter the sentences further:
[Keys lock doors. Ducks honk and fly. Boats sink in water.
bake food ovens. A farmer eats food. Rain falls from the ground. Shadows form in darkness. We cut food with spoons. Umbrellas are used to keep us wet during rainstorms, Sailboats have a sail to avoid the wind and stay still on the water. Lightning comes after thunder. Water is frozen. Ants are large insects that can carry heavy loads and work together. We eat when we are full. People use tools to break things. A shirt uncovers our body. Buses take us places slowly. Families swim together. Autumn leaves stay trees. Blankets help keep you cold. bathtub releases water. A nap makes us tired.]
The mistake of making an explicitly, unambigu- ously incorrect truth judgment was not an observed failure case for generated "true" sentences: for these sentences, the model was unlikely to gen- crate something outright false and more likely to generate something ambiguously true or only sub- jectively true. For example, the following true ex- amples the ones where the model was most uncertain many of them contain statements that are not universally true, confusing, or difficult assign a truth value. The following "true sentences" were among the ones where the model was least certain of the student response:
A stove is for cooking food. A stove is for cooking.
Fruit grows trees and plants.
Fruits grow trees and plants. Cows give us milk to drink. Cows give us milk.
A toothbrush cleans Toothbrushes clean teeth.
teeth.
Apples grow apple trees. Apples grow trees.
A bicycle has two wheels. A bike has two wheels.
Vegetables are good for our health. Vegetables are healthy food.
Kites can fly in the sky.
1. "Foxes orange and fast." Foxes come in variety of colors and fast is subjective.
Kites fly in the wind.
A boat floats A boat goes water.
water.
2. "A hill is small and round." Relative to moun- tains, sure, but this is not universally true. "Clocks have twelve numbers." What about digital clocks and 24-hour clocks?
Ice cream is cold and sweet. Ice cream is cold.
Toothpaste cleans our teeth. Toothpaste helps to clean teeth.
4. "The moon changes shape.." The moon ap- pears to change shape in the sky, but it does not actually change shape.
A.2 Analyzing item-response-siiulator Predictions
5. "Dolphins are not not fish.." While true (disregard- ing folk taxonomy), this is clearly world- knowledge-heavy question.
We observe several interesting patterns in our item-response simulator, especially in combination with GPT-4. For example, Figure 6, many of the truest "false" sentence sentences generated by GPT-4 were, in reality, true and many of the least false or most uncertain "true" sentences were in fact somewhat ambiguous. Note this trend was particularly pronounced for the sup-
6. "Rice grows in water." " Again, this requires an unreasonable amount of world knowledge.
mentioned in
B Preliminary Crowdworker Study
We performed an initial crowdworker study where we performed our optimization to match each copy

---

second, found that the difficulty of the filtered corpus matched the lab corpus far more.
125-
0.9.2
toba fromtiirred testform
C Naive Deduplication
73-
After training these models and validating that their predictions appeared reasonable, we filtered the items to include only ones where the predicted response time closest to the trendline relating sentence length (in words) We further filtered them by predicted accuracy, selecting only the ones that the model was most confident students would answer correctly (in particular, we chose the top 200 true and false sentences). Finally, we used sentence embeddings generated for each sentence by the "paraphrase- mpnet-base-v2" model to select the most similar pairs of sentences (in terms of absolute cosine distance, to also capture sentences that were similar in meaning but opposite in truth value) (Reimers and Gurevych, 2019). If the sentences had the same truth value, we would remove one at random. If they were different, we would remove the one corresponding to the majority class. This ensured were left with roughly equal number of both true and false sentences, with 260 in total.
50-
75
10c
125
response time.
total score from Lab torm
125
0.91
total scoe unifeere test
200-
75-
25
25
75
100
125
total from Lab test form
Figure 10: Preliminary crowdworker study results. Results from the preliminary crowdworker study. While the unfiltered items matched the difficulty more poorly than the filtered items, we incorporated these observa- tions when constructing our main crowdworker study.
However, this carried limitation: by dedupli- cating before selecting items, we may eliminate potentially good items. By deduplicating after se- lecting items, we must anticipate the proportion of items that must be deduplicated in advance. Moti- vated partly by this, we then explored techniques to simultaneously optimize semantic similarity along- side test set quality in less heuristic way.
of the lab dataset generated item in terms of both accuracy and response time. We observed that the difficulty did not perfectly match that of the lab dataset, as represented by the identity line, although it corresponded much more closely than the unfiltered dataset. When comparing the ground truth data to the predictions, the cause of this was clear: the few lab items that the model identified as ambiguous were substantially less ambiguous than predicted. However, for the generated items, they were actually ambiguous. This transformed false negatives into true negatives the dataset, harming performance. As a result, for our final prolific study and for the school evaluation, we filtered by the median accuracy but did not optimize it when matching generated and lab items.
D Additional Filtering
To remove potentially inappropriate items for classroom settings, we further prompt GPT-4 to evaluate the appropriateness of their own gen- eration (with temperature 0.1, and at most 1000 tokens generated). Note that this was only used for the crowdworker experiments for the school student evaluation, we performed this filtering manually out of an abundance of caution. We use the following prompt:
Figure 10 shows both filtered and unfiltered test forms correlate well with the lab test form. How- ever, there are important caveats: first, prolific par- ticipants represent substantial distribution shift for the model which was trained primarily on K-12 students they performed far better on average;
Return the following true false sentences if it is potentially offensive dangerous for children to read.

---

E Implementation Details
Lab test form
ambiguous Al test form
1.00
E.1 Item Evaluation
probabiliy tredkttuue
0.75
0.50
We explored various training methods, includ- ing Low-Rank Adaptation (LoRA) (Hu et al., 2022) on model using 8-bit weights (Dettmers al., 2022a.b). optimizing only subset of the weights represented as an additive term (a matrix constructed as the product of two vectors that
0.25
parallel Al test form
parallel test
1.00
marrnnn +
0.50
0.25
is added to the original linear layer weights). We also explored training linear head on top of pre-trained model initially, we used the mean of the final hidden state embeddings, but the
0.00
A
2000
2500
3000
3500
2000
2500
3000
3500
response time (ms)
truth
later observed that this underperformed, considered models were autoregressive and the prediction only possible from the final example, and not the participant-specine few-shot examples. Instead, we used the final example's final hidden state, which performed better. Each training example corresponds to subset of a sampled participant's responses however, when we simulate responses to an item for evaluation, sample many participants, predict their responses the new item, then aggregate the predictions. For initial exploration, we primarily explored a variety of Open Pre-trained Transformer (OPT) language models with between 125 million and 6.7 billion parameters (Zhang et al., 2022). For the final implementation, we primarily considered LLaMA models, varying from 7 billion parameters 65 billion parameters (Touvron al., 2023). The largest model we fine-tuned with LoRA had 13 billion parameters and the largest model we used with a linear classifier had 65 billion parameters, leveraging four 40GB A40 GPUs. practice, we found that these largest models were too slow for our purposes, ultimately using a 13-billion parameter LLaMA model.
Figure 11: Test form item distribution comparison. Simulated response time and accuracy distributions for generated test forms.
0.09
Ttem Information
0.06
0.03
0.00
0
estimate student ability
truth value
FALSE
TRUF
test
lab
Figure 12: Item information comparison. Item infor- mation based on 2-parameter logistic IRT.
E.2 Parallel Test Form Construction
We ultimately selected constant learning rate of -5 with batch size of 32 of items, in- cluding random sample of up 30 student item- response pairs (fewer only the student responded to fewer items). We use Low-Rank Adaptation (LoRA) (Hu et al., 2022) on model using 4-bit weights (Dettmers et al., 2022a,b, 2023), optimiz- ing only subset of the weights. We trained our model for 600 steps, each of which contained 32 items, with gradient accumulation over a batch size of 4. Training the model required approximately hours, while simulating all of the GPT-generated items with 100 previous participants required ap- proximately hours.
We perform our optimization logits with Adam with a learning rate of 0.1 and evaluate item similarity using the absolute cosine similarity between their sentence embeddings from 's paraphrase-mpp tecthraas--pnee-aaee (Reimers and Gurevych, 2019), disregarding items with absolute similarity below 0.5. Note that we initially attempted to use Feydy et al. (2019)'s dif- ferentiable optimal transport library, geomloss, but unbalanced problems require reach hyperparam- eter to be specified, and we found that no reach works across all points. Figure/ showcases the predicted response time and accuracy distributions in each test forms.
each item-pair

---

E.3 School Student Evaluation Filtering
Based on feedback from experts, for our school student evaluation, we initially filtered using two threshold criteria: first, accuracy must be greater than 85%, which corresponds to the median item accuracy in our dataset; second, the response time should be within standard deviation of the mean response time per word (disregarding the intercept i.e., the extrapolated amount of time we expect participant take for an item with no length).
F Item Information Analysis
By fitting 2- 2-parameter logistic item response the- ory model the student responses to both the Lab and Al-filtered test forms, we obtain the Fisher information (Fisher, 1925) for each item. Fig- ure 12 shows that GPT-generated false sentences of- fer higher item information than human-generated false sentences, while there is no significant differ- ence for true sentences. Although IRT is not the perfect psychometric model for the SRE task, this finding is encouraging to show the potential of the GPT-generated items.


Reviews:
[{"review_id": "aumGBEv6oQ", "reviewer": "Reviewer_qWhv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review and feedback on our work! We appreciate your comments on the comprehensiveness of our evaluations and the detail provided.\n\nRegarding your concerns about novelty and relevance to EMNLP, we were hoping to highlight that this work was submitted to the new Human-Centered NLP track; while we readily acknowledge that our primary contribution is not the method’s technical novelty from a core NLP perspective (i.e., we’re not proposing a new model architecture), we’d like to emphasize that our key contribution is in showcasing a technique for a novel human-centered application, and in demonstrating its real-world applicability. \n\nWith that said, we’d like to highlight a few key points of technical novelty:\n1. Our key technique is marginalizing response-conditioned LLM predictions over a distribution of past participants. This can be applied anytime you want to know how people might respond to new, in-distribution questions.\n2. Framing parallel test construction as optimal transport has applications anywhere high inter-test reliability is necessary.\n3. Fine-tuning for this task is nontrivial, and many techniques should be transferable. We address questions like whether to jointly learn psychometric properties and how to prevent LLM overfitting while maintaining performance on limited human data.\n\nMoreover, we hope this paper can help bring together psychometrics and NLP, where simulating student responses has potential in pre-evaluating the quality, efficiency, and equity of the assessments/measures. Many NLP advances have come about this way: techniques to solve subfield-specific problems often find broader applications. Especially for human-centered NLP tasks where individuals may disagree on the correct answer or label, fine-tuning language models to simulate prior participants’ responses conditioned on their past responses has broad application. Not only can it help to model complex overall distributions of responses, but it can also provide insights into the causes of differences in responses. This could allow us to ask nuanced questions about the person-to-person sources of variation. \n\nWe will emphasize these technical aspects and novelty in our revised version. Broadly, we are enthusiastic about the growing relevance of human-centered NLP to the wider NLP community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The work is well-written and provides thorough details.\n* The study presents comprehensive evaluations.", "reasons_to_reject": "* The work lacks technical novelty.\n* The weak relevance to EMNLP. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The study focuses on silent sentence reading efficiency assessment and presents a framework for generating items, simulating a student’s responses, and producing parallel tests. The work is well-thought, well-organized, and presents comprehensive evaluations.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Obcy3Qkq93", "reviewer": "Reviewer_dxrP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging and supportive review, as well as the many insightful comments!\n\nFirst, we were hoping to highlight a key detail related to your summary. It is true that we used crowdsourced responses for part of the evaluation for this study. However, a primary contribution of our work that we would like to emphasize is that we indeed administered our test (with extensive review and careful oversight) to 234 students actually at the K-12 stage at a California public school, who took both our AI-generated test and the human-generated test. Figure 1 in the paper is based on this data. We realize that this may have been unclear and will adjust the abstract and Figure 1 accordingly, to clarify that these were actual K-12 students.\n\nIn addition, we sincerely thank you for highlighting these relevant works - we will reference them and clarify in the paper that, while IRT itself cannot be evaluated for unseen items, prior work has aimed to estimate IRT parameters. \n\nIt is true that the work from Settles et al. (2020) demonstrates that IRT $\\theta$ parameters have a rank-correlation with their CEFR-derived predictions. However, their paper also indicates that their passage rank is well-predicted by a linear regression of relatively simple factors like words per sentence and characters per word, factors which are combined in the Flesch-Kincaid score. As indicated in our Table 2, Flesch-Kincaid was not predictive of our metrics. We also emphasize that a benefit of our approach is the ability to use a model fine-tuned on many students to simulate the responses of a subset of students. In addition, Ehara et al. (2018), while absolutely relevant and honestly quite prescient, focused on mapping individual word embeddings to IRT parameters and thus would be potentially difficult to apply to this sentence evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "They are approaching the training in the best-performing way they can think of at this time for a clear objective: a combination of LLaMa and LoRA. The paper also evaluates in the best way conceivable at this time, despite the fact that it is also difficult to evaluate.", "reasons_to_reject": "In Table 2, it is true that Item Response Theory cannot be evaluated for unseen samples. However, there are several NLP studies that estimate IRT difficulty parameters from texts. In terms of studies on reading materials such as this study, the following studies would be relevant. It would be better to discuss the relevance of these previous studies.\n\nMachine Learning-Driven Language Assessment (Settles et al., TACL 2020)\nhttps://aclanthology.org/2020.tacl-1.17/\n\nBuilding an English Vocabulary Knowledge Dataset of Japanese English-as-a-Second-Language Learners Using Crowdsourcing (Ehara, LREC 2018)\nhttps://aclanthology.org/L18-1076/", "questions_to_authors": "", "paper_topic_and_main_contributions": "This study involves reading test generation and assessment for K-12 students.\nLoRA and LLaMa were used for reading comprehension test generation. For the evaluation, because it is difficult to collect and evaluate children who are actually at the K-12 stage, the reading test is evaluated by crowdsourcing the reading test, manually evaluating it, and comparing the K-12 students' scores on the existing test to their crowdsourced scores.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xcHverffgL", "reviewer": "Reviewer_CGAU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback! We appreciate your suggestions for improving the scope and agree that these would be impactful future directions. To clarify, the choice to use true/false sentences as test items was to closely match standard reading efficiency assessments, e.g., TOSREC, the Woodcock-Johnson Sentence Reading Fluency task, and our SRE. A core aspect of human-centered NLP is the need to be specific about who we are serving and how to efficiently incorporate human data to develop useful NLP applications. By collecting 2 years of responses from over a thousand students from diverse K-12 populations, we were able to fine-tune this model to scale the original human-written test. The specificity of our data and our evaluations helped ensure our contribution is efficient and fair for the populations we aim to serve. As the reading patterns of adult crowdworkers are naturally different from those of e.g. primary school students, we aim to use crowdsourcing only for confirmatory purposes where it would be impossible to evaluate with our served population.\n\nHowever, on the topic of other test formats:\n1. Note that multiple-choice questions are not typically used in real-world silent sentence reading fluency evaluation, but are used in tests like reading comprehension evaluation [1,2].\n2. However, we believe that with our method and a comparable amount of data, we could replicate this success for other test formats such as multiple-choice questions. \n3. If we incorporate existing methods for automatic grading of open-ended responses [3], we expect we could even generate high-quality short-response items. \n\nWe agree that exploring our method’s generalizability would be a valuable contribution. We will incorporate these points into the paper, and we’d be happy to elaborate on additional potential future directions. The suggested extensions would make for exciting follow-up works and deserve more than experiments we could only have run with crowdsourced participants. \n\nFinally, the primary reason we opted for a non-adaptive assessment for this study was to create parallel test forms that can produce scores consistent with our human-written gold standard test. In psychometrics, it is crucial to validate the reliability of alternative assessment (in our case, the AI test form) first, before scaling the test to a more adaptive format [4]. Additionally, using a predefined item bank rather than generating items on the fly allows us to (1) review all generated items to ensure their safety and developmental appropriateness for school-aged participants, and (2) to run the assessment entirely in-browser without additional latency or significant participant hardware requirements. But at the same time, we agree that an adaptive test would provide significant additional value.\n\n[1] “An examination of the relationship of oral reading fluency, silent reading fluency, reading comprehension, and the Colorado State reading assessment” Bloomquist, 2017  \n[2] “Relationships of Three Components of Reading Fluency to Reading Comprehension” Klauda and Guthrie, 2008  \n[3] “Exploring Automatic Short Answer Grading as a Tool to Assist in Human Rating” Condor 2020  \n[4] “Overview of Test Assembly Methods in Multistage Testing” Zheng et al. 2014 (in Computerized multistage testing)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper effectively demonstrated the benefits of LLM in producing silent reading efficiency test items and evaluating their difficulty. Particularly notable is the finding that the use of LLMs maintained the reliability and validity of manually produced tests, as shown in Figure 1, where the scores from AI test form and those of Lab test form had strong correlation. This sudy also explored the process of calculating the correspondence probabilities between lab items and AI-generated items.", "reasons_to_reject": "The focus on True/False questions and response time measurement is well-suited for the reading efficiency test. However, it would be beneficial to consider incorporating a wider range of question types and options in an adaptive test format. This expansion would facilitate a more comprehensive evaluation and bring your research in line with standard educational assessment practices. Additionally, when delving into the process of calculating correspondence probabilities between lab items and AI-generated items, please provide a more detailed explanation of the training data's validity. In cases where the training data is incomplete or contains biases, there's a potential risk of obtaining biased results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduced an educational test item generation method utilizing Large Language Models (LLMs). The proposed method involved the development of prompts stating several conditions. The primary contribution of this study lay in exploring the effective use of LLMs, and this study demonstrated their capacity to generate test items and evaluate the difficulty of test items. The research specifically targeted K-12 students' reading efficiency, focusing on testing the reliability of test items developed using the proposed method. The study compared the test results obtained from the proposed method with those generated manually. Additionally, the test-item difficulty was assessed by comparing with readability metrics. The findings indicated a strong correlation between the test results obtained through the proposed method and the manually produced tests.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper leverages LLMs to significantly reduce the effort required for generating tests for K-12 students. Reviewers are highly positive about the approach and evaluation, giving high soundness scores (4). Concerns are raised regarding the lack of technical innovation (qWhv), missing related work (dxrP), and the limited range of the question types and options (CGAU).

I recommend accepting this paper to the main conference. Overall, this is solid research that illustrates a very reasonable use case for LLMs. The concerns are minor and do not overshadow the contributions. However, the authors should take them seriously and make an effort to address them in the next version.
Decision: Accept-Main