## 3 Materials and Methods
### Validation Measurements and Metrics  
Item response theory was used with parameters calculated using iterative maximum likelihood estimation method to provide a benchmark against which the performance of algorithms is measured. Benchmark values represent idealized IRT results if a full lengthIRT test was applied to calibration data. MLE method maximizes likelihood function so that observed data is most probable. It begins with some a priori value for a student's knowledge on each topic and item difficulty Elo ratings. Then student's knowledge ratings are recalculated to best reflect observed quiz outcome. Afterward item difficulty ratings are recalculated in light of newly estimated student's topic knowledge. This process was iteratively repeated until the mean change in Elo ratings fell below one point. For synthetic students and quizzes, for whom seed Elo rating values used for a generation were known, the mean error between MLE and seed Elo rating was calculated and is presented in Figure 3. The number of topics within the quiz was set to 15. The quiz length in the graph on the right was set to 80 items. The number of students in the graph on the right is 120. These numbers were chosen to resemble the parameters of the experiment with real student data. However, it is important to note that, synthetic student generation parameters may be significantly different from latent trait distribution in real students. Assessment of accuracy for LTT based experiments was measured against IRT MLE values. An argument could be made that the item difficulty should be personalized for each student, rather than relying on statistical data as in the case of IRT. However, this would require a more sophisticated psychometric theory than IRT and other approaches such as learning styles are heavily criticized [56; 57]. As for IRT, latent values were not used because they are unknown for real students and without reference to real values synthetic scores can be manipulated: if the deviation in topic knowledge is small, separating best known topics from worst will be harder. The set of suitable items according to IRT for a student was established by selecting topics least familiar to a student (lowest \(sElo_{t}\) values) using complete knowledge of quiz answers, then easiest items (lowest \(iElo\) values) from those topics were added to the set. Delta values for the selection of suitable items were chosen so that an average of 15 items were chosen for a student in synthetic and real experiments. This number was chosen for two reasons. Items in real quizzes mostly covered a narrow band of material. Selecting a few of them would mean students would quickly amend the gaps in their knowledge and would need to either retake formative assessment or study without guidance. Thus, wasting time. In this setting, a hypothetical average student with equal knowledge on every topic would have 1 item in need of studying from each topic of the course. Additionally, the number of suitable items depends on the proficiency of a student. If the number is chosen too low formative assessment will fail to provide a sufficient amount of material to the most knowledgeable students. Once the set of suitable items for each student was known assessment performance was measured using precision, or positive predictive value (_PPV_). _PPV_ describes the probability of providing suitable learning material to a student based on data gathered by formative assessment. _PPV_ is calculated, \(PPV=TP/(TP+FP)\). Where \(TP\) is true positive, the number of items in the set of suitable items according to a formative assessment  
Figure 3: Mean error of MLE calculated values as compared to seed values used in the generation of synthetic students. On the left mean error is a function of group size. On the right mean error as a function of quiz length. algorithm at some length of a quiz. TP is an intersection of the set of MLE selected suitable items and algorithm selected items.