Adaptive and Personalized Exercise Generation for Online Language Learning
Year: 2023

Abstract:
Adaptive learning aims to provide customized educational activities (e.g., exercises) to address individual learning needs. However, manual construction and delivery of such activities is a laborious process. Thus, in this paper, we study a novel task of adaptive and personalized exercise generation for online language learning. To this end, we combine a knowledge tracing model that estimates each student’s evolving knowledge states from their learning history and a controlled text generation model that generates exercise sentences based on the student’s current estimated knowledge state and instructor requirements of desired properties (e.g., domain knowledge and difficulty). We train and evaluate our model on real-world learner interaction data from Duolingo and demonstrate that LMs guided by student states can generate superior exercises. Then, we discuss the potential use of our model in educational applications using various simulations. These simulations show that our model can adapt to students’ individual abilities and can facilitate their learning efficiency by personalizing learning sequences.1

heading: 1 Introduction
text: Adaptive learning technologies which continuously monitor student progress to dynamically adjust the level or type of learning materials based on the individual’s abilities are quite popular (Becker et al., 2018). Empirical studies have shown various benefits of adaptive learning, such as improved student learning outcomes (Bailey et al., 2018; Holthaus et al., 2019), lower dropout rates (Daines et al., 2016), and increased instructor satisfaction (Yarnall et al., 2016). Despite their effectiveness, designing adaptive systems is challenging as it usually involves planning a series of exercises that is personalized and adaptive to each student, which requires
1Our implementation is available at https://github.com/ nlpcui/AdaptiveQG.
diverse exercise planning as well as an understanding of the student learning process.
On the other hand, powered by advances in neural NLP, works have been done for automatically generating text-based exercises or questions for educational purposes in second language learning (Heck and Meurers, 2022; Perez and Cuadros, 2017), mathematics (Polozov et al., 2015; Zhou and Huang, 2019; Wang et al., 2021), and computer science (Susanti et al., 2017). Nevertheless, how to apply these approaches in adaptive systems remains an open question. First, existing methods largely rely on pre-defined question templates or specified information sources (e.g., a passage), thereby resulting in limited knowledge coverage and low question difficulty control, and as a consequence, do not meet each student’s individual and nuanced learning needs. Besides, they are usually designed to generate standalone exercises, whereas adaptive learning systems usually require a continuous supply of exercises. Another related line of research studies exercise recommendation to customize learning content based on individual ca-
10184
pabilities and goals (Wu et al., 2020; Huang et al., 2022). However, these systems are limited by the diversity of the exercise pool.
To address the above limitations, we study the task of exercise generation in the context of adaptive learning, where we hypothesize that a student’s dynamic knowledge state holds the key to generating adaptive and personalized exercises. Specifically, we ground our study in the domain of language learning to create exercise sentences for translation, of which Figure 1 illustrates the overall process. We start with an assumption about the dynamics between exercise difficulty, vocabulary, and a student’s knowledge state (§ 3). Then, we propose an approach (§ 4) that marries knowledge tracing (KT; Corbett and Anderson (1994)), a technique for estimating students’ mastery states of knowledge components from their learning history, with a controlled text generation model that generates the next exercise based on instructor requirements, such as specified domain knowledge and target difficulty. We further explore various strategies to adapt the generation of exercises based on students’ changing knowledge states. In doing this, our model not only supports personalized generation where the instructor (or the system) can express some desired properties of the generated exercises but is also adaptive to each student’s learning progress.
We conduct extensive experiments on real-world student learning data from Duolingo2, a popular online language learning platform that offers structured and individualized learning content. Our results (§ 5) show that pre-trained LMs can help KT assess student language knowledge while student states estimated by KT can guide LMs to generate adaptive and personalized exercises. We further discuss the potential use of our model in educational applications with simulations. The simulations show that our model can dynamically adjust exercise difficulty to match individual learning progress and facilitate their learning efficiency by customizing exercise sequences.

heading: 2 Related Work
text: Adaptive Learning technologies that dynamically monitor student progress and adjust the course content based on an individual’s abilities have demonstrated various benefits in education (Becker et al., 2018). Such systems usually consist of three core components: (1) a domain model which refers to
2https://www.duolingo.com/
the content and structure of the topic to be taught, (2) a learner model which repeatedly measures and updates learner characteristics, and (3) an adaption model which combines information from the domain and learner model to offer adaptive instructions (Vagale and Niedrite, 2012; Imhof et al., 2020). In this study, we build the learner model based on the KT technique and combine the domain and adaption model into an LM which generates learning content adaptively based on user features captured by the learner model. Knowledge Tracing (Corbett and Anderson, 1994) is the technique to estimate students’ knowledge mastery s from their practiced exercises (e) and responses (r):
st+1 = fKT ((e1, r1), (e2, r2), ..., (et, rt)). (1)
Early KT approaches model fKT as variants of logistic regression, such as Item Response Theory (IRT) and Additive Factor Model (AFM) (Cen et al., 2008), or probabilistic models such as Bayesian Knowledge Tracing (Corbett and Anderson, 1994) and its variants (Yudelson et al., 2013; Käser et al., 2017). These approaches heavily rely on their assumptions of the learning process which are often incomplete. In recent years, neural networks have become the dominant method in this area. Piech et al. (2015) proposed the first Deep Knowledge Tracing model based on Recurrent Neural Networks. After that, various architectures have been applied to model different characteristics of learning, such as self-attention (Pandey and Karypis, 2019; Shin et al., 2021), memory networks (Abdelrahman and Wang, 2019), and graph neural networks (Tong et al., 2020). Exercise Generation. Previous exercise generation approaches for language learning primarily retrieve and manipulate text to create fixed types of exercises, such as gap fill and multiple-choice exercises (Agarwal and Mannem, 2011; Perez and Cuadros, 2017; Heck and Meurers, 2022), which are limited by the richness of the corpus. Besides them, some Question Generation (QG) approaches have been proposed for educational purposes (Zhao et al., 2022; Wang et al., 2021). While some of them allow for user control of certain question properties, they do not consider learners’ individual and dynamic learning needs and progress. Thus, they cannot achieve the goal of adaptive learning. Recently, Srivastava and Goodman (2021) proposed an adaptive question generation model that connects question difficulty with student knowledge.
However, it neither models students’ fine-grained knowledge states nor provides control over domain knowledge. Consequently, it is insufficient for practical use. Controlled Text Generation (CTG) methods aim to steer text generation toward certain attributes. Existing CTG approaches can be broadly classified into three types: directly training a classconditional language model (CCLM) (Keskar et al., 2019; Ziegler et al., 2019; Ficler and Goldberg, 2017), guiding a model via an attribute discriminator (Dathathri et al., 2020; Liu et al., 2020), or manipulating decoder’s logits (also referred to as weighted decoding) (Holtzman et al., 2018; Yang and Klein, 2021). This study explores difficulty and lexical control in generating language learning exercises. Additionally, we seek to adapt the model’s controllability to different users by building the dependency between control signals and individual states.

heading: 3 Problem Formalization
text: Let H≤n = {(e1, r1), ..., (en, rn)} be a student’s learning history consisting of n exercises and responses. Here, ei = {wi,1, ..., wi,|ei|} is an exercise sentence for translation and ri ∈ {0, 1}|ei| is the correctness label for each word in ei. We generate the next exercise en+1 based on:
• Cn+1: knowledge components that should be involved in en+1. In language learning, we consider a word as a knowledge component, and therefore Cn+1 = {c1, ..., c|Cn+1||c∗ ∈ V} is a subset of vocabulary V that should be included in the output. In general, the knowledge components can be user or system defined based on the current learning material.
• sn+1: a student’s knowledge state for the knowledge components (the vocabulary) after n interactions. sn+1 can be formalized as a |V|-dimensional vector with each entry between 0 and 1 indicating the mastery probability of that word.
• dn+1: the expected difficulty of en+1. We use individual performance to estimate problem difficulty. For a particular student, the difficulty of an exercise is defined as the expected number of word errors the student would make in translating it.
Given the above setting, we formalize our task as:
en+1 = argmax e
P (e|sn+1, dn+1, Cn+1), (2)
where en+1 satisfies the following constraints:
∀c ∈ Cn+1 : ∃i, en+1i:i+|c| = c, (3) dn+1 = ∑
w∈en+1 (1− sn+1[w]), (4)
corresponding to word constraint and difficulty constraint, respectively. Here, sn+1[w] represents the correct probability of translating word w; therefore, the sum of {1− s[w]|, w ∈ e} is the expected number of errors in translating e, which can be seen as a measure of the difficulty of e.
Our task is distinct from previous CTG works in two aspects: 1) our control is dynamic; student states acting as control are also learnable; 2) there is a strong dependency among control signals (Eqs. 3 and 4), which is non-trivial to learn. Note that in this work, we measure difficulty via student performance and only consider vocabulary knowledge in defining s for simplicity. Other definitions of sentence difficulty (e.g., definitions that incorporate other types of linguistic knowledge such as syntax) can be explored in future work.

heading: 4 Methodology
text: Our model is illustrated in Figure 2. We first employ a knowledge tracer T (§ 4.1) to estimate a student’s time-varying knowledge states. Then, we build an LM-based exercise generator G (§ 4.2) to create exercises based on estimated states and specified difficulty and knowledge components (words). We jointly optimize the two modules with an inconsistency loss (§ 4.3) at training and apply a constrained decoding strategy (§ 4.4) at inference. Finally, we discuss how our model can accommodate personalized learning recommendation algorithms on the fly (§ 4.5).

heading: 4.1 Knowledge Tracing
text: The goal of our knowledge tracing model T is to estimate a student’s latest knowledge state sn+1 given previous interactions H≤n. We adopt the deep knowledge tracing (DKT) model proposed by Piech et al. (2015). We concatenate past exercises as a word sequence e1:n = {w1,1, ..., wn,|en|} and past responses as a label sequence r1:n = {r1,1, ..., rn,|en|}, where wi,j and ri,j represent the jth word or label of the ith exercise. Then we
convert the two sequences into word embeddings e⃗1:n and label embeddings r⃗1:n and send them to an LSTM encoder to predict the next state sn+1:
hn = LSTM(⃗en + r⃗n;hn−1), (5) sn+1 = sigmoid(Ws ∗ hn + bs). (6)
The model is trained to predict the binary word labels of the next exercise using the estimated knowledge state. The cross-entropy loss for a single student’s history of N interactions is computed as:
Lce = |N |∑
i=1
|ei|∑
j=1
CE(ri,j , si[wi,j ]). (7)
We adopt the regularization strategy proposed by Yeung and Yeung (2018) to stabilize training:
Lr{1,2} = N∑
n=2
|V|∑
i=1
|sn(i) − sn−1(i)|{1,2}, (8)
where Lr1 ensures that only the states of relevant knowledge components are updated, and Lr2 penalizes the vibration. The final objective of T is LT = Lce+λ1 ∗Lr1 +λ2 ∗Lr2 with λ for balance.

heading: 4.2 Controllable Exercise Generator
text: Our exercise generator G is fine-tuned from a pretrained LM. Specifically, we generate an exercise e based on a student’s current knowledge state s, target words C, and expected difficulty d (we drop the interaction index to reduce clutter). We parameterize the inputs as follows:
x = [fs(s); fd(d);Emb(c1, ..., c|C|)], (9)
where knowledge state s and scalar difficulty d are projected to control vectors via two feedforward layers fs and fd, and C are mapped to word embeddings. The training objective for generating a single exercise is defined as:
LG = − |e|∑
t
logP (wt|w1, ..., wt−1,x). (10)
During training, we sample a proportion of words from reference exercises as C and calculate difficulty d from ground-truth correctness labels, whereas states s are estimated by T . At inference, d and C can be determined by instructors or the system, allowing automated and human intervention.

heading: 4.3 Joint Learning with Inconsistency Loss
text: We jointly optimize the knowledge tracer T and exercise generator G with an inconsistency loss inspired by Cui and Hu (2021), enabling the two modules to learn from each other. Concretely, after generating an exercise e, we calculate its difficulty using input state s via Eq. 4, which should be as close to the input difficulty d as possible:
Linc = |d− ∑
w∈e (1− s[w])|. (11)
Since the second term is non-differentiable due to the argmax operation involved in producing e, we replace it with "soft" tokens:
Linc = |d− |e|∑
t
(1− pt ⊙ s)|, (12)
where pt = softmax(ot/τ) is the tth distribution normalized from its logits ot ∈ R|V| with a temperature parameter τ , and ⊙ represents dot product.
For the generator G, this loss constrains the generation toward the target difficulty. For T , the LM distributions pθ provide similarity information between vocabulary words. This is analogous to the relationship of knowledge components, which has been shown helpful in knowledge tracing (Tong et al., 2020). The final objective of our model is L = LT + γ1LG + γ2Linc.

heading: 4.4 Lexical Difficulty Constrained Decoding
text: We propose a beam search-based decoding algorithm to enforce the constraints introduced in § 3. At each step, we update the beam according to:
Yt= argtopk y<t∈Yt−1,yt∈V
logP (y≤t|x) + ∑
Fi∈F αiFi(y≤t),
(13)
where Yt is the set of decoded hypotheses in step t and k is the beam size. The first term is the standard objective of beam search and the second term is a weighted combination of additional scoring functions in terms of the satisfaction of different constraints. We formulate our constraints F in Eqs. 3 and 4 as: Fc(y) = ∑
c∈C I(c,y), and Fd(y) = −|d− h(y)|,
corresponding to the satisfaction of word constraint and difficulty constraint, respectively. I(c, y) is a Boolean predicate indicating whether word c is included in sequence y and h(y) calculates its difficulty via Eq. 4.
Succinctly, the decoding algorithm works in three steps. First, we expand the current k hypotheses to k × |V| candidates. Then, we prune the search space by dropping candidates that are not in the top-kF list of any scoring functions F . Finally, we rescore the pruned candidates based on the full objective (Eq. 13) and select the k-best ones to update the beam.
However, we found that greedily applying Fd in the rescoring step would bias the decoder toward sequences with difficult words in the earlier steps. Drawing inspiration from Lu et al. (2022), we use lookahead heuristics that incorporate future estimates into the decoding process. Concretely, to score a subsequence y<t, we first greedily decode the next l + 1 steps "soft" tokens (i.e., distributions): ỹt:t+l=[pt, ...,pt+l]. Then, we combine the constraint satisfaction of decoded y<t and the estimated future ỹt:t+l: F̃c(y<t)= ∑
c∈C max(I(c,y<t), max j∈[t,t+l] P (yj = c)),
F̃d(y<t) = −|d− h(y<t)− t+l∑
j=t
1− pj ⊙ s|.
The procedure of our decoding algorithm is in Appendix A.

heading: 4.5 Plug-and-Play Personalized Generation
text: Our model can be flexibly plugged into an existing personalized learning recommendation algorithm to automatically generate novel and customized exercises. We showcase this functionality using the EXPECTIMAX curriculum planning strategy derived from DKT. Given a student’s current state sn, we can calculate the expected knowledge state after
practicing a new exercise e using our KT model T :
s̃n+1 = ∑
r∈{0,1}|e| P (r) ∗ T (sn, (e, r)), (14)
where T (·) computes the updated knowledge state given a new interaction (e, r). The probability of label sequence r is computed from sn assuming conditional independence P (r) = ∏|e| i=1 P (ri), where P (ri) = sn[ei]. EXPECTIMAX scores e based on how well it can improve a student’s average knowledge state, i.e., Fk(e) = s̃n+1 − sn , where s denotes mean of the vector. We incorporate Fk into the decoding objective (Eq. 13) and call it EXPECTIMAX-GEN.
In principle, our model can accommodate different recommendation algorithms with different ranking functions Fk. The key benefit is that our model can generate novel exercises, while retrievalbased systems can only select exercises from an existing pool.

heading: 5 Experimental Results and Analysis
text: We experiment on the English track of Duolingo Second Language Acquisition Modeling (SLAM) dataset (Settles et al., 2018), which contains about 1 million interactions of 2.6k learners over the first 30 days of learning a second language. For each student, we use the first 80% of interactions for training, and the subsequent and the last 10% for validation and testing, respectively. Details of the dataset and experimental setup are in Appendix B.
We first evaluate the ability of the KT model to estimate student knowledge states in § 5.1. Then, we analyze the effectiveness of the exercise generator in § 5.2. Lastly, we showcase the superiority of our model in two educational scenarios with simulation experiments in § 5.3.

heading: 5.1 Knowledge Tracing Evaluation
text: We use the standard AUC (ROC) as the metric of knowledge tracing in accordance with Settles et al. (2018). We denote our DKT model jointly trained with the LM-based exercise generator as DKTLM and compare it with the following baselines: 1) Ensemble (Osika et al., 2018) which is one of the winning methods of the SLAM challenge that combines a RNN and a GBDT classifier. We reimplement this model to use texts only as input and remove other side features, such as response time. We do this because we are interested in its performance in a general setting where we do not assume the availability of diverse side information; 2) the standard DKT (Piech et al., 2015) which is trained only with the KT loss LT . We use it to verify whether jointly learning with an LM can help predict student language knowledge.
We present the results in Table 1, where we can see that DKT outperforms the Ensemble model when only text features are used, and our best model DKTLM,τ=2 outperforms DKT on all metrics. We hypothesize the performance gain comes from the word similarity information entailed in the output distributions pθ of the LM. This can be regarded as the relationship between knowledge components, which is demonstrated effective in knowledge tracing (Tong et al., 2020). To verify this, we tune the temperature τ which controls the sparsity of output distributions: τ → 0 produces a sparse distribution that is too assertive and provides little relationship information, while τ → ∞ produces a uniform distribution where all words are evenly related. The results in the second section of Table 1 suggest that a medium τ improves the performance, while a small (τ=1) or large (τ=5) is harmful, particularly for predicting unseen data.
The broader message from this observation is that the knowledge encoded in pre-trained LMs has the potential to improve knowledge tracing in the domain of language learning. We also conduct an analysis of the influence of regularization terms Eq. 8, detailed in Appendix C.

heading: 5.2 Exercise Generation Evaluation
text: The main results of exercise generation are presented in Table 2, which are split according to whether the exercises are seen in the training set. Evaluation metrics include reference-based BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005), KC-Coverage which is the percentage of target knowledge components (words) that appear in the outputs, D-MAE which is the mean absolute error between the input difficulty and output difficulty, Invalid which is the percentage of exercises that have grammar errors detected using an automatic tool3. Since we generate exercises for language learning, we expect a valid exercise to be grammatically correct. We analyze the performance from the following aspects. Lexical Controllability. We first examine the lexical controllability of our model, which is crucial for generating personalized exercises for language learning. We compare our model with two baselines:1) EGH which generates the next exercise based on the student’s historical interactions; and 2) AGQH+d4 which generates the next exercise based on historical interactions and a target difficulty. The two baselines perform poorly on BLEU, METEOR, and KC-Coverage metrics, particularly
3https://github.com/jxmorris12/language_tool_python. 4We obtain its results using the code released by the authors. Note that AQG is built on a different definition of difficulty. Thus, the D-MAE result might bias toward our model. We report this metric for reference only.
for unseen data. This indicates that they cannot predict the accurate content of the next exercise based on historical data or difficulty information, possibly because there is no strong connection within a sequence of exercises or such connection cannot be captured by an LM. We note that EGH performs well on the validness metric. However, upon inspecting its results, we found the model almost only copies exercises from history, with less than 0.02% novel generations. The same issue is observed in AQGH+d where more than 90% exercises are repetitive. We follow Srivastava and Goodman (2021) to improve its novelty using a repetition penalty during the generation, but this results in far more invalid exercises (1.7%). In comparison, our model achieves a better balance between generalization ability and fluency. Effect of Student Modeling. To investigate whether student modeling helps exercise generation, we build two baselines without student knowledge states: 1) EGC which conditions generation on target KCs (words) only, and 2) EGC+d on both target words and difficulty. The former variant can be considered a keyword-to-text generation model, while the latter imposes additional difficulty control. Our full model APEGs+C+d significantly outperforms both of them, which proves our aforementioned hypothesis that a student’s dynamic knowledge states must be considered in generating adaptive and personalized exercises. An interesting observation is that incorporating difficulty control improves the performance on unseen data, indicating the model to some degree learns generalizable difficulty information. Nevertheless, our further analysis shows the model is not adaptive to students of different abilities, which will be discussed in § 5.3. Ablation Study. The key challenge of our task is to learn the dependency between student knowledge, vocabulary, and exercise difficulty (Eqs. 3 and 4). To understand which parts of our model contribute to this goal, we build two ablated variants by removing the joint learning strategy (§ 4.3) and the constrained decoding algorithm (§ 4.4), re-
spectively. As shown in the second section of Table 2, the search-based method is slightly better than the learning-based method, while combining them leads to the best performance.
We further explore the effect of the lookahead strategy on difficulty constraints. Table 3 presents the ablation results on the validation set, where we can see lookahead strategy improves both generation quality and controllability. To understand how it works, we measure the distribution of difficulty in different regions of exercise sentences. Such distribution is computed as the accumulated word difficulty in four equally sized segments of 2000 sampled sentences. As shown in Figure 3, the difficult words of reference exercises are largely concentrated in the 2nd and 4th quarter. Our decoding algorithm with lookahead produces a similar result, while removing lookahead would bias the distribution toward 2nd and 3rd quarter. This confirms our assumption that naively applying Fd would greedily select difficult words in the early steps, which is not the distribution of reference exercises. Our decoding algorithm avoids this issue by estimating the future and therefore achieves better results. Upper Bound Analysis. When we train our model, we use ground-truth difficulty d and target words C obtained from references; however, the student states s are estimated from the KT model. We conduct an upper bound analysis to understand the influence of the accuracy of s on the generation performance. Since a student’s actual mastery of every vocabulary word is not available, we choose to replace the ground-truth difficulty levels d with those estimated from s. As shown in the last section of Table 2, all metrics are considerably boosted when the inconsistency between states s and difficulty d is eliminated. This again proves the effect
The degree of highlight represents a student’s mastery of vocabulary words (the darker the harder).
of incorporating student states and explains how such information comes to play: the knowledge states explicitly convey the dynamics between control signals d, C, and target exercises e, which is non-trivial to learn by the model itself. Case Study. We provide a few cases in Table 4. We can see our model can dynamically adjust the exercise content according to specified words, target difficulty, as well as students’ different mastery states of the vocabulary. The exercises generated for advanced students (avg. state = 0.65) are generally more difficult than for poor students (avg. state = 0.32) under the same input difficulty.

heading: 5.3 Educational Applications
text: In this subsection, we showcase the potential applications of our model in two educational scenarios with simulation experiments.

heading: 5.3.1 Adaptive Difficulty Calibration
text: A crucial requirement for adaptive learning systems is to dynamically adjust the difficulty of learning items to match each student’s learning
progress (Becker et al., 2018). However, previous difficulty-controlled question generation approaches are mainly based on inherent problem difficulty, independent of individual abilities (Susanti et al., 2017; Kumar et al., 2019). Ideally, our model can achieve this goal by learning the dependency between difficulty and student knowledge states. To verify this, we generate 50 additional exercises of specified difficulties for each student after their existing interactions. At each step, we construct input by sampling a target word from the vocabulary and a difficulty level from a uniform distribution [1, 3]. We compare our full model APEGs+C+d with its variant EGC+d which achieves the best difficulty controllability for unseen data. This baseline can be considered a vanilla non-adaptive difficultycontrolled exercise generation model.
In this simulation, we are interested in whether the difficulty controllability of our model can adapt to students of various knowledge levels. To this end, we rank students based on their average knowledge states s and split the result accordingly. As shown in Figure 4, the difficulty controllability of the baseline is not reliable across different groups. In particular, it tends to generate harder (up to 2 × din) exercises for the bottom 10 percentile students but easier (up to 12 × din) ones for the top 10 percentile students, although it performs well for the intermediate 80 percentile students. In comparison, our adaptive model is also slightly biased toward the intermediate group but much more consistent than the baseline, with less than 20% fluctuations on average. Besides, we can see from the shadows that the baseline experiences huge variances at each step, indicating it is not adaptive to different knowledge states, even though the students within a group are at a similar level.

heading: 5.3.2 Improving Learning Efficiency
text: We now examine whether our model can be used to improve student learning efficiency by personalizing exercise sequences. To this end, we customize 30 continuous exercises for 50 sampled students using our proposed EXPECTIMAX-GEN (§ 4.5) and the original EXPECTIMAX. Both of them aim to maximize the expected knowledge state of the next step s̃n+1. For the former, at each step, we first find the best single word that can maximize s̃n+1 and then generate the next exercise based on the selected word and a fixed difficulty of 1. For the latter, we directly select the best exercise from the pool. We update students’ knowledge states after each practice and repeat this process until we collect 30 exercises. We compare the change in s̃ to measure which strategy is more efficient in improving students’ knowledge.
The simulation results are shown in Figure 5. We also include a randomly selected exercise sequence as a lower bound, which turns out to harm student learning most of the time. The decrease in knowledge state is possibly caused by overly difficult exercises which would lead to wrong answers and reduce the predicted probability. Under the same practice opportunities, exercises generated by EXPECTIMAX-GEN lead to faster knowledge growth than those selected by EXPECTIMAX. Upon further inspection, we found about 70% of them are unseen in the corpus. This explains the efficiency of EXPECTIMAX-GEN as it can create novel exercises targeting individual needs on the fly while EXPECTIMAX is limited by the pool.

heading: 5.3.3 Qualitative Discussions on Simulation
text: Our simulations are based on the DKT model. We note that some previous studies have observed inconsistencies between DKT behaviors and the human learning process (Shen et al., 2021). Thus, we adopt a simple regularization approach (Eqs. 5 and 6) to alleviate such inconsistencies (Yeung and Yeung, 2018), which we found can reduce the variance of simulation results and improve KT performance (Appendix C).
A popular argument regarding the relationship between the difficulty of learning content and student outcomes is that the level of difficulty should be set just above the learner’s current knowledge, i.e., d ≈ 0.5 (Settles and Meeder, 2016; GallegoDurán et al., 2018). During the simulations, we found EXPECTIMAX does not follow this heuristic but tends to generate relatively easy exercises (d < 0.3 mostly) repeatedly using certain words, consistent with the finding in Tschiatschek et al. (2022). One possible reason is that easier exercises are more likely to produce correct answers, which in turn increases the averaged predicted probability of DKT (i.e., estimated knowledge state).
Nevertheless, the above observations do not influence our conclusion as the superiority of our model comes from its ability to adapt to students’ knowledge (§ 5.3.1) and generate customized exercises targeting individual needs (§ 5.3.2), independent of the simulation policy.

heading: 6 Conclusion
text: We propose an adaptive and personalized exercise generation model combining recent advances in knowledge tracing and controllable generation using pre-trained LMs. Our approach works by learning the dynamics between exercise difficulty and student vocabulary knowledge in the domain of language learning. Experimental results on real-world language learning data from Duolingo demonstrate that our model can generate adaptive and personalized exercises needed in an Educational setting. We further showcase our model’s applicability in Education with simulation studies.
Ethics Statement
The learner data used in this study are anonymized by Settles et al. (2018) and, to the best of our knowledge, do not contain sensitive information. We foresee no further ethical or privacy concerns with the work.
Limitations
We state the limitations of this work from the following aspects. First, we make an initial assumption about the dynamics between exercise difficulty, vocabulary, and student knowledge. While we believe our assumption is sensible in the domain of language learning, we acknowledge that we make some simplifications for the ease of modeling. For example, we measure difficulty using individual performance, whereas a better way could be combining it with inherent problem difficulty, e.g., text complexity. Besides, we only consider vocabulary mastery in defining student knowledge and predicting their performance. Exploring more dimensions of language knowledge (e.g., syntax) might lead to a finer-grained personalization. Second, our model relies on student learning logs to estimate their realtime knowledge states. This model might face the cold start problem when dealing with insufficient history. Though it is beyond the scope of this study, techniques like computerized adaptive testing can be used to combat this problem. Lastly, due to the lack of a real learning environment, we discuss the educational promise of our model with simulation experiments. In the future, a user study can be incorporated to validate our conclusions.

heading: A Decoding Algorithm
text: Algorithm 1 Pseudo-code for our Lexical Difficulty Constrained Decoding Input: Target words C, difficulty d, a collection of score
functions F and their weights α, max step T , beam size k Output: k hypotheses YT in the last step 1: Y0 ← InitBeam() ▷ {<BOS>} 2: for t = 1, t ≤ T, t++ do 3: Yt ← ∅ 4: Candidates← Generate(Yt−1, 1) ▷ expand 5: for F ∈ F do ▷ prune candidates 6: Yt ← Yt ∪ argtopk
y≤t∈Candidates F (y≤t)
7: end for 8: for y≤t ∈ Yt do ▷ generate l-step lookaheads 9: ỹt+1:t+l = Generate(y≤t, l)
10: end for 11: Yt ← argtopk
y≤t∈Yt
∑ Fi∈F αiFi(y≤t ◦ ỹt+1:t+l)
12: end for 13: return YT

heading: B Experimental Setup
text: B.1 Dataset Details The statistics of our dataset are summarized in Table 5. Each interaction records a target sentence, per-token correctness labels of the student’s response, and meta information such as user nationality and response time. We group interactions by user_id (anonymous) in temporal order to obtain per-student interaction sequences. Refer to Settles et al. (2018) for more descriptions of the dataset.
B.2 Implementation Details We implement our models using the Transformers library (Wolf et al., 2020)5. Our knowledge tracing model is a three-layer LSTM with a hidden size of 100. We train it for 10 epochs with the regularization weights λ1 = 0.5, λ2 = 0.1, selected on the validation set. For the exercise generator, we fine-tune a pre-trained BART-base
5https://huggingface.co/docs/transformers/index
(Lewis et al., 2020) for up to 10 epochs. An early stop strategy is applied when the loss on the validation set does not decrease for three continuous epochs. We first train the DKT and exercise generator separately until both of them converge. Then, we jointly optimized the two models with hypearameters: γ1 = 1, γ2 = 0.8, τ = 2. During generation, we set the beam size to 4. The weights α for word and difficulty constraints are set to 0.1 and 0.5 as the word constraint is easy to achieve in our experiments. We use Nvidia Tesla A100 with 40 GB of GPU memory for training and inference. On a single GPU, one training epoch of the exercise generator takes about 30 minutes, and that of DKT takes about 7 minutes when they are separately trained. Joint training takes a longer time, about an hour for one epoch. We report the average results over three runs.
C Influence of Regularization in KT
To inspect the influence of regularization terms (Eq. 8) on the KT performance, we conduct a grid search for λ1 and λ2 on the validation set. As can be seen from Table 6 and Table 7, Lr1 consistently improves exercise-level performance at the cost of sacrificing word-level performance, whereas Lr2 with a suitable weight (λ2 = 0.3) can improve both in most cases. This suggests the students’ knowledge states transit gradually over time. We choose λ1 = 0.5, λ2 = 0.1 for the best balance.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethical and Privacy Considerations
3 A3. Do the abstract and introduction summarize the paper’s main claims? 3
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: Appendix B.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Appendix B.1
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Appendix B.1
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix B.1
C 3 Did you run computational experiments?

heading: Appendix B.2

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix B.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix B.2

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix B.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work