Using Deep Reinforcement Learning to Train and Evaluate Instructional Sequencing Policies for an Intelligent Tutoring System
Authors: Jithendaraa Subramanian
David Mostow
Venue:  (2021)
Keywords: Deep Reinforcement Learning
Intelligent Tutoring Systems
Adaptive policy
Instructional Sequencing

Abstract: We present STEP, a novel Deep Reinforcement Learning solution to the problem of learning instructional sequencing.  STEP has three components: 1. Simulate the student by fitting a knowledge tracing model to data logged by an intelligent tutoring system. 2. Train instructional sequencing policies by using Proximal Policy Optimization. 3. Evaluate the learned instructional policies by estimating their local and global impact on learning gains.  STEP leverages the student model by representing the student’s knowledge state as a probability vector of knowing each skill and using the student’s estimated learning gains as its reward function to evaluate candidate policies.  A learned policy represents a mapping from each state to an action that maximizes the reward, i.e. the upward distance to the next state in the multi-dimensional space. We use STEP to discover and evaluate potential improvements to a literacy and numeracy tutor used by hundreds of children in Tanzania.

Under review as a conference paper at ICLR 2021
USING DEEP REINFORCEMENT LEARNING TO TRAIN AND EVALUATE INSTRUCTIONAL SEQUENCING POLI- CIES FOR AN INTELLIGENT TUTORING SYSTEM
Anonymous authors Paper under double-blind review
ABSTRACT
We present STEP, a novel Deep Reinforcement Learning solution to the problem learning instructional sequencing. STEP has three components: Simulate the tutor by specifying what to sequence and the student by fitting a knowledge tracing model to data logged by an intelligent tutoring system. 2. Train instructional se- quencing policies by using Proximal Policy Optimization. 3. Evaluate the learned instructional policies by estimating their local and global impact on learning gains. STEP leverages the student model by representing the student's 's knowledge state as probability vector of knowing each skill and using the student's estimated learning gains as its reward function evaluate candidate policies. learned pol- icy represents mapping from each state to an action that maximizes the reward, i.c. the upward distance the next state in the multi-dimensional space. We use STEP to discover and evaluate potential improvements to a literacy and numeracy tutor used by hundreds of children in Tanzania.
INTRODUCTION
An Intelligent Tutoring System (ITS) aims teaching set of skills to users by individualizing instructions. Giving instruction to users requires many sequential decisions, such as what to teach, what activities to present, what problems include, and what help give. Our aim is take deci- sions which maximize long-term rewards in the form of learning gains, so Reinforcement Learning (RL) a natural approach pursue, and was first proposed by Liu 19600).
The goal of an RL agent is to learn policy defined as a mapping from state space 5 to action space A. Given any state, the RL agent follows series of actions proposed by the learned policy to maximize the long-term expected reward. In the context of an ITS, we specify the RL agent as follows:
State sy: We define the state as a combination of the student state and the tutor state. The tutor state determines the set of actions available the RI agent given timestep. We represent the student state as a vector of probabilities where element i is the estimated probability that the student knows skill i.
Action oz: The action taken by the RL agent corresponds to tutor decision at particular grain size.
Reward rx(8t, ae): Defined as the average difference between prior and posterior knowl- edge states based on the simulated student's 's response to the tutor action ae to the student simulator.
Next state 8 The knowledge vector of student after a Bayesian update based on the simulated student's response to tutor action a, in state 8, is the updated student knowl- edge state. The updated tutor state is given by the tutor simulator. The updated student knowledge state and tutor state, together gives the next state s1+1-
We instantiate STEP in the context of RoboTutor, a Finalist in the Global Learning XPRIZE Com- petition to develop open source Android tablet tutor to teach basic literacy and numeracy chil-

---

Under review as a conference paper at ICLR 2021
dren without requiring adult intervention. XPRIZE independently field-tested the Swahili version of RoboTutor for months in 28 villages in Tanzania.
Figure shows diagrammatic overview of STEP and the rest of the paper is organized as follows. Section discusses the simulation of tutor and student (the environment block). Section elaborates on the training of decision policies (the RL agent block). Section 4 evaluates the learned policies. Section relates this work to prior research. Section concludes.
Agent
State: Knowledge State Tutor State (Content area and position) Action: Thresholds, Promotion Decisions, activity
Action
Next state is returned as the updated P(Know) after knowledge tracing posterior and the updated tutor state
Tutor Simulator
Tutor Simulator presents stimulus based on tutor decisions (ike promotion rules, content area, matrix position)
Reward: Avg across skill(Poosterioc Prior. Prior_Know)
Tutor Simulator uses updated P(Know), thresholds, agent_type and other factors to update content_ area, matrix_position
Student model performs knowledge tracing based on simulated response
Student Simulator Simulates student response
Environment
Figure I: The RL setup for STEP
2 SIMULATING THE TUTOR AND THE STUDENT
To apply RL, we need to simulate the tutor's actions and the student's 's responses to them.
2.1 TUTOR SIMULATOR
The data for this paper comes from the version of RoboTTutor used during the last 3 months of XPRIZE's 15-month field study. This version rotates through three content areas (literacy, numeracy, and stories), tracking the child's position in each area's curricular sequence of successively more advanced activities. lets the child select among doing the activity that position, advancing to the next activity, repeating the same activity (from the previous content area), or exiting lotorTutor. After selecting an activity, the child may complete all or part of it before selecting the next activity. Robo'Tutor has 1710 learning activities, each of which gives assisted practice of one or more skills on a sequence of items, such as letters or words to write, number problems to solve, or sentences to read. Each item requires one more steps. Each step may take one more attempts.
LOW
MID
HI
0
100
BACK
SAME
NEXT
SKIP
Figure 2: Threshold on percentage correct attempts and their effects on tutor decisions.
The simulated tutor state identifies the current content and the child's 's position in it. RoboTutor (actual or simulated) updates the position in the content area based on the percentage of correct attempts to perform the steps in activity. Specifically, it uses fixed heuristic thresholds (called
2

---

Under review as a conference paper at ICLR 2021
LOW, MID, HI) this percentage to demote BACK to the previous position, stay at the SAME position, promote to the NEXT position, or SKIP to the position thereafter. Figure 2 gives an illustration of the same.
2.2 STUDENT SIMULATOR
A student simulator should behave like students who the tutor. Accordingly, the simulator uses a Bayesian Knowledge Tracing (BKT) student model trained on logged data using HOT-DINA. has the same Guess, Slip, and Learn parameters standard BKT, but estimates the Knew parameter based on skill difficulty and discrimination and student proficiency from Item Response Theory. Thus, HOT-DINA extrapolates from the student's knowledge of other skills, and other students' knowledge of this skill, albeit at a high computational cost to fit the model. Xu & Mostow (2014) found HOT-DINA to have higher predictive accuracy than standard BKT.
To limit computation time, we fit the model on logged data from a single village, consisting of 42,010 attempts by children to apply 22 skills. We one proficiency parameter for each child and 5 parameters for each skill (Guess, Slip, Learn, Difficulty, and Discrimination), 118 parameters in total. (Fitting 5 separate parameters per activity instead of per skill might achieve higher accuracy but would require fitting 8,558 paramcters.) We use MCMC sampling for Bayesian inference with PyStan rather than the OpenBUGS Gibbs sampling package used in the original HOT-DINA work because PyStan is faster and handles larger datasets. Nevertheless, fitting the 118 --raammeterr HOT- DINA model to 42,010 attempts took approximately days on a supercomputer with 128 GB and 28 cores.
Table I: Converged parameters for the HOT-DINA student model
Parameter
Mean (converged)
SD (converged)
01
-0.17
0.93
02
1.77
0.64
0g
-0.65
0.72
04
-0.73
1.13
0.
-0.38
0.78
0s
-0.28
0.55
07
-0.48
0.7
0g
0.14
0.47
b1
1.25
0.73
b2
1.25
0.72
b3
1.25
0.72
b4
1.25
0.72
b5
1.42
0.71
bg
1.49
0.68
Table shows converged values for a subset of HOT-DINA parameters. For example, the eight values refer to the 8 student proficiency parameters of the student model. For simplicity, we show only the first 6 values of b (skill difficulty parameter) in the table. Once we obtain the model parameters, we need two things to be done for the student simulator to be successful: given an activity, we should be able simulate whether student gets activity right or wrong. Based on this response, we should be able to perform knowledge tracing over multiple skills to update the student's knowledge probabilities. For simulating a student's performance on an activity we first estimate P(Getting Activity Correct) as in equations 2 and 3 below. We then simulate the student response (right or wrong) by doing biased coin flip based on this estimated probability. Since we now have simulated student response, the update equations 3-5. The next few lines cover some basic notation and update equations for simulated learning of student. It should be noted that variables a, y, and Y are all binary, ie., they take on value of either 1 or 0.
perform knowledge tracing over multiple skills using
0,
Proficiency of student n
ag
Discrimination of skill k
3

---

Under review as a conference paper at ICLR 2021
by
Difficulty of skill k
if activity exercises skill k, otherwise
an
(t)
Probability that student knows skill at time-step
Unk
Probability that student answers an activity exercising only skill k at time-step t
y(O)
=1 Probability that student n gets activity j correct at time-step
nj
95k
(0) nn k=1
II
(1)
1 + e.r(e-eppp.. (0,, bk))
with 1) = (1 slips) (a (1) 1) + quesss(a (8)
(n 0)
(2)
(Yr)) 1) II
(ink = 1)0r
(t)
(3)
k=1
(0 = 1| nj = 1) = (a 1) * 1- slipk
(4)
(Nnnk = 1)
95k
(a = 1|x nj = 0) = (a = 0) *
guessk
(5)
(wink = 1)
(a (++1)
1) 1) (a (t)
1 1| nj
+ (learns (a 01 (())
(6)
3 TRAINING POLICIES WITH PPO
We have already discussed the student simulator and tutor simulator in last section. In this section, we discuss the training policy using STEP in the context of RoboTutor.
3.1 THE REWARD FUNCTION
The RL agent learns a decision policy - that is, a mapping from states to actions - that maximizes the total expected reward of following the policy wo. As the reward function for student n, we the knowledge gain as estimated by the student model, i.c. posterior minus prior estimates of Pr(student knows skill k), averaged over all skills. The posterior and prior refer to the knowledge states before and after applying the bayesian updates (equations 3-5) on an activity decided by action ag The information for prior knowledge implicitly present in the knowledge state of s; In order save computational time. we learn policy for episodes of 100 timesteps using PPO after which the episode terminates. Though our experiments stick to finite-horizon undiscounted returns with 100 steps, is trivial to extend this approach to any finite number of steps even to infinite-horizon discounted returns with discount factor 7 E (0, 1) so the rewards vanish at large timesteps. The reward function T, for student at given step is given by learning gains of a student due attempting activity, as given in equation (7) where K is the total number of skills (22 for RoboTTutor). The returns are just the sum of rewards over T=100 steps. (Previous methods used reward=0 based on correct attempt or something else. Useful to mention this?)
(t+1) = 1) (a = 1) aak
(1 ar)
K
According to the student model trained by HOT-DINA on the 8 children's log data, their prior averaged 0.55 and their posterior averaged 0.73, a gain of 0.18 over their final usage consisting of 42,010 attempts (up to 3 months). Their posterior after their first 100 attempts averaged across the students was 0.64, for an average gain per attempt of 0.09/100 0.0009.
We can train different types of RL agents depending on their state space and range of actions, which depend on how far they depart from RoboTutor's current decision policy.
4

---

Under review as a conference paper at ICLR 2021
3.2 STATES, ACTIONS AND RL AGENT TYPES
We model student n's state as the vector of estimated probabilities of student n knowing skill k [a 1), (a 1)]² Depending the RL agent type, the tutor state may include the current (active) content area (literacy, numeracy or stories) and the student's current position in the curricular sequence for that area; just the content area; neither.
Alternative ranges of actions for each agent type:
Type threshold actions (LOW, MID, HI), each action E (0.0, 1.0)
Type 2: promote-demond decisions choosing of BACK, SAME, NEXT, and SKIP.
action from Discrete(4) action space.
Type an activity from the current content area. action from Discrete(x) action space where is the number of activities in the current content area.
Type 4: any activity from any content area. action from a Discrete(1710) action space.
Table 2 summarizes types of RL agents we consider, whose tutor simulators operate in the follow- ing ways:
Type preserves RoboTutor's current choices but adjusts thresholds that affect promote- demote decisions indirectly
Type 2 eliminates the need for thresholds by choosing promote-demote decisions directly from state rather than from thresholds
Type 3 can jump any activity within the current content area
Type 4 can jump any activity in any content area; area rotation constraint is removed
Table 2: Tutor state and action range for each agent type
Type
Tutor State
Action Range
#Actions
1
Content area, position in curriculum
Thresholds
3, Continuous
Content area, position in curriculum
BACK, SAME, NEXT, SKIP
1, Discrete
3
Content area
Any activity in content area
Discrete
4
None
Any activity in any content area
Discrete
EVALUATING LEARNED POLICIES
We evaluate the learned policies along two metrics which assess the local and global impacts. The local impact is the average change in reward by replacing single historical choice of activity with the activity chosen by the policy. The global impact is the overall change in reward per attempt by following the learned policy from the first attempt. Table evaluates the learned policies, agent type, by their impact on learning gains (expected reward) of the first 100 attempts averaged across the 8 children, compared to the historical baseline of 0.0009.
Table 3: Evaluating policies based on local and global impact
Agent Type
Local impact
Global impact
9,98x
1.37x
2
10.81x
1. 1.44x
3
10.99x
1.66x
4
11.64x
2.47x
From the table, increasing trend for both the local and global impacts and is in-line with our beliefs that less constrained the RL agent is. the greater the impact. Interestingly, local impacts seem to exceed global impacts for all cases. This is because successive attempts have independent
5

---

Under review as a conference paper at ICLR 2021
rewards since the Prior(Know at step t) does not depend on the policy- cacesy--ppoosedd-cccc action at step for local impacts. Thus, if some less-known activity allows a large one-time gain at step 1, the local impact allows it to occur multiple times in subsequent steps, beating average global gains per step.
Figure 3 evaluates the current RoboTTutor policy (red) versus the agent's learned policy the sim- ulated student for agent types to 4. Every agent type has 8 subplots associated to each simulated student built off the students' data that we logged through Rutor, The y-axis corresponds to student's average knowledge across skills and the x-axis corresponds to the number of attempts of a student. Since we restricted the ame-horizon to 100, we can see that the x-values have an upper limit of 100 attempts.
E20091
Figure 3: RoboTTutor policy (red) vs learned policy (green) studying student knowledge vs attempts for all 4 agent types
5 RELATION TO PRIOR WORK
Various researchers have worked on Reinforcement Learning for instructional sequencing Doroudi et al. (2019). Table 4 summarizes work that used BKT for student modeling or Deep RL for opti- mization.
Prior work by Yudelson et al. (2013) and Pardos & Heffernan (2011) used BKT methods that fit a parameter for the probability of already knowing a skill prior to instruction. In contrast, we use HOT-DINA, a higher-order BKT-IRT hybrid that estimates this probability based on skill difficulty and student proficiency, achieving higher accuracy than standard BKT.
Recent work by Shen et al. (2018) instructional sequencing used deep reinforcement learning, specifically Deep N-Networks. STEP uses a more powerful deep RL method, namely Proximal Policy Optimization (PPO) Schulman et (017).

---

Under review as a conference paper at ICLR 2021
Some prior work reviewed by Doroudi et al. (2019) specifies the reward 1 when the probability of knowing a skill reaches 0.95 and otherwise. In contrast, we define reward as estimated learning gain, so to differentiate between actions that yield different gains in student knowledge.
Table 4: Prior work and their approaches
Paper
Student model
Adaptive policy
David al. (2016)
BKT
Threshold
Whitehill & Movellan (2017)
POMDP
Policy Gradient
Doroudi et al. (2017)
BKT
Mastery
Shen et al. (2018)
Model-Free
DQN
Segal al. (0188)
BKT
Threshold
Doroudi et al. (2019)
BKT
Inc Time
This paper
HOT-DINA
PPO
CONCLUSION
This paper contributes novel framework for optimizing instructional sequencing in intelligent tutoring system by combining knowledge tracing with deep reinforcement learning and evaluating the learned decision policy historical data. We fit simulated student on authentic log data from real children using Robo'Tutor in Tanzania, in contrast to earlier work that used synthetic data. We trained the student model using HOT-DINA because is more accurate than other knowledge tracing methods. We used Proximal Policy Optimization because it learns better than previous reinforce- ment learning methods applied previously to ITS. use knowledge probabilities estimated by the student model as state and directly optimize for learning gains which we set the reward. We evaluated the learned policies' local and global impact on expected knowledge gains relative to a historical baseline and explained the somewhat surprising results observed.
The work has several limitations. The evaluation based on data from 8 children from one village to save computational expense. The evaluation extrapolates from historical data. Future work should test the actual impact of learned policies on children's learning. We also do not make predictions on kids backing out while doing activities and remove the 10-item per activity constraint while performing our experiments. Future work should include predicting student disengagement.
We 118- 18-parameter HOT-DINA model to parameter HOT-DINA model might have been more accurate since it has parameters per activity instead of per skill. Developing other student models that are more accurate than HOT-DINA might be fruitful. We focused on learning decision policies for choosing activities. Future work could explore optimizing tutor decisions other levels of granularity, such as selecting which items to practice and what assistance to provide.
computational expense, while the 8,558-
REFERENCES
Y. B. David, A. Segal, and Y. K. Gal. Sequencing educational content in classrooms using bayesian knowledge tracing. Proceedings of the Sixth International COnference on Learning Analytics Knowledge, pp. 354-363, 2016.
S. Doroudi, V. Aleven, and Brunskill. Robust evaluation matrix: Towards a more principled of- fline exploration of instructional policies. Proceedings of the Fourth (2017) ACM Conference Learning @ Scale, pp. 3-12, 2017.
S. Doroudi, V. Aleven, and E. Brunskill. Where's the reward? Int J Artif Intell Educ, 29:568-623 2019.
Chung Laung Liu. A study in machine-aided learning. PhD thesis, Massachussets Institute of Technology, 1960.
Z. A. Pardos and N. T. Heffernan. Kt-idem: Introducing item difficulty to the knowledge tracing model. UMAP, 2011.
7

---

Under review as a conference paper at ICLR 2021
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
A. Segal, Y. B. David, J. J. Williams, K. Gal, and Y. Shalom. Combining difficulty ranking with multi-armed bandits sequence educational content. arXiv preprint 1804, 05212, 2018.
Shen, M. S. Ausin, B. Mostafavi, and M. Chi. Improving learning & reducing time: constrained action-based reinforcement learning approach. Proceedings of the 2018 Conference on User Modeling Adaptation and Personalization, 2018.
Whitchill and J. Movellan. Approximately optimal teaching of approximately optimal learners. IEEE Transactions Learning Technologies, 2017.
Yanbo Xu and Jack Mostow. A unified 5-dimensional framework for student models. Proceedings of the EDM2014 Workshop on Approaching Twenty Years of Knowledge Tracing, 2014.
Michael V. Yudelson, Kenneth R. Koedinger, and Geoffrey J. Gordon. Individualized bayesian knowledge tracing models. International Conference on Artificial Intelligence Education, 2013.
8


Reviews:
[{"review_id": "v0KWedwiPC6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper intends to contribute “a novel framework for optimizing instructional sequencing in ain intelligent tutoring system”. More specifically, this framework uses deep reinforcement learning and evaluates learned policies on historical data.\n\nA major strength of this paper is working with real human data from an application with obvious positive human impact. That working with this rich data comes necessarily with only working with a small amount of data is understandable, and it is not a weakness of the paper.\n\nThe most significant weakness of the paper is that it does not articulate a contribution that centers on representation learning -- the focus of this conference. A representation of student knowledge over time is learned (the 118-parameter HOT-DINA model), but this representation was already contributed by past researchers and does not seem to match the intended contribution of the paper. An action policy for controlling tutor behavior as a function of student knowledge state is also learned, but policy (or its internal details) are not examined from a representation learning standpoint. (For example, does some aspect of the different learned student-specific policies appear to recover/mimic another known aspect of those student identities? If so, would that be a good or bad result?)\n\nThe next most significant weakness is that when the paper makes novel choices, those choices are neither evaluated nor strongly motivated based on the past literature. Why PPO over DQN? (PPO can work in certain settings that DQN cannot, but past work already demonstrated DQN for a very similar setting.) Why HOT-DINA over BKT? (HOT-DINA is a much more expressive model, but the small amount of valuable historical data for this setting may limit the effectiveness of expressive models due to overfitting.)\n\nAdditional weaknesses are noted in the additional section-by-section feedback below.\n\nThis reviewer recommends (2) strong rejection. This is not inherently a paper about representation learning, and, even as a generic applied machine learning paper, it does not sufficiently motivate or evaluate the intended contribution of “a novel framework” for using machine learning in a specific application.\n\nQuestions for the authors:\n\n* Are there structural reasons why Q learning (e.g. DQN) cannot be applied in this setting?\n* Is there a way to verify that HOT-DINA is not overfitting in a way that makes evaluating the system on the same historical information used to train it unsound?\n\nSection-by-section notes:\n\nTitle\n- Focus on “instructional sequence policies” (for ITS, using RL), was hoping for something that sounded immediately relevant to representation learning.\n\nAbstract\n- It sounds like the method might not be as important as the application.\nHopefully we’ll hear more about the representation learned because this is an ICLR submission.\n\nIntroduction\n- The introduction doesn’t state the intended conclusion or motivate the novel parts of the work for the reader.\n- There’s a learned policy in here, but what is the learned representation you want to highlight for this specific venue on representation learning (ICLR)?\n\nSimulating\n- This is how you fit a pre-existing representation to new data, it doesn’t seem like this is the contribution of the paper.\n\nTraining\n- Missing use of domain knowledge:\n  * Is the the 100 timesteps considered in PPO training based on 100 being a typical number of interaction steps with the ITS among the Tanzanian students?\n  * Even with a finite horizon, the rate at which students decide to exit the activity could be used to motivate a discount factor < 1. Presumably you have this information as well.\n- Notes like “Useful to mention this?” suggest the paper was submitted in an incomplete state.\n- It’s interesting to list the design alternatives for representing actions, but each should be contextualized with references to past research that used something similar.\n\nEvaluating\n- The evaluation in terms of local and global impact is unfamiliar to this reviewer (who knows other RL+ITS work). Not enough information is given to pin down exactly what local impact measures.\n- What is the source and meaning of the historical baseline number?\n- Figure 3 is too busy to interpret. Consider presenting it as a chart that aggregates across students using a single line (plus error bars) to represent the mean (plus stddev) state over time for the two policies.\n- The way student-specific models are “evaluated against historical data” on those specific students suggests we are just testing on the training data. Why is this a valid methodology for this application? Cite past work on evaluating RL-based ITS systems to motivate your methods.\n\nRelationship\n- Consider covering work by others much much earlier in the paper so that the reader can understand why you made the choice you did (and that you can convince them that you know RL has been applied to ITS across many decades previously).\n- Near “STEP uses a more powerful deep RL method” -- it is true that policy gradient can be applied in certain applications where Q-learning cannot, but if we aren’t given a note as to whether this is the case in the current application. Based on previous work, it seems like Q learning was applicable. Thus, using policy gradient methods (including PPO) would seem to add needless complication.\n- This section indeed state how the current work is different from past work, but it does not motivate the differences. If others were successful with different methods, why change them for this paper?\n\nConclusion\n- “This paper contributes a novel framework” -- what is novel is the combination of the HOT-DINA student knowledge model with the PPO reinforcement learning approach (within a larger framework shared by many other papers).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UmlMS4uXOx_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper describes variants of an intelligent tutoring system (ITS) developed using a newer (but previously published) variant of Knowledge Tracing (HOT-DINA) for assessing student proficiency and an RL algorithm (PPO) for making decisions on items and content areas to try next.  An empirical simulation calibrated to 8 students is reassessed on the same student simulations and improvements over the original tutoring system are empirically demonstrated.  Four variants with differing levels action granularity and knowledge racing are analyzed.\n\nReview:\nI really like the direction this paper is going in but the results as currently reported seem rushed and under-analyzed.  The paper is shorter than the max length of ICLR papers yet omits crucial details about the action selection strategies and presents the empirical gains with charts that lack labels and no textual analysis (just graphs without context) of the individual student trajectories.  There is also no comparison to the existing state-of-the-art from Shen et al. (cited).  Also, the empirical study seems to have been done on the same 8 students on which the model was calibrated, which likely caused significant overfitting and puts the generality of the empirical results in doubt. Finally, much of the IRT terminology is not defined until later in the paper and the related work on RL for intelligent tutoring systems could use some additions (mentioned below).  In summary, this is a good idea and the results are showing promise, but the paper is not ready for publication in its current form.\n\nDetails:\nMy biggest concern with the paper is the lack of rigor in the empirical analysis.  First and foremost, it appears the data from the same 8 students were used for calibrating the model (both the knowledge tracing and the PPO decision-making training) and then those same students were considered in the simulation (testing) of the models.  It seems likely the models were overfit to the data from these students.  A proper empirical study in this kind of educational setting needs to train the models on data from one set of students and then use a holdout set to test it.\n\nNext, the metrics and charts reported in the paper do not make the improvement clear.  The “local impact” metric reported in the paper does not seem like a good way of assessing improvement since (as the authors admit) it multi-counts improvements from one step in subsequent steps.  I suggest removing that metric for the more grounded global metric.  The charts reported in Figure 3 are not understandable or analyzed in the text.  Why are some of the sub-plots blank?  And where are the labels saying which chart is associated with each of the 4 variants?   We can’t tell which problem setting matches each chart.  Finally, while the green line (new method) outperforms the baseline in the majority of charts, there are some where it does not.  But no analysis or explanations are provided despite there being plenty of room in the paper.\n\nFinally on the empirical side, the authors reference the work of Shen et al. who applied DQN to this ITS problem, but the authors do not provide an empirical comparison to this DQN based approach.  Since that is the state of the art, it seems necessary to apply that algorithm here and see if the gains over the baseline are comparable to the new algorithm.\n\nOn the algorithm side, while most of the approach is fairly clear, the description of the “Type 1 agent” omits crucial details about how items or subject areas are actually selected.  Unlike the other 3 cases, where actions are clearly related to items, Type 1 has an action that moves a threshold.  But how is that helpful?  Can’t an agent just move the threshold very low so it thinks all students have mastered all skills?  And how does moving a threshold determine an item to be given to a student.  More detail is needed to understand this case, which seems much different from the other 3.\n\nOn terminology, the paper often uses terms (such as (Guess, Slip, Learn…). on page 3 or  “b” in the last paragraph of page 3)or presents results (for instance the thetas in table 1) before the definitions of these terms.  Since readers at ICLR are unlikely to have an IRT background, the definitions on page 4 need to be moved up to a terminology section towards the beginning.\nOn related work, the paper did a good job referencing several very recent papers but failed to reference some of the papers related to core concepts and also lacks references to other slightly older works that used the same mix of student knowledge tracing with offline RL.  \n\nExamples:\nHOT-DINA first appears on page 3 but no citation is given\nItem response theory is mentioned on page 3 with no citation\n\nOther RL for tutoring systems work;\n“Cognitive modeling to represent growth (learning) using Markov decision processes” – builds a Bayesian representation of student skills and then uses a POMDP to plan in the belief space over skills (similar to the current work’s representation)\n\n“Learning a Skill-Teaching Curriculum with Dynamic Bayes Nets” – similar to current work, calibrates a Bayes Net based on student data  and uses RL to create new policies.\n\nThe last sentence of the first paragraph of section 3.1 seems to be an author’s note or question to co-authors.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5YkkST5sn0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\t1. Summary of paper:\n\t\ta. The paper contributes a deep RL approach to learning instructional sequencing. The approach called STEP starts by simulating tutor and student models. The tutor simulation is based on the RoboTutor ITS and the student simulations are fit to historical data of children interacting with RoboTutor using the HOT-DINA approach. The instruction sequencing model is then trained using PPO and evaluated using novel measures of sequencing decision impact on local and global learning gains (estimated from running student simulation). The paper contributes one set of experimental comparisons between four variations of the sequencing agent and the RoboTutor baseline in 8 runs of the two systems.\n\t2. Strengths\n\t\ta. At a high level, the proposed approach is well motivated, using RL to optimise parameters in the existing tutoring system or more granularly make sequential decisions about what activity to provide to the student.\n\t\tb. The local and global reward design is a strong contribution that uses historical data and the student simulation (knowledge tracing) to estimate credit for policy decisions in a counterfactual manner.\n\t\tc. The future work discusses the limitation of not using actual children's scores to evaluate this learning model.\n\t3. Weaknesses\n\t\ta. Novelty/impact and context within related literature: \n\t\t\ti. It is difficult to judge the novelty of this work since the related work section is too brief and does not actually describe several works compared against. Additionally, table 4 is not descriptive enough, has potentially relevant work undescribed (Whitehall & Movellan's 2017 POMDP/policy gradient approach), and has work referenced previously missing from it (Yudelson et al. 2013, Pardos & Heffernan 2011).\n\t\tb. Experimental rigour:\n\t\t\ti. Two strong claims are made in the related work section that do not have sufficient experimental evidence. 1) A direct comparison of BKT to HOT-DINA for modelling student knowledge gains is required to show the impact of this central claimed contribution. 2) A direct comparison against Shen et al. 2018 is required to show that PPO is indeed more effective in this domain. This is also important since it is a central claimed contribution.\n\t\tc. Clarity: The clarity of the paper needs significant effort to improve. Instances below.\n\t\t\ti. The structure of the paper reads like a technical report rather than an empirical investigation. The contents would be far easier to understand with a different structure emphasising a research question, background to understand/motivate it, methodology to answer it, results, and discussion.\n\t\t\tii. Several sections are far clearer in the supplementary data document. The reproducibility of the paper is boosted by this. Given the extra space available, several sections could stand to be transferred to the main paper. My original review did not include supplementary data and points around describing data and examples are boosted by adding them to the main paper.\n\t\t\tiii. Until significant rereads, it isn't clear what the relationship is between RoboTutor and STEP/this work. With my current understanding, RoboTutor is an external system that has been used to collect data on children learning various skills using it. This data was then used to evaluate STEP in terms of estimated learning gains.\n\t\t\tiv. The tutor simulator section describes how RoboTutor functions. The student simulator describes how the knowledge tracing model works. Example differences between activities, skills, steps, etc. would make the content clearer. \n\t\t\tv. In the tutor simulation section it states that the child can select activities, so this part is replaced by RL decision-making in agent type 3 and 4, right? This relates to my question about RoboTutor and this work. I am understanding that the current work simulates the exact decision-making process of RoboTutor but can vary/change that process according to the different agent types. Is that correct?\n\t\t\tvi. Bayesian Knowledge Tracing needs a citation and a brief explanation in a background section.\n\t\t\tvii. HOT-DINA needs a citation and a brief explanation in a background section.\n\t\t\tviii. Item Response Theory needs a citation and a brief explanation in a background section.\n\t\t\tix. What is the difference in computational cost between a BKT approach and HOT-DINA?\n\t\t\tx. A clearer highlighting of what data was used would make it easier to read the article. What was the contents of the data collected to enable knowledge tracing in the student simulator? This also ties in to the comment about examples. Adding a running example of an activity, skill, step for the tutoring task would enable a description of what data is collected to measure student knowledge in the data set.\n\t\t\txi. What do the guess, slip, learn, etc. Parameters measure?\n\t\t\txii. \"We use MCMC sampling for Bayesian inference with PyStan rather than the OpenBUGS Gibbs sampling package used in the original HOT-DINA work because PyStan is faster and handles larger datasets.\" This line needs references for all software used, but most importantly, a reference to the original HOT-DINA work is necessary.\n\t\t\txiii. The sole paragraph on page 3 is difficult to parse and seems important to understand how the student simulator works. The paragraph is dense and conversational in style referencing a sequence of steps without making them clearer and referencing equations that are on the next page. It would be much clearer to have the exposition and equations interspersed and use pseudo-code or a flowchart to explain the steps performed to simulate the student (at least a list).\n\t\t\txiv. \"We can train different types of RL agents depending on their state space and range of actions, which depend on how far they depart from RoboTutor’s current decision policy.\" This would make sense as the start of a new section on the experimentation.\n\t\t\txv. A stronger partitioning of content would also be achieved by calling a potential new section at this point methodology, experiments, or evaluation. This would also help organise the next section of state, action, and agent types into a concrete experiment for which the paper is describing the state and action spaces.\n\t\t\txvi. In section 3.2, it is confusing to have states, then actions, then agent types described. It seems like one experiment with 4 experimental variants (agent types) and a baseline (RoboTutor).\n\t\t\txvii. Table 2 does not convey much more information than the explanation before it.\n\t\t\txviii. Figure 4 is very difficult to parse. Instead of showing 36 subplots (with 4 empty subplots) for 8 students and 4 agents comparing against the RoboTutor baseline, it would be far easier to compare performance against students or against agent types, by combining all 8 student type runs for each variant into a single variance-shaded run in a single graph (e.g. using https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.fill_between.html). That would allow for much higher information density and an at a glance comparison between the 5 runs being compared combined across all 8 students. At the very least this should be done by combining all 5 runs into 8 separate graphs, though the previous approach is preferable.\n\t\t\txix. Related work is far too brief and does not make clear what is being compared for many cited works. E.g. It isn't clear why the works that specify reward a certain way in Doroudi et al. (2019) show a disadvantage to the current approach, the works in table 4 don't specify why the current approach is an advance over their contributions.\n\t\td. Reproducibility: \n\t\t\ti. Section 3 (and the paper in general) contains far too little information about the policy representation, learning hyperparameters, network architecture, etc. to understand the contribution.\n\t4. Recommendation: \n\t\ta. Per the weaknesses in the review above, I recommend the paper for rejection. I don't think the weaknesses in experimental rigour can be fixed in time, content space, or degree to support acceptance. The significant editing required to fix the other issues also seem unrealistic in time and space.\n\t5. Minor Issues\n\t\ta. \"(Previous methods used reward=0 or 1 based on correct attempt or something else. Useful to mention this?)\" There is a comment remaining in the paper that should have been removed.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an important and interesting problem and a potentially interesting approach. Unfortunately, the reviewers agree that the current version isn't appropriate for ICLR in its current form. However, hopefully the feedback will be useful for the authors in revising and resubmitting this paper to another venue.
Decision: Reject