Title: EduQate: Generating Adaptive Curricula through RMABs in Education Settings
Venue: NeurIPS 2024
Accepted: no
Keywords: Machine Learning in Education

Restless Multi-armed bandits

Adaptive Curricula
Abstract: There has been significant interest in the development of personalized and adaptive educational tools that cater to a student's individual learning progress. A crucial aspect in developing such tools is in exploring how mastery can be achieved across a diverse yet related range of content in an efficient manner. While Reinforcement Learning and Multi-armed Bandits have shown promise in educational settings, existing works often assume the independence of learning content, neglecting the prevalent interdependencies between such content. In response, we introduce Education Network Restless Multi-armed Bandits (EdNetRMABs), utilizing a network to represent the relationships between interdependent arms. Subsequently, we propose EduQate, a method employing interdependency-aware Q-learning to make informed decisions on arm selection at each time step. We establish the optimality guarantee of EduQate and demonstrate its efficacy compared to baseline policies, using students modeled from both synthetic and real-world data.

Abstract

There has been significant interest in the development of personalized and adaptive educational tools that cater to a student's individual learning progress. A crucial aspect in developing such tools is in exploring how mastery can be achieved across a diverse yet related range of content in an efficient manner. While Reinforcement Learning and Multi-armed Bandits have shown promise in educational settings, existing works often assume the independence of learning content, neglecting the prevalent interdependencies between such content. In response, we introduce Education Network Restless Multi-armed Bandits (EdNetRMABs), utilizing a network to represent the relationships between interdependent arms. Subsequently, we propose EduQate , a method employing interdependency-aware Q-learning to make informed decisions on arm selection at each time step. We establish the optimality guarantee of EduQate and demonstrate its efficacy compared to baseline policies, using students modeled from both synthetic and real-world data.

1 Introduction

The COVID-19 pandemic has accelerated the adoption of educational technologies, especially on eLearning platforms. Despite abundant data and advancements in modeling student learning, effectively capturing the learning process with interdependent content remains a significant challenge [9]. The conventional rules-based approach to creating personalized learning curricula is impractical due to its labor-intensive nature and need for expert knowledge. Machine learning-based systems offer a scalable alternative, automatically generating personalized content to optimize learning [22, 24].

One possible approach to model the learning process is the Restless Multi-Armed Bandits (RMAB, [26]), where a teacher agent selects a subset of arms (concepts) to teach each round. However, RMAB's assumption that arms are independent is unrealistic in educational settings. For example, solving a math question on the area of a triangle requires knowledge of algebra, arithmetic, and geometry. Practicing this question should enhance proficiency in all three areas. Models that ignore such interdependencies may inaccurately predict knowledge levels by assuming each exercise impacts only a single area.

In response to this challenge, we introduce an interdependency-aware RMAB model to the education setting. We posit that by acknowledging and modeling the learning dynamics of interdependent content, both teachers and algorithms can strategically leverage overlapping utility to foster mastery over a broader range of topics within a curriculum. We advocate for RMABs as a fitting model for this context, as the inherent dynamics of such a model align closely with the learning process.

In this study, our objective is to derive a teacher policy that effectively recommends educational 33 content to students, accounting for interdependencies among the content to enhance overall utility 34 (that characterizes understanding and retention of content). Our contributions are as follows: 35

36

37

38

39

40

41

42

43

We introduce Restless Multi-armed Bandits for Education (EdNetRMABs), enabling the modeling of learning processes with interdependent educational content.

We propose EduQate, a Whittle index-based heuristic algorithm that uses Q-learning to compute an inter-dependency-aware teacher policy. Unlike previous methods, EduQate does not require knowledge of the transition matrix to compute an optimal policy.

We provide a theoretical analysis of EduQate, demonstrating guarantees of optimality.

We present empirical results on simulated students and real-world datasets, showing the effectiveness of EduQate over other teacher policies.

2 Related Work and Preliminaries 44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

2.1 Restless Multi-Armed Bandits

The selection of the right time and manner for limited interventions is a problem of great practical importance across various domains, including health intervention [17, 5], anti-poaching operations [20], education [13, 6, 2], etc. These problems share a common characteristic of having multiple arms in a Multi-armed Bandit (MAB) problem, representing entities such as patients, regions of a forest, or students' mastery of concepts. These arms evolve in an uncertain manner, and interventions are required to guide them from "bad" states to "good" states. The inherent challenge lies in the limited number of interventions, dictated by the limited resources (e.g., public health workers, the number of student interactions). RMAB, a generalization of MAB, offers an ideal model for representing the aforementioned problems of interest. RMAB allows non-active bandits to also undergo the Markovian state transition, effectively capturing uncertainty in arm state transitions (reflecting uncertain state evolution), actions (representing interventions), and budget constraints (illustrating limited resources).

RMABsand the associated Markov Decision Processes (MDP) for each arm offer a valuable model for representing the learning process. Firstly, leveraging the MDPs associated with each arm provides the flexibility to adopt nuanced modeling of learning content, accommodating different learning curves for various content based on students' strengths and weaknesses. Secondly, the transition probabilities serve as a useful mechanism to model forgetting (through state decay due to passivity or negligence) and learning (through state transitions to the positive state from repeated practice). Considering these aspects, RMABs prove to be a beneficial framework for personalizing and generating adaptive curricula across a diverse range of students.

In general, computing the optimal policy for a given set of restless arms in RMABs is recognized as a PSPACE-hard problem [18]. The Whittle index [26] provides an approach with a tractable solution that is provably optimal, especially when each arm is indexable. However, proving indexability can be challenging and often requires specification of the problem's structure, such as the optimality of threshold policies [17, 16]. Moreover, much of the research on Whittle Index policies has focused on two-action settings or requires prior knowledge of the transition matrix of the RMABs. Meeting these conditions proves challenging in the educational context, where diverse students interact with educational systems, each possessing different prior knowledge and distinct learning curves for various topics.

WIQL [5], on the other hand, employs a Q-learning-based method to estimate the Whittle Index and has demonstrated provable optimality without requiring prior knowledge of the transition matrix. We utilize WIQL as a baseline method in our subsequent experiments.

In a recent investigation by [12], RMABs were explored within a network framework, requiring the 77 agent to manage a budget while allocating a high-cost, high-benefit resource to one arm to 'unlock" 78 potential lower-cost, intermediate-benefit resources for the arm's neighbors. The network effects 79 emphasized in their work are triggered by an intentional, active action, enabling the agent to choose 80 to propagate positive externalities to a selected arm's neighbors within budget constraints. In contrast, 81 our study delves into scenarios where network effects are indirect results of an active action, and the 82 agent lacks direct control over such effects. Thus, the challenge lies in accurately modeling these 83 network effects and leveraging them when beneficial. 84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

2.2 Reinforcement Learning in Education

In the realm of education, numerous researchers have explored optimizing the sequencing of instructional activities and content, assuming that optimal sequencing can significantly impact student learning. RL is a natural approach for making sequential decisions under uncertainty [1]. While RL has seen success in various educational applications, effectively sequencing interdependent content in a personalized and adaptive manner has yielded mixed or insignificant results compared to baseline teacher policies [11, 21, 8]. In general, these RL works focus on data-driven methods using student activity logs to estimate students' knowledge states and progress, assuming that the interdependencies between learning content are encapsulated in students' learning histories [9, 3, 19]. In contrast, our work focuses on modelling these interdependencies directly.

Of particular relevance are factored MDPs applied to skill acquisition introduced by [11]. While factored MDPs account for interdependencies amongst skills, decentralized policy learning is infeasible as policies must consider the joint state space. Our work leverages the advantage of decentralized policy learning provided by RMABs and introduces a novel decentralized learning approach that exploits interdependencies between arms.

Complementary to RL methods in education is the utilization of knowledge graphs to uncover relationships between learning content [9]. Existing research primarily focuses on establishing these relationships through data-driven methods (e.g. [7, 23]) often leveraging student-activity logs. In this work, we complement such research by presenting an approach where bandit methods can effectively operate with knowledge graphs derived by such methods.

3 Model

In this section, we introduce the Restless Multi-Armed Bandits for Education (EdNetRMABs). It is important to note that while we specifically apply EdNetRMABs to the education setting, the framework can be seamlessly translated to other scenarios where modeling the effects of active actions within a network is critical. For ease of access, a table of notations is provided in Table 2.

In education, a teacher recommends learning content, or items, to maximize student education, often with content from online platforms. Items are grouped by topics, such as 'Geometry," where exposure to one piece of content can enhance knowledge across others in the same group. This cumulative learning effect which we refer to as 'network effects", implies that exposure to an item is likely to positively impact the student's success on items within the same group. A successful teacher accurately estimates a student's knowledge state over repeated interactions, leveraging these network effects to promote both breadth and depth of understanding through recommendations.

3.1 EdNetRMABs

The RMAB model tasks an agent with selecting k arms from N arms, constrained by a limit on the number of arms that can be pulled at each time step. The objective is to find a policy that maximizes the total expected discounted reward, assuming that the state of each arm evolves independently according to an underlying MDP.

The EdNetRMABs model extends RMABs by allowing for active actions to propagate to other arms dependent on the current arm when it is being pulled, thus relaxing the assumption of independent arms. This is operationalized by organising the arms in a network, and pulling of an arm results in changes for its neighbors, or members in the same group.

When applied to education setting, the EdNetRMABs is formalized as follows:

Arms Each arm, denoted as i ∈ 1 , ..., N , signifies an item. In the context of this networked environment, each arm belongs to a group ϕ ∈ { 1 , ..., L } representing the overarching topic that encompasses related items. It's important to note that arm membership is not mutually exclusive, allowing arms to be part of multiple groups. This flexibility enables a more nuanced modeling of interdependencies among educational content. For instance, a question involving the calculation of the area of a triangle may span both arithmetic and geometry groups.

State space In this framework, each arm possesses a binary latent state, denoted as s i ∈ { 0 , 1 } , 133 where '0" represents an 'unlearned" state, and '1" indicates a 'learned" state. Considering all arms 134 collectively, these states serve as a representation of the student's overall knowledge state. In the 135 current work, it is assumed that the states of all arms are fully observable, providing a comprehensive 136 model of the student's understanding of the various educational concepts. 137

138

139

140

141

142

143

144

Action space To capture the network effects associated with arm pulls, we depart from the conventional RMAB framework with a binary action space A = { 0 , 1 } by introducing a pseudo-action. In this modified setup, the action space is extended to A = { 0 , 1 , 2 } , where actions 0 and 2 represent 'no-pull" and 'pull", as commonly used in bandit literature. Notably, in EdNetRMABs, a third action 1 is introduced to simulate the network effects resulting from pulling another arm within the same group. It is important to clarify that agents do not directly engage with action 1 but we employ it solely for modeling network effects, hence the term 'pseudo-action".

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

Transition function For a given arm i , let P a,i s,s ′ represent the probability of the arm transitioning from state s to s ′ under action a . It's noteworthy that, in typical real-world educational settings, the actual transition functions governing the states of the arms are often unknown and, even for the same concept, may vary among students due to differences in prior knowledge. To address this challenge, we adopt model-free approaches in this study, devising methods to compute the teacher policy without relying on explicit knowledge of these transition functions. In the following experiments, we maintain the assumption of non-zero transition probabilities, and enforce constraints that are aligned with the current domain [17]: (i) The arms are more likely to stay in the positive state than change to the negative state: P 0 0 , 1 < P 0 1 , 1 , P 1 0 , 1 < P 1 1 , 1 and P 2 0 , 1 < P 2 1 , 1 ; (ii) The arm tends to improve the latent state if more efforts is spent on that arm, i.e., it is active or semi-active: P 0 0 , 1 < P 1 0 , 1 < P 2 0 , 1 and P 0 1 , 1 < P 1 1 , 1 < P 2 1 , 1 .

With the formalization of the EdNetRMABs model provided, we now apply it to an educational context. In this scenario, the agent assumes the role of a teacher and takes actions during each time step t ∈ { 1 , ..., T } . Specifically, at each time step, the teacher recommends an item for the student to study. We represent the vector of actions taken by the teacher at time step t as a t ∈ { 0 , 1 , 2 } N . Here, arm i is considered to be active at time t if a t ( i ) = 2 and passive when a t ( i ) = 0 . When arm i is pulled, the set of arms that share the same group membership as arm i , denoted as ϕ -i under goes the pseudo-action, represented as a t ( j ) = 1 for all j ∈ ϕ -. In our framework, the teacher agent acts on exactly one arm per time step to simulate the real-world constraint that the teacher can only recommend one concept to students ( ∑ i I a t ( i )=2 = 1 , ∀ t ). Subsequent to taking action, the teacher receives s t ∈ { 0 , 1 } N , a vector reflecting the state of all arms, and reward r t = ∑ N i =1 s t ( i ) . The vector s t represents the overall knowledge state of the student. The teacher agent's goal, therefore, is to maximize the long term rewards, either discounted or averaged.

4 EduQate

Q-learning [25] is a popular reinforcement learning method that enables an agent to learn optimal actions in an environment by iteratively updating its estimate of state-action value, Q ( s, a ) , based on the rewards it receives. At each time step t , the agent takes an action a using its current estimate of Q values and current state s , thus received a reward of r ( s ) and new state s ′ . We provide an abridged introduction to Q-learning in the Appendix F.

Expanding upon Q-learning, we introduce EduQate , a tailored Q-learning approach designed for learning Whittle-index policies in EdNetRMABs. In the interaction with the environment, the agent chooses a single item, represented by arm i , to recommend to the student. In this context, the agent possesses knowledge of the group membership ϕ i of the selected arm and observes the rewards generated by activating arm i and semi-activating arms in ϕ -i . EduQate utilizes this interaction to learn the Q-values for all arms and actions.

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

Algorithm 1 Q-Learning for EdNetRMABs (EduQate)

Input:

Number of arms

N

Initialize Q i ( s, a ) ← 0 and λ i ( s ) ← 0 for each state s ∈ S and each action a ∈ { 0 , 1 , 2 } , for each arm i ∈ 1 , ..., N .

Initialize replay buffer D with capacity C .

With probability ϵ , select one arm uniformly at random. Otherwise, select arm with highest Whittle Index, i = arg max i λ i .

for arm

if in

n

n

1

, ..., N

then

=

i

̸

Set arm to passive,

n

else

Set arm

a

t

n

to active,

n

a

n

= 0

t

= 2

for

-

i

do

j

ϕ

∈

Set arms in same group as to semi-active,

i

end for end if

end for

Execute actions

a

t

Store experience

(

Sample minibatch and observe reward

t

j

a

= 1

r

and next state

t

s

s

t

a

in replay buffer

.

r

s

,

,

,

)

D

t

t

t

1

t

+

B

of Experience from replay buffer for

Experience in minibatch

Update

.

D

do

Q

n

(

s, a

B

)

Compute end for

end for

Here, λ i is the Whittle Index estimate for arm i . In essence, the Whittle Index of arm i is computed as the linear combination of the value associated with taking action on arm i over passivity and the value of associated with semi-actively engaging with members from same group, compared to passivity.

To improve the convergence of Q-learning, we incorporate Experience Replay [15]. This involves saving the teacher algorithm's previous experiences in a replay buffer and drawing mini-batches of samples from this buffer during updates to enhance convergence. In Section 4.1, we prove that EduQate will converge to the optimal policy. However, in practice, we may not have enough episodes to fully train EduQate. Therefore, we propose Experience Replay to mitigate the cold-start problem common in RL applications, a common problem where initial student interactions with sub-optimal teachers can lead to poor learning experiences [3].

The pseudo-code is provided in Algorithm 1. Similar to WIQL [5], we employ a ϵ -decay policy that facilitates exploration and learning in the early steps, and proceeds to exploit the learned Q-values in later stages.

4.1 Analysis of EduQate

In this section, we analyze EduQate closely, and show that EduQate does not alter the optimality guarantees of Q-learning under the constraint that k = 1 (Theorem 1). Our method relies on the assumption that teachers are limited to assign 1 item to the student at each time step. Theorem 2 analyzes EduQate under the conditions that k > 1 . Since our setting involves the semi-active actions, we should compute Equation 1. To reiterate, ϕ i here refers to the group that arm i belongs to, and ϕ -i is the same group but does not include arm i . If arm i is selected, then all the remaining arms in group ϕ -i should be semi-active.

λ

n

using Q-learning update in Equation 11.

using Equation 1

do

+1

for all arms

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

Theorem 1 Choosing the top arm with the largest λ value in Equation 1 is equivalent to maximizing the cumulative long-term reward.

̸

Proof. According to the approach, we select the arm according to the λ value. Assume arm i has the highest λ value, then for any arm j where j = i , we have

According to the definition of λ in Equation 1, we move the negative part to the other side, and the left side becomes:

and the right side is similar. There are three cases:

arm i and arm j are not connected, and group ϕ i and ϕ j has no overlap, i.e., ϕ i ∩ ϕ j = ∅ . We add ∑ z/ ∈ ϕ i ∧ z/ ∈ ϕ j Q ( s z , a z = 0) on both sides. This denotes the addition of Q ( s z , a z = 0) for all arm z

that are not included in the set of ϕ i or ϕ j . We have the left side:

(3)

Similarly, we do the same for the right side and thus, the equation 2 becomes

arm i and arm j are not connected, but group ϕ i and ϕ j has overlap, i.e., ϕ i ∩ ϕ j = ∅ . In this case, we add ∑ z/ ∈ ϕ i ∧ z/ ∈ ϕ j Q ( s z , a z = 0) -∑ z ∈ ϕ i ∩ ϕ j Q ( s z , a z = 0) on both sides.

̸

̸

arm i and arm j are connected, and group ϕ i and ϕ j has overlap, i.e., ϕ i ∩ ϕ j = ∅ , and { i, j } ⊂ ϕ i ∩ ϕ j . This case is similar to the previous one, we add ∑ z/ ∈ ϕ i ∧ z/ ∈ ϕ j Q ( s z , a z = 0) -∑ z ∈ ϕ i ∩ ϕ j Q ( s z , a z =

on both sides.

The detailed proof is provided in Appendix B.

□

Thus when k = 1 , selecting the top arm according to the λ value is equivalent to maximizing the cumulative long-term reward, and is guaranteed to be optimal.

Theorem 2 When k > 1 , selecting the k arms is a NP-hard problem. The non-asymptotic tight upper bound and non-asymptotic tight lower bound for getting the optimal solution are o ( C ( n, k )) and ω ( N ) , respectively.

Proof Sketch. This problem can be considered as a variant of the knapsack problem. If we disregard the influence of the shared neighbor nodes for two selected arms, then selecting arm i will not influence the future selection of arm j . In such instances, the problem of selecting the k arms is simplified to the traditional 0/1 knapsack problem, a classic NP-hard problem. Therefore, when considering the effect of shared neighbor nodes for two selected arms, this problem is at least as challenging as the 0/1 knapsack problem. □

When k > 1 , it is difficult to compute the optimal solution, we provide a heuristic greedy algorithm with the complexity of O ( (2 N -k ) ∗ k 2 ) in Section C in the appendix.

5 Experiment

In this section, we demonstrate the effectiveness of EduQate against benchmark algorithms on synthetic students and students derived from a real-world dataset, the Junyi Dataset and the OLI Statics dataset. All experiments are run on CPU only. In our experiments, we compare EduQate with the following policies:

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

EduQate

TW

WIQL

Myopic

Random

Simulated

Aunp

170

0.4

Rewards

0.3

0.2

Average

0.1

0.0

T

十

0

50

0

50

0

50

Timesteps

Threshold Whittle (TW) : This algorithm, proposed by [17], utilizes an efficient closed-form approach to compute the Whittle index, considering only the pull action as active. It operates under the assumption that transition probabilities are known and stands as the state-of-the-art in RMABs.

WIQL : This algorithm employs a Q-learning-based Whittle Index approach [5]. It learns Q-values using the pull action as the only active strategy and calculates the Whittle Index based on the acquired Q-values.

Myopic : This strategy disregards the impact of the current action on future rewards, concentrating solely on predicted immediate rewards. It selects the arm that maximizes the expected reward at the immediate time step.

Random : This strategy randomly selects arms with uniform probability, irrespective of the underlying state.

Inspired by work in healthcare settings [12, 14], we compare the policies by the Intervention Benefit ( IB ), as shown in the following equation:

where EQ represents EduQate, and Random represents a policy where the arms are selected at random. Prior work in educational settings has demonstrated that random policies can yield robust learning outcomes through spaced repetition [9, 10]. Therefore, to establish efficacy, successful algorithms must demonstrate superiority over random policies. Our chosen metric, IB , effectively compares the extent to which a challenger algorithm π outperforms a random policy in comparison to our algorithm.

5.1 Experiment setup

In all experiments, we commence by initializing all arms in state 0 and permit the teacher algorithms to engage with the student for a total of 50 actions, pulling exactly 1 arm (i.e. k = 1 ) at each time step. Following the completion of these actions, the episode concludes, and the student state is reset. This process is iterated across 800 episodes, for a total of 30 seeds. The datasets used in our experiment are described below:

Synthetic dataset. Given the domain-motivated constraints on the transition functions highlighted in Section 3.1, we create a simulator based on N = 50 , S ∈ { 0 , 1 } , N topics = 20 . We randomly assign arms to topic groups, and allow arms to be assigned to be more than one topic. Under this method, number of arms under each group may not be equal. For each trial, a new transition matrix is generated to simulate distinct student scenarios.

Junyi dataset. The Junyi dataset [7] is an extensive dataset collected from the Junyi Academy 1 , an eLearning platform established in 2012 on the basis of the open-source code released by Khan

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

Academy. In this dataset, there are nearly 26 million student-exercise interactions across 250 000 students in its mathematics curriculum. For this experiment, we selected the top 100 exercises with the most student interactions to create our student models. Using our method to generate groups, the resultant EdNetRMAB has N = 100 and N topics = 21 .

OLI Statics dataset. The OLI Statics dataset [4] comprises student interactions with an online Engineering Statics course 2 . In this dataset, each item is assigned one or more Knowledge Components (KCs) based on the related topics. After filtering for the top 100 items with the most student interactions, the resultant EdNetRMAB includes N = 100 items and N topics = 76 distinct topics.

5.2 Creating student models

In this section, we outline the procedure for generating student models aimed at simulating the learning process. To clarify, a student model in this context is defined as a set of transition matrices for all items. These matrices are employed with EdNetRMABs to simulate the learning dynamics.

We employ various strategies to model transitions within the RMAB framework. Active transitions are determined by assessing the average success rate on a question before and after a learning intervention. Passive transitions are influenced by difficulty ratings, with more challenging questions more prone to rapid forgetting. Semi-active transitions, on the other hand, are computed as proportion of active transition, guided by similarity scores. Here, we provide an outline and the full details can be found in Appendix D.

Active Transitions. We use data on students' correct response rate after interacting with an item to create the transition matrix for action 2, based on the change in correctness rates before and after a learning intervention.

Passive Transitions. To construct passive transitions for items, we use relative difficulty scores to determine transitions based on difficulty levels. We assume that higher difficulty correlates with a greater likelihood of forgetting, resulting in higher failure rates. Specifically, higher difficulty values correspond to higher P 0 1 , 0 values, indicating a greater likelihood of forgetting. The transition matrix for the passive action a = 0 is then randomly generated, with values influenced by difficulty levels.

Semi-active Transitions. To derive semi-active transitions, we use similarity scores between exercises from the Junyi dataset. We first normalize these scores to the range [0 , 1] . Then, for any chosen arm, we compute its transition matrix under the semi-active action a = 1 as a proportion of its active action transitions, P 1 0 , 1 = σ ( P 2 0 , 1 ) , where σ signifies the similarity proportion.

The arm's transition matrix for the semi-active action varies due to different similarity scores between pairs in the same group. To address this, we use the average similarity score to determine the proportion. Since the OLI dataset does not contain similarity ratings, we assume a constant similarity rating of σ = 0 . 8 for all pairs.

6 Results

The experimental results for the synthetic, Junyi, and OLI datasets are shown in Table 1. We report the average intervention benefit IB and final episode rewards from thirty independent runs for five algorithms: EduQate, TW, WIQL, Myopic, and Random. EduQate consistently outperforms the other policies across all datasets, demonstrating higher intervention benefits and average rewards.

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

Synthetic Network

N

= 100

,

N

topics

= 20

Junyi network, abridged to

N

topics

= 7

for brevity.

OLI network,

N

= 100

,

N

topics

= 76

.

In terms of IB , we note that all challenger policies do not exceed 50%, indicating two key points. First, as noted in prior works [9], our results confirm that random policies in educational settings are robust and difficult to surpass, even when algorithms are equipped with knowledge of the learning dynamics. Second, our interdependency-aware EduQate performs well over random policies and other algorithms, highlighting the importance of considering network effects and interdependencies in EdNetRMABs.

Notably, WIQL, which relies solely on Q-learning for active and passive actions, performs worse than a random policy, likely due to misattributing positive network effects to passive actions. Despite having access to the transition matrix, TW does not perform as well as the interdependency-aware EduQate. While it has demonstrated effectiveness in traditional RMABs, TW weaknesses become evident in the current setting, where pulling an arm has wider implications to other arms. Overall, EduQate has demonstrated robust and effective performance in maximizing rewards across different datasets. Figure 1 shows the average rewards obtained in the final episode for each algorithm.

Figure 2 provides visualizations of the networks generated from synthetic students and mined from real-world datasets. The synthetic dataset produces networks with distinct isolated groups, contrasting with the more intricate and interconnected networks from the Junyi and OLI datasets, reflecting real-world complexities. Despite these differing topologies and levels of interdependency, EduQate performs well under all network setups. In Appendix E.1, we explore the effects of different network topologies by varying the number of topics while limiting the membership of each item. We find that as network interdependencies are reduced, the network effects diminish, and such EdNetRMABs can be approximated to traditional RMABs with independent arms. Under these conditions, our algorithm does not perform as well as other baseline policies.

Finally, an ablation study detailed in Appendix E.2 examines the effectiveness of the replay buffer in EduQate. The study shows that the replay buffer helps overcome the cold-start problem, where initial learning episodes provide sub-optimal experiences for students [3].

7 Conclusion and Limitations

In this paper, we introduced EdNetRMABs to the education setting, a variant of MAB designed to model interdependencies in educational content. We also proposed EduQate, a novel Whittle-based learning algorithm tailored for EdNetRMABs. Unlike other Whittle-based algorithms, EduQate computes an optimal policy without requiring knowledge of the transition matrix, while still accounting for the network effects of pulling an arm. We demonstrated the guaranteed optimality of a policy trained under EduQate and showcased its effectiveness on synthetic and real-world datasets, each with its own characteristic.

Our work assumes that student knowledge states are fully observable and available at all times, which 342 is a limitation. Despite this, we believe our work is significant and can inspire further research to 343 improve efficiencies in education. For future work, we aim to extend EduQate to handle partially 344 observable states and address the cold-start problem in education systems by minimizing the initial 345 exploratory phase. 346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

References

Richard C Atkinson. Ingredients for a theory of instruction. American Psychologist , 27(10): 921, 1972.

Aqil Zainal Azhar, Avi Segal, and Kobi Gal. Optimizing representations and policies for question sequencing using reinforcement learning. International Educational Data Mining Society , 2022.

Jonathan Bassen, Bharathan Balaji, Michael Schaarschmidt, Candace Thille, Jay Painter, Dawn Zimmaro, Alex Games, Ethan Fast, and John C Mitchell. Reinforcement learning for the adaptive scheduling of educational activities. In Proceedings of the 2020 CHI conference on human factors in computing systems , pages 1-12, 2020.

Norman Bier. Oli engineering statics - fall 2011 (114 students), 2011. URL https:// pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=590 .

Arpita Biswas, Gaurav Aggarwal, Pradeep Varakantham, and Milind Tambe. Learn to intervene: An adaptive learning policy for restless bandits in application to preventive healthcare. arXiv preprint arXiv:2105.07965 , 2021.

Colton Botta, Avi Segal, and Kobi Gal. Sequencing educational content using diversity aware bandits. 2023.

Haw-Shiuan Chang, Hwai-Jung Hsu, and Kuan-Ta Chen. Modeling exercise relationships in e-learning: A unified approach. In EDM , pages 532-535, 2015.

Shayan Doroudi, Vincent Aleven, and Emma Brunskill. Robust evaluation matrix: Towards a more principled offline exploration of instructional policies. In Proceedings of the fourth (2017) ACM conference on learning@ scale , pages 3-12, 2017.

Shayan Doroudi, Vincent Aleven, and Emma Brunskill. Where's the reward? a review of reinforcement learning for instructional sequencing. International Journal of Artificial Intelligence in Education , 29:568-620, 2019.

Hermann Ebbinghaus. Über das gedächtnis: untersuchungen zur experimentellen psychologie . Duncker & Humblot, 1885.

Derek Green, Thomas Walsh, Paul Cohen, and Yu-Han Chang. Learning a skill-teaching 374 curriculum with dynamic bayes nets. In Proceedings of the AAAI Conference on Artificial 375 Intelligence , volume 25, pages 1648-1654, 2011. 376

Christine Herlihy and John P. Dickerson. Networked restless bandits with positive externalities, 377 2022. 378

379

380

Andrew S Lan and Richard G Baraniuk. A contextual bandits framework for personalized learning action selection. In EDM , pages 424-429, 2016.

381

382

383

384

385

386

387

388

Dexun Li and Pradeep Varakantham. Avoiding starvation of arms in restless multi-armed bandits. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , pages 1303-1311, 2023.

Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning , 8:293-321, 1992.

Keqin Liu and Qing Zhao. Indexability of restless bandit problems and optimality of whittle index for dynamic multichannel access. IEEE Transactions on Information Theory , 56(11): 5547-5567, 2010.

Aditya Mate, Jackson A Killian, Haifeng Xu, Andrew Perrault, and Milind Tambe. Collapsing 389 bandits and their application to public health interventions. arXiv preprint arXiv:2007.04432 , 390 2020. 391

Christos H Papadimitriou and John N Tsitsiklis. The complexity of optimal queueing network 392 control. In Proceedings of IEEE 9th Annual Conference on Structure in Complexity Theory , 393 pages 318-322. IEEE, 1994. 394

395

396

397

Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. Deep knowledge tracing. Advances in neural information processing systems , 28, 2015.

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

Yundi Qian, Chao Zhang, Bhaskar Krishnamachari, and Milind Tambe. Restless poachers: Handling exploration-exploitation tradeoffs in security domains. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems , pages 123-131, 2016.

Avi Segal, Yossi Ben David, Joseph Jay Williams, Kobi Gal, and Yaar Shalom. Combining difficulty ranking with multi-armed bandits to sequence educational content. In Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27-30, 2018, Proceedings, Part II 19 , pages 317-321. Springer, 2018.

Shitian Shen, Markel Sanz Ausin, Behrooz Mostafavi, and Min Chi. Improving learning & reducing time: A constrained action-based reinforcement learning approach. In Proceedings of the 26th conference on user modeling, adaptation and personalization , pages 43-51, 2018.

Anni Siren and Vassilios Tzerpos. Automatic learning path creation using oer: a systematic literature mapping. IEEE Transactions on Learning Technologies , 2022.

Utkarsh Upadhyay, Abir De, and Manuel Gomez Rodriguez. Deep reinforcement learning of marked temporal point processes. Advances in Neural Information Processing Systems , 31, 2018.

Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning , 8(3):279-292, 1992.

Peter Whittle. Restless bandits: Activity allocation in a changing world. Journal of applied probability , pages 287-298, 1988.

Appendix/Supplementary Materials 416

A Table of Notations 417

B Proof for the theorem

We rewrite the theorem here for ease of explanation.

Theorem 3 Choose top arms according to the λ value in Equation 1 is equivalent to maximize the cumulative long-term reward.

̸

Proof. According to the approach, we select the arm according to the λ value. Assume arm i has the highest λ value, then for any arm j , where i = j , we have

418

419

420

421

422

423

There are two cases: 424

arm i and arm j are not connected, and group ϕ i and ϕ j has no overlap, i.e., ϕ i ∩ ϕ j = ∅ . We 425 add ∑ z/ ∈ ϕ i ∧ z/ ∈ ϕ j Q ( s z , a z = 0) on both sides, we can have the left side: 426

(6)

Similarly, the right side becomes 427

Thus, the equation 2 becomes 428

̸

· arm i and arm j are not connected, but group ϕ i and ϕ j has overlap, i.e., ϕ i ∩ ϕ j = ∅ . In this 429 case, we add ∑ z/ ∈ ϕ i ∧ z/ ∈ ϕ j Q ( s z , a z = 0) -∑ z ∈ ϕ i ∩ ϕ j Q ( s z , a z = 0) on both sides, we can have the 430

left side: 431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

Similarly, the right side becomes

̸

arm i and arm j are connected, and group ϕ i and ϕ j has overlap, i.e., ϕ i ∩ ϕ j = ∅ , and { i, j } ⊂ ϕ i ∩ ϕ j . This case is similar to the previous one, we add ∑ z/ ∈ ϕ i ∧ z/ ∈ ϕ j Q ( s z , a z = 0) -∑ z ∈ ϕ i ∩ ϕ j Q ( s z , a z = 0) on both sides, we can have the left side: Q ( s , a = I i ) and the right side Q ( s , a = I j ) .

□

We show that, using Theorem 1, selecting the top arms according to the λ value is guaranteed to maximize the cumulative long-term reward, thus proving it to be optimal.

̸

However when it comes to the case where k > 1 , selecting the top k arms according to the λ value is not guaranteed to be optimal. Let the Φ denote the set of arms that are selected, i.e., a i = 2 if i ∈ Φ . Because once the arm i is added to the selected arm set Φ , the benefit of selecting arm j will also be influenced if the arm j has the shared connected neighbor arms with arm i , i.e., ϕ i ∩ ϕ j = ∅ . To this end, finding the optimal solution is difficult, as we need to list all the possible solution sets. The non-asymptotic tight upper bound and non-asymptotic tight lower bound for getting the optimal solution are o ( C ( n, k )) and ω ( N ) , respectively.

̸

We provide the proof for Theorem 2: Proof. When considering the influence of the shared neighbor nodes for two selected arms, then selecting arm i will influence the future benefit of selecting arm j if arm i and arm j have the overlapped neighbor nodes, i.e., ϕ i ∩ ϕ j = ∅ . This is because the calculation of λ j , as some arms z ∈ ϕ i ∩ ϕ j already receive the semi-active action a = 1 due to the selection of arm i , the subsequent selection of arm j would not double introduce the benefit from those arms z who already included in ϕ i . However, if the top k arms ranked according to their λ value do not have any overlaps in their connected neighbor nodes, i.e, ϕ i ∩ ϕ j = ∅ for ∀ i, j , where arm i and arm j are top k arms according to λ value. We can directly add those top k arms to the action set Φ , and the solution is guaranteed to be optimal. Then we have the non-asymptotic tight lower bound for getting the optimal solution which is ω ( N ) . Otherwise, if the top k arms ranked according to their λ value have any overlaps in their connected neighbor nodes, to get the optimal solutions, we need to list all possible combinations of the k arms, which have the C ( n, k ) cases, and computing the corresponding sum of the λ value. In this case, we can derive that the non-asymptotic tight upper bound for getting the optimal solution is o ( C ( n, k )) . □

C Greedy algorithm when k > 1

When k > 1 , it is difficult to compute the optimal solution as we might list all possible solutions, and the complexity is O ( C ( n, k )) , Thus we provide a heuristic greedy algorithm to find the near-optimal solutions. The process to decide the selected arm set Φ is as follows:

We first compute the independent λ value for each arm i , where i ∈ { 1 , . . . , N } , where λ i = Q ( s i , a i = 1) -Q ( s i , a i = 0) + ∑ j ∈ ϕ -i ( Q ( s i , a i = 2) -Q ( s i , a i = 0)) ;

We add the arm with the top λ value to the set Φ ;

We recompute the λ value for the each arm, note that we will remove Q ( s j , a j ) in the λ equation if j ∈ Φ or j ∈ ϕ j for ∀ i ∈ Φ ;

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

we add the arm with the top λ value to the set Φ , and repeat the step 3 and 4 until we add k arms to set Φ .

The intuition of such a heuristic greedy algorithm is to add the arm that maximizes the marginal gain to the action. And the complexity for the greedy algorithm is O ( (2 N -k ) ∗ k 2 ) .

D Generating Student Models from Junyi and OLI Dataset

In this section, we describe the features in Junyi and OLI dataset which we use in developing the transition matrices.

The datasets contain the following features which we use in various aspects to generate the student models and the network:

Topic & Knowledge Component Classification: Items are classified into topics (Junyi) or KCs (OLI). This classification is employed to group items and establish the initial network.

Similarity: The Junyi dataset offers expert ratings for exercise similarity, enabling a nuanced approach to form richer group memberships. High similarity scores group exercises together, irrespective of topic tags.

Difficulty: The Junyi dataset provides expert ratings to determine the relative difficulty of exercise pairs. In the OLI dataset, we use the overall correct response rate as a measure of difficulty.

Rate of Correctness: By analyzing student-exercise interactions, we calculate the frequency of correct answers for each question, offering insights into the improvement of knowledge over time.

D.1 Active Transitions

Junyi Dataset The Junyi dataset contains earned_proficiency feature which indicates if the student has achieved mastery of the topic based on Khan Academy's algorithm 3 . Thus, we take the number of attempts before earned_proficiency=1 as P 2 0 , 1 , and the errors made during mastery as P 2 1 , 0 .

OLI Dataset We possess records of students' accuracy on quiz questions after studying specific topics. To derive the transition matrix for the student with the corresponding action 2, we utilize the change in correctness rate before and after a learning intervention.

Given that proportion of correct attempts at time t as a t , then a t +1 = P 2 0 , 1 (1 -a t ) + P 2 1 , 1 ( a t ) . We use a linear regressor to estimate the respective P 2 , constraining it to produce positive values and clipping the values to 0 . 99 when required.

D.2 Passive Transitions

To construct passive transitions for exercises, we utilize relative difficulty scores to determine transitions based on difficulty levels. We operate under the assumption that the difficulty of an exercise is linked to its likelihood of being forgotten, thereby resulting in a higher failure rate. More precisely, higher difficulty values of an exercise correspond to higher P 0 1 , 0 values, indicating a greater likelihood of forgetting. The transition matrix for the passive action a = 0 is then randomly generated, with the values influenced by the difficulty levels.

D.3 Semi-active Transitions

To derive semi-active transitions, the Junyi dataset contains similarity scores between two distinct exercises, quantifying their similarity on a 9-point Likert scale. Once the transition matrices are computed under the active action a = 2 for all arms, we proceed to calculate the transition matrix

for the semi-active action a = 1 . This involves normalizing the similarity scores to the range [0 , 1] , 512 denoted as σ . For any chosen arm/topic, we can then compute its neighbor's transition matrix under 513 the semi-active action a = 1 with P 1 0 , 1 = σ ( P 2 0 , 1 ) , where σ signifies the similarity proportion. It is 514 worth noting that an arm's transition matrix for the semi-active action varies due to different neighbors 515 being selected - different neighbors correspond to different similarity scores. 516

To address this, we can store the transition matrix of semi-active actions for different neighbor 517 selection scenarios, preserving the flexibility of our algorithm. In this work, for simplicity, we opt 518 not to distinguish the impact of different neighbors being selected. Instead, we calculate the average 519 similarity for all arms in a group average them, and use the resultant average as σ . 520

521

522

For the OLI Statics dataset, we use a constant value of σ = 0 . 8 since there are no similarity scores available.

523

524

E Additional Experiment Results and Discussion

E.1 Comparing Different Network Setups

EduQate (witout Replay Buffer)

TW

WQL

Myopic

Random

No. of topics = 20

000

Rewards

0.25

0.20

Average

0.15

0.10

T

0.05

0.00

0

10

20

30

40

50

No. of topics = 30

0.30

0.25

Rewards

0.20

0.15

0.10

0.05

000

0

10

20

30

40

50

No. of topics = 40

0.25

一

Rewards

0.20

0.15

Average

0.10

900

一

0.00-

0

10

20

30

40

50

Timestep

We present the results for different network setups in Table 3. We note that as the number of topics 525 approach the number of arms (i.e. N topics = { 30 , 40 } , all algorithms perform in a highly unstable 526 manner, as reflected in the standard deviations presented. We emphasizes here that the performance 527 of EduQate is dependent on the quality of the network it is working on, and tends to thrive in more 528 complex, yet realistic scenarios, such as the Junyi dataset presented in Figure 2. We present an 529 example of a graph generated when N topics = 40 in Figure 4, where we notice that many arms do 530 not belong to a group. Under this network, the EdNetRMAB can be approximated to a traditional 531 RMAB, where the arms are independent of each other.

CCCCCCCCO

EduQate

Simulated

OLI

0.4

Rewards

0.3

0.2

Average

0.1

0.0

十

0

400

800

0

400

800

0

400

800

Episodes

Episodes

Episodes

E.2 Ablation of Replay Buffer 533

We investigate the importance of the Experience Replay buffer in EduQate, as shown in Figure 5 and 534 Table 4. For the Simulated and Junyi datasets, EduQate without Experience Replay (EduQate-) does 535 not achieve the performance levels of the full EduQate algorithm within 800 episodes, highlighting the 536 importance of methods that aid Q-learning convergence. In real-world applications, slow convergence 537 can result in students experiencing a curriculum similar to a random policy, leading to sub-optimal 538 learning experiences during the early stages. This issue is known as the cold-start problem [3]. 539 Future work in EdNetRMABs should explore methods to overcome cold-start problems and improve 540 convergence in Q-learning-based methods. 541

542

F Q-Learning

Q-learning [25] is a popular reinforcement learning method that enables an agent to learn optimal 543 actions in an environment by iteratively updating its estimate of state-action value, Q ( s, a ) , based on 544 the rewards it receives. The objective, therefore, to learn Q ∗ ( s, a ) for each state-action pair of an 545 MDP, given by: 546

where V ∗ ( s ′ ) is the optimal expected value of a state, is given by: 547

Q-learning estimates Q ∗ through repeated interactions with the environment. At each time step t , 548 the agent takes an action a using its current estimate of Q values and current state s , thus received a 549 reward of r ( s ) and new state s ′ . Q-learning then updates the current estimate using the following: 550

where α ∈ [0 , 1] is the learning rate that controls updates, and γ is the discount on future rewards 551 associated with the MDP. 552

G Experiment Details and Hyperparameters 553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

H NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We summarize our contributions and provide the scope of the paper in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations were discussed in the final section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proofs are provided in Appendix 4.1.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided inappendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experriment details are provided in both the main body and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code and the transition matrices are provided as supplementary materials. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Relevant details are provided in the main body, as well as the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In our experiments, we report and display the standard error across all seeds. Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The current paper only requires CPU-level of compute and is mentioned in the Experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All datasets used were anonymized by the respective authors.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The current work has positive implications for applied machine learning in education settings, and is discussed in the Introduction section. As far as we can see, we don't think there are negative impacts for education.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The current paper does not release any new assets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Code [17] and datasets [7, 4] were appropriately cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes a variant of Multi-Armed Bandits (MAB) named EdNetRMABs that model the interdependencies between learning contents to meet the real-world education scenarios. Subsequently, the authors introduce EduQate, an interdependency-aware Q-learning algorithm to optimize content recommendation given the EdNetRMABs. The paper demonstrates the theoretical and empirical effectiveness of EduQate with the experimental results of synthetic and real-world data.

STRENGTHS:
1. This work uses RMABs in education to model the learning process with interdependent educational content as the different knowledge concepts have relevance.
2. The paper introduces EduQate employs Q-learning to make decisions on arm selection that do not require knowledge of the transition matrix to compute an optimal policy and provides related theoretical analysis.

WEAKNESSES:
1. As this work considers interdependency awareness in content recommendation in educational scenarios, the generated group in the experiment is not clear. The authors need to clarify how to capture relevance between the selected exercises of each topic in different datasets. Furthermore, does the different number of topics influence the experimental results?
2. The definition of state space representing a student's knowledge states a binary value. However, in real-world scenarios, the knowledge states are multi-level. A fine-grained knowledge state estimation result is essential for the content recommendation in adaptive learning.
3. The paper lacks the details of EdNetRMABs. Some illustrations would help to understand.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces **EduQate**, a system that generates adaptive educational curricula using restless multi-armed bandits (RMABs). This method aims to efficiently achieve mastery across multiple interdependent educational contents. Unlike traditional methods that assume learning contents are independent, EduQate acknowledges and leverages the interdependencies between different educational concepts.\n\nThe paper addresses key challenges in modeling and optimizing the learning process in educational environments where the learning of different topics is interconnected. By considering these interdependencies, the proposed EduQate system ensures more accurate and effective personalized learning.\n\nSTRENGTHS:\n**Originality** The paper introduces the concept of EdNetRMABs, a model that considers the interdependencies among learning contents in educational settings. This approach contrasts with existing methods that typically assume independence among different educational topics. By employing the Whittle index and Q-learning to develop the EduQate algorithm, the paper further demonstrates originality in adapting and integrating these techniques to address the specific challenges of personalized education.\n\n**Quality** The authors provide clear definitions and a well-structured EdNetRMABs model. The optimality guarantees of the EduQate algorithm are well-explained and supported through rigorous proofs. Additionally, the empirical results are compelling, demonstrating the effectiveness of the proposed method over benchmark strategies using both synthetic and real data.\n\n**Clarity** The paper is clearly written and well-organized, making complex concepts easier to understand. The authors systematically introduce the problem, the proposed solution, and both theoretical and empirical validations. The use of figures helps to elucidate the model and results. However, the clarity could be further enhanced by including more detailed explanations of the algorithm's implementation and practical applications.\n\n**Significance** The significance of the paper lies in its potential impact on the field of educational technology. By addressing the interdependencies among learning contents, EduQate offers a more accurate and efficient approach to personalized education.\n\nWEAKNESSES:\n1. **Clarity of Explanations**: Due to the limited length, many explanations in the main text are not very clear. For instance, the survival and design of the student model and the content and information of the datasets are not thoroughly explained. For example, there is a dataset mentioned later that lacks similarity content, which should have been noted when introducing the dataset.\n\n2. **Inconsistency in Notation**: In Section 4.1, \"Analysis of EduQate,\" the variable k in the second line has a different font from the k mentioned later. Additionally, the content related to k mentioned earlier is too far from this section, making it difficult for readers to understand and potentially leading to misunderstandings.\n\n3. **Simplified Modeling**: The model uses only one pseudo-state, where if any content within a group is learned, all unlearned content in the group is marked as pseudo-state. This approach seems overly simplified. For example, in Case 1, if only one piece of content in a group is learned, all other content is marked as pseudo-state. In Case 2, if many pieces of content in a group are learned, the remaining content is also marked as pseudo-state. Although both cases result in a pseudo-state, their actual significance differs, and it seems the authors did not consider this distinction.\n\n4. **Bidirectional Relationships**: The relationships between knowledge points in the paper are bidirectional, with only grouping relationships. However, in practice, many relationships between knowledge points within the same group are unidirectional. Recommending subsequent courses without recommending prerequisite courses first can lead to students needing to self-study prerequisite courses when learning subsequent courses, increasing their learning burden. The modeling in the paper does not seem to account for such unidirectional structures, instead using groups to link knowledge points, which could lead to this reverse learning issue.\n\n5. **User Experience**: The discussion on user experience for educators and students is insufficient. Including qualitative and quantitative feedback from pilot implementations would be useful. The authors should conduct user research to gather insights on the intuitiveness and user-friendliness of the system for the target users. This could include surveys, interviews, and usability testing, focusing on ease of use, effectiveness, and areas needing improvement.\n\n6. **Scalability of the EduQate Algorithm**: Although the paper mentions the efficiency of the EduQate algorithm, it lacks an in-depth exploration of its scalability in large-scale educational environments. Detailed performance benchmarking of the algorithm, including analysis of computation time and resource utilization, would be beneficial. The authors should provide a comprehensive performance analysis of the algorithm as data scale and complexity increase, including potential optimization techniques for handling large datasets.\n\nQUESTIONS:\n1. **Why not use Deep Q-learning?**: The method used in the paper seems to have poor scalability and may only be suitable for relatively fixed knowledge systems. There is a lack of discussion on its scalability in large-scale educational environments.\n\n2. **Sample Sizes in Datasets**: How were the sample sizes for the datasets set? In Junyi and OLI Statics, the quantity N is 100, but the number of topics varies. Is this N an empirical value or does it represent all the data in the original datasets? What is the actual N in this task, i.e., how many knowledge points should a model handle in practice? These aspects are not discussed in the main text, which is confusing.\n\n3. **E(BI) Result of 100**: The paper reports an E(BI) result of 100 for the proposed method, but it is unclear what this result signifies compared to the random method. The paper also states that other results did not exceed 50, implying that these results are worse than random. Could you clarify why not exceeding 50 indicates worse performance than the random method and explain the significance of the E(BI) result of 100 in this context?\n\nLIMITATIONS:\nThe authors have explicitly acknowledged the limitations of their work, providing explanations for these constraints and discussing their impact on the research. The paper does not pose any negative social impact, as it aims to improve educational practices through a more personalized approach to learning.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes EduQate, an innovative framework that leverages EdNetRMABs to achieve interdependence among knowledge points. By using Q-learning, EduQate implements optimal strategies for personalized learning, offering optimality guarantees without needing explicit knowledge of transition functions governing student learning states. This approach dynamically adapts educational content to individual progress, optimizing learning experiences and outcomes. The effectiveness of EduQate is validated using three real-world datasets, demonstrating its capability to identify and implement the most effective teaching strategies.\n\nSTRENGTHS:\nOverall, this article provides valuable insights and has a good overall quality. It has a certain impact on the AI education domain. The proposed approach, EduQate, adapts the Q-learning method of the multi-armed bandit model and introduces active and passive arms to establish correlations between knowledge points, which demonstrates a level of innovation. The article ensures the validity of the approach at the theoretical level, making it more persuasive. In terms of narrative, the article maintains logical coherence and is comprehensible. The discussed topics are indeed important in the field of education.\n\nWEAKNESSES:\nThe formatting of the article is messy, and the images are not very clear. The layout on the sixth page is problematic, with formulas and text mixed together, making it confusing. The images on the seventh page and some of the appendix images are blurred. I suggest using a different format to redraw them.\n\nThere are too few baselines in the experimental section, and there are very few validation experiments. Merely presenting outstanding performance in one experimental metric is insufficient to demonstrate the superiority of your method. It would be beneficial to include some reinforcement learning baselines for comparison. Other reinforcement learning methods, such as the Bayesian network approach mentioned in the citations, can also accomplish the task. Therefore, the necessity of your method is unclear in the paper.\n\nYour method does not consider the specific circumstances of practical problems. It raises fairness concerns. Ensuring that all knowledge points are fairly selected in a specific educational context is a crucial issue. Although the article presents a good framework, I believe it is not feasible to use it.\n\nQUESTIONS:\nYour claim seems to have ambiguity. In equation 5 of the proof, the expression for \n\n$Q(s_i,a_i=1)-Q(s_i,a_i=0)+\\sum_{j∈ϕ_i^-}(Q(s_j,a_j=1)-Q(s_j,a_j=0))$\n\nHowever, in equation 1, the expression for λ_i is given as:\n\n$Q(s_i,a_i=2)-Q(s_i,a_i=0)+\\sum_{j∈ϕ_i^-}(Q(s_j,a_j=1)-Q(s_j,a_j=0))$\n\nDid I miss something?\n\nRegarding equation 9, I am unable to comprehend how $Q(s,a=I_i)$ is derived from the previous step without further information.\n\nLIMITATIONS:\nAs mentioned above, there are many factors to consider in the specific field of education. These factors include fairness, mapping each question's knowledge points to the ARMs mentioned above, and the presence of cold-start problems, among others. Time is also a crucial factor to consider during real-time usage, although the paper does not mention specific algorithmic time information in practice.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a solution to generate personalized learning curricula in educational settings, focusing on the challenge of accounting for interdependencies between learning topics. It argues that existing approaches, often based on the Restless Multi-Armed Bandit (RMAB) framework, fall short by assuming independence of learning content which is unrealistic in educational settings. To address this the paper introduces a new model Restless Multi-armed Bandits for Education (EdNetRMABS) enabling the capture of relationships between independent learning items. Building upon this model, the authors propose a new algorithm called EduQate, which leverages Q-learning and the Whittle index to compute an interdependency-aware teacher policy for recommending educational content. Notably, EduQate doesn't require prior knowledge of the transition matrix, unlike traditional Whittle index methods.\n\nThe authors provide a theoretical analysis demonstrating the optimality of EduQate for the case of recommending a single item at each time step (k=1). While finding the optimal solution for recommending multiple items (k>1) is proven to be NP-hard, a heuristic greedy algorithm is proposed to find solutions.\n\nThrough experiments on synthetic and real-world datasets (Junyi and OLI), the paper demonstrates the superiority of EduQate over baseline policies, including Threshold Whittle (TW), WIQL, Myopic, and Random. EduQate consistently achieves higher intervention benefits and average rewards across all datasets. Further analysis reveals the effectiveness of the replay buffer in EduQate, mitigating the \"cold-start problem\" common in reinforcement learning applications.\n\nSTRENGTHS:\n1. The paper is well organized and very well written with comprehensive explanations of the model, algorithm and the experimental setup.\n2. Addressing the crucial issue of interdependency in learning content is a significant contribution. EdNetRMABs offer a realistic model for educational settings, moving beyond the simplifying assumption of independence prevalent in traditional RMAB approaches.\n3. The paper presents a rigorous theoretical analysis of EduQate, proving its optimality for the k=1 case and providing complexity bounds for the k>1 case.\n4. The authors address the cold-start problem by incorporating experience replay and perform ablations to show its effectiveness. This is a practical enhancement relevant for real-world applications.\n\nOverall, this paper presents a strong contribution to the field of adaptive learning by introducing a novel and effective approach for generating personalized curricula that account for interdependencies in learning content.  The theoretical analysis, empirical results, and focus on practical considerations make this work both insightful and impactful.\n\nWEAKNESSES:\nThe assumption of fully observable knowledge states is a significant limitation. Future work should explore extending the model and algorithm to handle partial observability, a more realistic scenario in education.\nWhile the complexity analysis is provided, further investigation on the scalability of EduQate to larger datasets and more complex networks would strengthen its practical applicability.\nThe current work primarily focuses on maximizing long-term rewards. Further analysis on balancing exploration and exploitation in EdNetRMABs could offer valuable insights for curriculum design.\n\nQUESTIONS:\nCan EduQate's recommendations be easily interpreted?  Understanding the rationale behind recommendations can enhance trust and facilitate pedagogical insights.\nAre there any potential ethical implications of using EdNetRMABs and EduQate in education?\n\nLIMITATIONS:\n1. The experiments are based on simulated students and existing datasets. Real-world classrooms involve numerous factors not accounted for in the model, such as student motivation, engagement or diverse learning styles. This may limit the practical utility of this method.\n2. The paper does not address the interpretability of EduQate's recommendations.\n3. The assumption that the student states are fully observable is a major limitation of this work. This can result in overconfident recommendations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a way to extend RMAB with Q-learning to accommodate the fact that some items/arms belong to a group. RMAB can be considered as a weakened version of contextual bandit CB but also a strong version of CB since it considers state transitions (explicitly defined on arms).\n\nSTRENGTHS:\nThis is a very solid paper with quite a few reasons to accept it:\n1. The contribution of education simulators. Can the authors comment on whether they will release their simulators publicly for other education or bandit communities to use in the future? Currently, there is a lack of high-quality education simulators that are based on real-world data. This could be a great contribution. I hope the authors make an effort to make such resources public.\n2. The technical contribution of introducing a pseudo-action that extends RMAB and WIQL.\n3. Clean writing and presentation.\n\nMAB has been used in many different real-world settings and is the main algorithm behind many learning platforms. Any innovation in this space will have a huge impact on students around the world. Unfortunately, this area is very niche and requires high technical sophistication. Unless someone can convince me that this paper's algorithm has already been published elsewhere or is fully derivative of some other work, I think it's a great paper to present at NeurIPS.\n\nI might also be a bit concerned if the authors decide not to share their code/algorithm/simulators to encourage more future research in this direction.\n\nWEAKNESSES:\n1. Can add some more ablation studies.\n2. Some clarifications (see question section).\n\nQUESTIONS:\nI have a few questions related to the technical contributions of the paper. None of these questions are reasons for rejection. I just want some clarification from the authors.\n1. Equation (1) -- it seems to me that this altered the original definition of Whittle Index by summing over pseudo-actions. Can you explain by doing so, would it change the optimality of Whittle Index as a solution to RMAB? For example, how do we know under such an extension, the Whittle Index can still get the optimal solution? Is Theorem 1 answering this question? If so, does the original Whittle Index algorithm guarantee optimality (I assume over sublinear regret?) for both k=1 and k>1 arm selection strategy? I apologize if my understanding of RMAB is inaccurate.\n2. If it's easy to run, can you add an ablation study that's under misspecification of $\\phi$? Often in education, the grouping of items/materials is not exact -- and can often be wrong. The membership function $\\phi$ has errors. I'm a bit curious about how much the quality of $\\phi$ impacts EduQate. I understand it's hard to add experiments given the tight rebuttal period, so don't worry if this is too much of an ask.\n3. Eq 4 shows IB as a metric. My question is: if both $\\pi$ and $EQ$ are worse than random -- then IB will still be positive. This is not a metric that shows \"successful algorithms must demonstrate superiority over random policies\" -- this is a metric that says we should have a policy better than the baseline, but it can be worse. Do I understand this correctly? IB is a valid metric to use. Also, why is IB 100% in Table  1? Maybe my intuition is just off, but how is 100% achieved? Can you give an example?\n\nLIMITATIONS:\nThe authors discussed limitations in a section.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-14"}