EduQate: Generating Adaptive Curricula through RMABs in Education Settings

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:42:19

Abstract: There has been significant interest in the development of personalized and adaptive educational tools that cater to a student's individual learning progress. A crucial aspect in developing such tools is in exploring how mastery can be achieved across a diverse yet related range of content in an efficient manner. While Reinforcement Learning and Multi-armed Bandits have shown promise in educational settings, existing works often assume the independence of learning content, neglecting the prevalent interdependencies between such content. In response, we introduce Education Network Restless Multi-armed Bandits (EdNetRMABs), utilizing a network to represent the relationships between interdependent arms. Subsequently, we propose EduQate, a method employing interdependency-aware Q-learning to make informed decisions on arm selection at each time step. We establish the optimality guarantee of EduQate and demonstrate its efficacy compared to baseline policies, using students modeled from both synthetic and real-world data.

Main paper: 
 introduction: 
 The COVID-19 pandemic has accelerated the adoption of educational technologies, especially on eLearning platforms. Despite abundant data and advancements in modeling student learning, effectively capturing the learning process with interdependent content remains a significant challenge [9] . The conventional rules-based approach to creating personalized learning curricula is impractical due to its labor-intensive nature and need for expert knowledge. Machine learning-based systems offer a scalable alternative, automatically generating personalized content to optimize learning [22, 24] .One possible approach to model the learning process is the Restless Multi-Armed Bandits (RMAB, [26] ), where a teacher agent selects a subset of arms (concepts) to teach each round. However, RMAB's assumption that arms are independent is unrealistic in educational settings. For example, solving a math question on the area of a triangle requires knowledge of algebra, arithmetic, and geometry. Practicing this question should enhance proficiency in all three areas. Models that ignore such interdependencies may inaccurately predict knowledge levels by assuming each exercise impacts only a single area.In response to this challenge, we introduce an interdependency-aware RMAB model to the education setting. We posit that by acknowledging and modeling the learning dynamics of interdependent content, both teachers and algorithms can strategically leverage overlapping utility to foster mastery over a broader range of topics within a curriculum. We advocate for RMABs as a fitting model for this context, as the inherent dynamics of such a model align closely with the learning process.In this study, our objective is to derive a teacher policy that effectively recommends educational content to students, accounting for interdependencies among the content to enhance overall utility (that characterizes understanding and retention of content). Our contributions are as follows:1. We introduce Restless Multi-armed Bandits for Education (EdNetRMABs), enabling the modeling of learning processes with interdependent educational content.2. We propose EduQate, a Whittle index-based heuristic algorithm that uses Q-learning to compute an inter-dependency-aware teacher policy. Unlike previous methods, EduQate does not require knowledge of the transition matrix to compute an optimal policy.3. We provide a theoretical analysis of EduQate, demonstrating guarantees of optimality. 4 . We present empirical results on simulated students and real-world datasets, showing the effectiveness of EduQate over other teacher policies.2 Related Work and Preliminaries 
restless multi-armed bandits: 
 The selection of the right time and manner for limited interventions is a problem of great practical importance across various domains, including health intervention [17, 5] , anti-poaching operations [20] , education [13, 6, 2] , etc. These problems share a common characteristic of having multiple arms in a Multi-armed Bandit (MAB) problem, representing entities such as patients, regions of a forest, or students' mastery of concepts. These arms evolve in an uncertain manner, and interventions are required to guide them from "bad" states to "good" states. The inherent challenge lies in the limited number of interventions, dictated by the limited resources (e.g., public health workers, the number of student interactions). RMAB, a generalization of MAB, offers an ideal model for representing the aforementioned problems of interest. RMAB allows non-active bandits to also undergo the Markovian state transition, effectively capturing uncertainty in arm state transitions (reflecting uncertain state evolution), actions (representing interventions), and budget constraints (illustrating limited resources).RMABs and the associated Markov Decision Processes (MDP) for each arm offer a valuable model for representing the learning process. Firstly, leveraging the MDPs associated with each arm provides the flexibility to adopt nuanced modeling of learning content, accommodating different learning curves for various content based on students' strengths and weaknesses. Secondly, the transition probabilities serve as a useful mechanism to model forgetting (through state decay due to passivity or negligence) and learning (through state transitions to the positive state from repeated practice). Considering these aspects, RMABs prove to be a beneficial framework for personalizing and generating adaptive curricula across a diverse range of students.In general, computing the optimal policy for a given set of restless arms in RMABs is recognized as a PSPACE-hard problem [18] . The Whittle index [26] provides an approach with a tractable solution that is provably optimal, especially when each arm is indexable. However, proving indexability can be challenging and often requires specification of the problem's structure, such as the optimality of threshold policies [17, 16] . Moreover, much of the research on Whittle Index policies has focused on two-action settings or requires prior knowledge of the transition matrix of the RMABs. Meeting these conditions proves challenging in the educational context, where diverse students interact with educational systems, each possessing different prior knowledge and distinct learning curves for various topics.WIQL [5] , on the other hand, employs a Q-learning-based method to estimate the Whittle Index and has demonstrated provable optimality without requiring prior knowledge of the transition matrix. We utilize WIQL as a baseline method in our subsequent experiments.In a recent investigation by [12] , RMABs were explored within a network framework, requiring the agent to manage a budget while allocating a high-cost, high-benefit resource to one arm to "unlock" potential lower-cost, intermediate-benefit resources for the arm's neighbors. The network effects emphasized in their work are triggered by an intentional, active action, enabling the agent to choose to propagate positive externalities to a selected arm's neighbors within budget constraints. In contrast, our study delves into scenarios where network effects are indirect results of an active action, and the agent lacks direct control over such effects. Thus, the challenge lies in accurately modeling these network effects and leveraging them when beneficial.In the realm of education, numerous researchers have explored optimizing the sequencing of instructional activities and content, assuming that optimal sequencing can significantly impact student learning. RL is a natural approach for making sequential decisions under uncertainty [1] . While RL has seen success in various educational applications, effectively sequencing interdependent content in a personalized and adaptive manner has yielded mixed or insignificant results compared to baseline teacher policies [11, 21, 8] . In general, these RL works focus on data-driven methods using student activity logs to estimate students' knowledge states and progress, assuming that the interdependencies between learning content are encapsulated in students' learning histories [9, 3, 19] . In contrast, our work focuses on modelling these interdependencies directly.Of particular relevance are factored MDPs applied to skill acquisition introduced by [11] . While factored MDPs account for interdependencies amongst skills, decentralized policy learning is infeasible as policies must consider the joint state space. Our work leverages the advantage of decentralized policy learning provided by RMABs and introduces a novel decentralized learning approach that exploits interdependencies between arms.Complementary to RL methods in education is the utilization of knowledge graphs to uncover relationships between learning content [9] . Existing research primarily focuses on establishing these relationships through data-driven methods (e.g. [7, 23] ) often leveraging student-activity logs. In this work, we complement such research by presenting an approach where bandit methods can effectively operate with knowledge graphs derived by such methods.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations were discussed in the final section. 
model: 
 In this section, we introduce the Restless Multi-Armed Bandits for Education (EdNetRMABs). It is important to note that while we specifically apply EdNetRMABs to the education setting, the framework can be seamlessly translated to other scenarios where modeling the effects of active actions within a network is critical. For ease of access, a table of notations is provided in Table 2 .In education, a teacher recommends learning content, or items, to maximize student education, often with content from online platforms. Items are grouped by topics, such as "Geometry," where exposure to one piece of content can enhance knowledge across others in the same group. This cumulative learning effect which we refer to as "network effects", implies that exposure to an item is likely to positively impact the student's success on items within the same group. A successful teacher accurately estimates a student's knowledge state over repeated interactions, leveraging these network effects to promote both breadth and depth of understanding through recommendations.The RMAB model tasks an agent with selecting k arms from N arms, constrained by a limit on the number of arms that can be pulled at each time step. The objective is to find a policy that maximizes the total expected discounted reward, assuming that the state of each arm evolves independently according to an underlying MDP.The EdNetRMABs model extends RMABs by allowing for active actions to propagate to other arms dependent on the current arm when it is being pulled, thus relaxing the assumption of independent arms. This is operationalized by organising the arms in a network, and pulling of an arm results in changes for its neighbors, or members in the same group.When applied to education setting, the EdNetRMABs is formalized as follows:Arms Each arm, denoted as i ∈ 1, ..., N , signifies an item. In the context of this networked environment, each arm belongs to a group ϕ ∈ {1, ..., L} representing the overarching topic that encompasses related items. It's important to note that arm membership is not mutually exclusive, allowing arms to be part of multiple groups. This flexibility enables a more nuanced modeling of interdependencies among educational content. For instance, a question involving the calculation of the area of a triangle may span both arithmetic and geometry groups.State space In this framework, each arm possesses a binary latent state, denoted as s i ∈ {0, 1}, where "0" represents an "unlearned" state, and "1" indicates a "learned" state. Considering all arms collectively, these states serve as a representation of the student's overall knowledge state. In the current work, it is assumed that the states of all arms are fully observable, providing a comprehensive model of the student's understanding of the various educational concepts.Action space To capture the network effects associated with arm pulls, we depart from the conventional RMAB framework with a binary action space A = {0, 1} by introducing a pseudo-action. In this modified setup, the action space is extended to A = {0, 1, 2}, where actions 0 and 2 represent "no-pull" and "pull", as commonly used in bandit literature. Notably, in EdNetRMABs, a third action 1 is introduced to simulate the network effects resulting from pulling another arm within the same group. It is important to clarify that agents do not directly engage with action 1 but we employ it solely for modeling network effects, hence the term "pseudo-action".Transition function For a given arm i, let P a,i s,s ′ represent the probability of the arm transitioning from state s to s ′ under action a. It's noteworthy that, in typical real-world educational settings, the actual transition functions governing the states of the arms are often unknown and, even for the same concept, may vary among students due to differences in prior knowledge. To address this challenge, we adopt model-free approaches in this study, devising methods to compute the teacher policy without relying on explicit knowledge of these transition functions. In the following experiments, we maintain the assumption of non-zero transition probabilities, and enforce constraints that are aligned with the current domain [17] : (i) The arms are more likely to stay in the positive state than change to the negative state: P 0 0,1 < P 0 1,1 , P 1 0,1 < P 1 1,1 and P 2 0,1 < P 2 1,1 ; (ii) The arm tends to improve the latent state if more efforts is spent on that arm, i.e., it is active or semi-active: P 0 0,1 < P 1 0,1 < P 2 0,1 andP 0 1,1 < P 1 1,1 < P 2 1,1 .With the formalization of the EdNetRMABs model provided, we now apply it to an educational context. In this scenario, the agent assumes the role of a teacher and takes actions during each time step t ∈ {1, ..., T }. Specifically, at each time step, the teacher recommends an item for the student to study. We represent the vector of actions taken by the teacher at time step t as a t ∈ {0, 1, 2} N . Here, arm i is considered to be active at time t if a t (i) = 2 and passive when a t (i) = 0. When arm i is pulled, the set of arms that share the same group membership as arm i, denoted as ϕ - i under goes the pseudo-action, represented as a t (j) = 1 for all j ∈ ϕ -. In our framework, the teacher agent acts on exactly one arm per time step to simulate the real-world constraint that the teacher can only recommend one concept to students ( i I a t (i)=2 = 1, ∀t ). Subsequent to taking action, the teacher receives s t ∈ {0, 1} N , a vector reflecting the state of all arms, and reward r t = N i=1 s t (i). The vector s t represents the overall knowledge state of the student. The teacher agent's goal, therefore, is to maximize the long term rewards, either discounted or averaged.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Answer: [Yes] Justification: Code and the transition matrices are provided as supplementary materials.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
eduqate: 
 Q-learning [25] is a popular reinforcement learning method that enables an agent to learn optimal actions in an environment by iteratively updating its estimate of state-action value, Q(s, a), based on the rewards it receives. At each time step t, the agent takes an action a using its current estimate of Q values and current state s, thus received a reward of r(s) and new state s ′ . We provide an abridged introduction to Q-learning in the Appendix F.Expanding upon Q-learning, we introduce EduQate, a tailored Q-learning approach designed for learning Whittle-index policies in EdNetRMABs. In the interaction with the environment, the agent chooses a single item, represented by arm i, to recommend to the student. In this context, the agent possesses knowledge of the group membership ϕ i of the selected arm and observes the rewards generated by activating arm i and semi-activating arms in ϕ - i . EduQate utilizes this interaction to learn the Q-values for all arms and actions.To adapt Q-learning to EdNetRMABs, we propose leveraging the learned Q-values to select the arm with the highest estimate of the Whittle index, defined as: EQUATIONHere, λ i is the Whittle Index estimate for arm i. In essence, the Whittle Index of arm i is computed as the linear combination of the value associated with taking action on arm i over passivity and the value of associated with semi-actively engaging with members from same group, compared to passivity.To improve the convergence of Q-learning, we incorporate Experience Replay [15] . This involves saving the teacher algorithm's previous experiences in a replay buffer and drawing mini-batches of samples from this buffer during updates to enhance convergence. In Section 4.1, we prove that EduQate will converge to the optimal policy. However, in practice, we may not have enough episodes to fully train EduQate. Therefore, we propose Experience Replay to mitigate the cold-start problem common in RL applications, a common problem where initial student interactions with sub-optimal teachers can lead to poor learning experiences [3] .The pseudo-code is provided in Algorithm 1. Similar to WIQL [5] , we employ a ϵ-decay policy that facilitates exploration and learning in the early steps, and proceeds to exploit the learned Q-values in later stages.In this section, we analyze EduQate closely, and show that EduQate does not alter the optimality guarantees of Q-learning under the constraint that k = 1 (Theorem 1). Our method relies on the assumption that teachers are limited to assign 1 item to the student at each time step. Theorem 2analyzes EduQate under the conditions that k > 1. Since our setting involves the semi-active actions, we should compute Equation 1. To reiterate, ϕ i here refers to the group that arm i belongs to, and ϕ - i is the same group but does not include arm i. If arm i is selected, then all the remaining arms in group ϕ - i should be semi-active.Theorem 1 Choosing the top arm with the largest λ value in Equation 1 is equivalent to maximizing the cumulative long-term reward.Proof. According to the approach, we select the arm according to the λ value. Assume arm i has the highest λ value, then for any arm j where j ̸ = i, we haveλ i ≥ λ j (2)According to the definition of λ in Equation 1, we move the negative part to the other side, and the left side becomes:Q(s i , a i = 1) + i∈ϕ - i (Q(s i , a i = 1)) + Q(s j , a j = 0) + j∈ϕ - j (Q(s j , a j = 0))and the right side is similar. There are three cases:• arm i and arm j are not connected, and group ϕ i and ϕ j has no overlap, i.e., ϕ i ∩ ϕ j = ∅. We add z / ∈ϕi∧z / ∈ϕj Q(s z , a z = 0) on both sides. This denotes the addition of Q(s z , a z = 0) for all arm z that are not included in the set of ϕ i or ϕ j . We have the left side:EQUATIONSimilarly, we do the same for the right side and thus, the equation 2 becomesQ(s, a = I i ) ≥ Q(s, a = I j )• arm i and arm j are not connected, but group ϕ i and ϕ j has overlap, i.e., ϕ i ∩ ϕ j ̸ = ∅. In this case, we addz / ∈ϕi∧z / ∈ϕj Q(s z , a z = 0) - z∈ϕi∩ϕj Q(s z , a z = 0) on both sides.• arm i and arm j are connected, and group ϕ i and ϕ j has overlap, i.e., ϕ i ∩ϕ j ̸ = ∅, and {i, j} ⊂ ϕ i ∩ ϕ j . This case is similar to the previous one, we addz / ∈ϕi∧z / ∈ϕj Q(s z , a z = 0) - z∈ϕi∩ϕj Q(s z , a z = 0) on both sides.The detailed proof is provided in Appendix B. □Thus when k = 1, selecting the top arm according to the λ value is equivalent to maximizing the cumulative long-term reward, and is guaranteed to be optimal.Theorem 2 When k > 1, selecting the k arms is a NP-hard problem. The non-asymptotic tight upper bound and non-asymptotic tight lower bound for getting the optimal solution are o(C(n, k))and ω(N ), respectively.Proof Sketch. This problem can be considered as a variant of the knapsack problem. If we disregard the influence of the shared neighbor nodes for two selected arms, then selecting arm i will not influence the future selection of arm j. In such instances, the problem of selecting the k arms is simplified to the traditional 0/1 knapsack problem, a classic NP-hard problem. Therefore, when considering the effect of shared neighbor nodes for two selected arms, this problem is at least as challenging as the 0/1 knapsack problem. □When k > 1, it is difficult to compute the optimal solution, we provide a heuristic greedy algorithm with the complexity of O( (2N -k) * k) in Section C in the appendix. 
experiment: 
 In this section, we demonstrate the effectiveness of EduQate against benchmark algorithms on synthetic students and students derived from a real-world dataset, the Junyi Dataset and the OLI Statics dataset. All experiments are run on CPU only. In our experiments, we compare EduQate with the following policies: • Threshold Whittle (TW): This algorithm, proposed by [17] , utilizes an efficient closed-form approach to compute the Whittle index, considering only the pull action as active. It operates under the assumption that transition probabilities are known and stands as the state-of-the-art in RMABs.• WIQL: This algorithm employs a Q-learning-based Whittle Index approach [5] . It learns Q-values using the pull action as the only active strategy and calculates the Whittle Index based on the acquired Q-values.• Myopic: This strategy disregards the impact of the current action on future rewards, concentrating solely on predicted immediate rewards. It selects the arm that maximizes the expected reward at the immediate time step.• Random: This strategy randomly selects arms with uniform probability, irrespective of the underlying state.Inspired by work in healthcare settings [12, 14] , we compare the policies by the Intervention Benefit (IB), as shown in the following equation:EQUATIONwhere EQ represents EduQate, and Random represents a policy where the arms are selected at random.Prior work in educational settings has demonstrated that random policies can yield robust learning outcomes through spaced repetition [9, 10] . Therefore, to establish efficacy, successful algorithms must demonstrate superiority over random policies. Our chosen metric, IB, effectively compares the extent to which a challenger algorithm π outperforms a random policy in comparison to our algorithm.In all experiments, we commence by initializing all arms in state 0 and permit the teacher algorithms to engage with the student for a total of 50 actions, pulling exactly 1 arm (i.e. k = 1) at each time step.Following the completion of these actions, the episode concludes, and the student state is reset. This process is iterated across 800 episodes, for a total of 30 seeds. The datasets used in our experiment are described below:Synthetic dataset. Given the domain-motivated constraints on the transition functions highlighted in Section 3.1, we create a simulator based on N = 50, S ∈ {0, 1}, N topics = 20. We randomly assign arms to topic groups, and allow arms to be assigned to be more than one topic. Under this method, number of arms under each group may not be equal. For each trial, a new transition matrix is generated to simulate distinct student scenarios.Junyi dataset. The Junyi dataset [7] is an extensive dataset collected from the Junyi Academy In this section, we outline the procedure for generating student models aimed at simulating the learning process. To clarify, a student model in this context is defined as a set of transition matrices for all items. These matrices are employed with EdNetRMABs to simulate the learning dynamics.We employ various strategies to model transitions within the RMAB framework. Active transitions are determined by assessing the average success rate on a question before and after a learning intervention. Passive transitions are influenced by difficulty ratings, with more challenging questions more prone to rapid forgetting. Semi-active transitions, on the other hand, are computed as proportion of active transition, guided by similarity scores. Here, we provide an outline and the full details can be found in Appendix D.Active Transitions. We use data on students' correct response rate after interacting with an item to create the transition matrix for action 2, based on the change in correctness rates before and after a learning intervention.Passive Transitions. To construct passive transitions for items, we use relative difficulty scores to determine transitions based on difficulty levels. We assume that higher difficulty correlates with a greater likelihood of forgetting, resulting in higher failure rates. Specifically, higher difficulty values correspond to higher P 0 1,0 values, indicating a greater likelihood of forgetting. The transition matrix for the passive action a = 0 is then randomly generated, with values influenced by difficulty levels.Semi-active Transitions. To derive semi-active transitions, we use similarity scores between exercises from the Junyi dataset. We first normalize these scores to the range [0, 1]. Then, for any chosen arm, we compute its transition matrix under the semi-active action a = 1 as a proportion of its active action transitions, P 1 0,1 = σ(P2 0,1 ), where σ signifies the similarity proportion.The arm's transition matrix for the semi-active action varies due to different similarity scores between pairs in the same group. To address this, we use the average similarity score to determine the proportion. Since the OLI dataset does not contain similarity ratings, we assume a constant similarity rating of σ = 0.8 for all pairs. 
results: 
 The experimental results for the synthetic, Junyi, and OLI datasets are shown in Table 1 . We report the average intervention benefit IB and final episode rewards from thirty independent runs for five algorithms: EduQate, TW, WIQL, Myopic, and Random. EduQate consistently outperforms the other policies across all datasets, demonstrating higher intervention benefits and average rewards. In terms of IB, we note that all challenger policies do not exceed 50%, indicating two key points.First, as noted in prior works [9] , our results confirm that random policies in educational settings are robust and difficult to surpass, even when algorithms are equipped with knowledge of the learning dynamics. Second, our interdependency-aware EduQate performs well over random policies and other algorithms, highlighting the importance of considering network effects and interdependencies in EdNetRMABs.Notably, WIQL, which relies solely on Q-learning for active and passive actions, performs worse than a random policy, likely due to misattributing positive network effects to passive actions. Despite having access to the transition matrix, TW does not perform as well as the interdependency-awareEduQate. While it has demonstrated effectiveness in traditional RMABs, TW weaknesses become evident in the current setting, where pulling an arm has wider implications to other arms. Overall, EduQate has demonstrated robust and effective performance in maximizing rewards across different datasets. Figure 1 shows the average rewards obtained in the final episode for each algorithm.Figure 2 provides visualizations of the networks generated from synthetic students and mined from real-world datasets. The synthetic dataset produces networks with distinct isolated groups, contrasting with the more intricate and interconnected networks from the Junyi and OLI datasets, reflecting real-world complexities. Despite these differing topologies and levels of interdependency, EduQate performs well under all network setups. In Appendix E.1, we explore the effects of different network topologies by varying the number of topics while limiting the membership of each item. We find that as network interdependencies are reduced, the network effects diminish, and such EdNetRMABs can be approximated to traditional RMABs with independent arms. Under these conditions, our algorithm does not perform as well as other baseline policies.Finally, an ablation study detailed in Appendix E.2 examines the effectiveness of the replay buffer in EduQate. The study shows that the replay buffer helps overcome the cold-start problem, where initial learning episodes provide sub-optimal experiences for students [3] .Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Relevant details are provided in the main body, as well as the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion and limitations: 
 In this paper, we introduced EdNetRMABs to the education setting, a variant of MAB designed to model interdependencies in educational content. We also proposed EduQate, a novel Whittle-based learning algorithm tailored for EdNetRMABs. Unlike other Whittle-based algorithms, EduQate computes an optimal policy without requiring knowledge of the transition matrix, while still accounting for the network effects of pulling an arm. We demonstrated the guaranteed optimality of a policy trained under EduQate and showcased its effectiveness on synthetic and real-world datasets, each with its own characteristic.Our work assumes that student knowledge states are fully observable and available at all times, which is a limitation. Despite this, we believe our work is significant and can inspire further research to improve efficiencies in education. For future work, we aim to extend EduQate to handle partially observable states and address the cold-start problem in education systems by minimizing the initial exploratory phase. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: In our experiments, we report and display the standard error across all seeds.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The current work has positive implications for applied machine learning in education settings, and is discussed in the Introduction section. As far as we can see, we don't think there are negative impacts for education. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The current paper does not release any new assets. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: P i,as,s ′ is the probability of transition from state s to s ′ when arm i is taking action a. Q i (s i , a i ) Q i (s i , a i ) is the state-action value function for the arm i when taking action a i with state s i .V i (s i )The value function for arm i at the state s i .We rewrite the theorem here for ease of explanation.Theorem 3 Choose top arms according to the λ value in Equation 1 is equivalent to maximize the cumulative long-term reward.Proof. According to the approach, we select the arm according to the λ value. Assume arm i has the highest λ value, then for any arm j, where i ̸ = j, we haveλi ≥ λj Q(si, ai = 1) -Q(si, ai = 0) + i∈ϕ - i (Q(si, ai = 1) -Q(si, ai = 0)) ≥ Q(sj, aj = 1) -Q(sj, aj = 0) + j∈ϕ - j (Q(sj, aj = 1) -Q(sj, aj = 0)) Q(si, ai = 1) + i∈ϕ - i (Q(si, ai = 1)) + Q(sj, aj = 0) + j∈ϕ - j (Q(sj, aj = 0)) ≥ Q(sj, aj = 1) + j∈ϕ - j (Q(sj, aj = 1)) + Q(si, ai = 0) + i∈ϕ - i (Q(si, ai = 0))(5) There are two cases:• arm i and arm j are not connected, and group ϕ i and ϕ j has no overlap, i.e., ϕ i ∩ ϕ j = ∅. We add z / ∈ϕi∧z / ∈ϕj Q(s z , a z = 0) on both sides, we can have the left side:EQUATION) Similarly, the right side becomesEQUATIONThus, the equation 2 becomesEQUATION• arm i and arm j are not connected, but group ϕ i and ϕ j has overlap, i.e., ϕ i ∩ ϕ j ̸ = ∅. In this case, we add z / ∈ϕi∧z / ∈ϕj Q(s z , a z = 0) -z∈ϕi∩ϕj Q(s z , a z = 0) on both sides, we can have the left side:EQUATIONSimilarly, the right side becomesEQUATION• arm i and arm j are connected, and group ϕ i and ϕ j has overlap, i.e., ϕ i ∩ ϕ j ̸ = ∅, and {i, j} ⊂ ϕ i ∩ ϕ j . This case is similar to the previous one, we addz / ∈ϕi∧z / ∈ϕj Q(s z , a z = 0) - z∈ϕi∩ϕj Q(s z , a z = 0)on both sides, we can have the left side: Q(s, a = I i ) and the right sideQ(s, a = I j ).We show that, using Theorem 1, selecting the top arms according to the λ value is guaranteed to maximize the cumulative long-term reward, thus proving it to be optimal.However when it comes to the case where k > 1, selecting the top k arms according to the λ value is not guaranteed to be optimal. Let the Φ denote the set of arms that are selected, i.e., a i = 2 if i ∈ Φ. Because once the arm i is added to the selected arm set Φ, the benefit of selecting arm j will also be influenced if the arm j has the shared connected neighbor arms with arm i, i.e., ϕ i ∩ ϕ j ̸ = ∅.To this end, finding the optimal solution is difficult, as we need to list all the possible solution sets.The non-asymptotic tight upper bound and non-asymptotic tight lower bound for getting the optimal solution are o(C(n, k)) and ω(N ), respectively.We provide the proof for Theorem 2: Proof. When considering the influence of the shared neighbor nodes for two selected arms, then selecting arm i will influence the future benefit of selecting arm j if arm i and arm j have the overlapped neighbor nodes, i.e., ϕ i ∩ ϕ j ̸ = ∅. This is because the calculation of λ j , as some arms z ∈ ϕ i ∩ ϕ j already receive the semi-active action a = 1 due to the selection of arm i, the subsequent selection of arm j would not double introduce the benefit from those arms z who already included in ϕ i . However, if the top k arms ranked according to their λ value do not have any overlaps in their connected neighbor nodes, i.e, ϕ i ∩ ϕ j = ∅ for ∀i, j, where arm i and arm j are top k arms according to λ value. We can directly add those top k arms to the action set Φ, and the solution is guaranteed to be optimal. Then we have the non-asymptotic tight lower bound for getting the optimal solution which is ω(N ). Otherwise, if the top k arms ranked according to their λ value have any overlaps in their connected neighbor nodes, to get the optimal solutions, we need to list all possible combinations of the k arms, which have the C(n, k) cases, and computing the corresponding sum of the λ value. In this case, we can derive that the non-asymptotic tight upper bound for getting the optimal solution is o(C(n, k)). □When k > 1, it is difficult to compute the optimal solution as we might list all possible solutions, and the complexity is O(C(n, k)), Thus we provide a heuristic greedy algorithm to find the near-optimal solutions. The process to decide the selected arm set Φ is as follows:1. We first compute the independent λ value for each arm i, where i ∈ {1, . . . , N }, whereλ i = Q(s i , a i = 1) -Q(s i , a i = 0) + j∈ϕ - i (Q(s i , a i = 2) -Q(s i , a i = 0));2. We add the arm with the top λ value to the set Φ;3. We recompute the λ value for the each arm, note that we will remove Q(s j , a j ) in the λ equation if j ∈ Φ or j ∈ ϕ j for ∀i ∈ Φ;4. we add the arm with the top λ value to the set Φ, and repeat the step 3 and 4 until we add k arms to set Φ.The intuition of such a heuristic greedy algorithm is to add the arm that maximizes the marginal gain to the action. And the complexity for the greedy algorithm is O( (2N -k) * k).In this section, we describe the features in Junyi and OLI dataset which we use in developing the transition matrices.The datasets contain the following features which we use in various aspects to generate the student models and the network:• Topic & Knowledge Component Classification: Items are classified into topics (Junyi) or KCs (OLI). This classification is employed to group items and establish the initial network.• Similarity: The Junyi dataset offers expert ratings for exercise similarity, enabling a nuanced approach to form richer group memberships. High similarity scores group exercises together, irrespective of topic tags.• Difficulty: The Junyi dataset provides expert ratings to determine the relative difficulty of exercise pairs. In the OLI dataset, we use the overall correct response rate as a measure of difficulty.• Rate of Correctness: By analyzing student-exercise interactions, we calculate the frequency of correct answers for each question, offering insights into the improvement of knowledge over time.Junyi Dataset The Junyi dataset contains earned_proficiency feature which indicates if the student has achieved mastery of the topic based on Khan Academy's algorithm 3 . Thus, we take the number of attempts before earned_proficiency=1 as P 2 0,1 , and the errors made during mastery asP 2 1,0 .OLI Dataset We possess records of students' accuracy on quiz questions after studying specific topics. To derive the transition matrix for the student with the corresponding action 2, we utilize the change in correctness rate before and after a learning intervention.Given that proportion of correct attempts at time t as a t , then a t+1 = P 2 0,1 (1 -a t ) + P 2 1,1 (a t ). We use a linear regressor to estimate the respective P 2 , constraining it to produce positive values and clipping the values to 0.99 when required.To construct passive transitions for exercises, we utilize relative difficulty scores to determine transitions based on difficulty levels. We operate under the assumption that the difficulty of an exercise is linked to its likelihood of being forgotten, thereby resulting in a higher failure rate. More precisely, higher difficulty values of an exercise correspond to higher P 0 1,0 values, indicating a greater likelihood of forgetting. The transition matrix for the passive action a = 0 is then randomly generated, with the values influenced by the difficulty levels.To derive semi-active transitions, the Junyi dataset contains similarity scores between two distinct exercises, quantifying their similarity on a 9-point Likert scale. Once the transition matrices are computed under the active action a = 2 for all arms, we proceed to calculate the transition matrix for the semi-active action a = 1. This involves normalizing the similarity scores to the range [0, 1], denoted as σ. For any chosen arm/topic, we can then compute its neighbor's transition matrix under the semi-active action a = 1 with P 1 0,1 = σ(P 2 0,1 ), where σ signifies the similarity proportion. It is worth noting that an arm's transition matrix for the semi-active action varies due to different neighbors being selected -different neighbors correspond to different similarity scores.To address this, we can store the transition matrix of semi-active actions for different neighbor selection scenarios, preserving the flexibility of our algorithm. In this work, for simplicity, we opt not to distinguish the impact of different neighbors being selected. Instead, we calculate the average similarity for all arms in a group average them, and use the resultant average as σ.For the OLI Statics dataset, we use a constant value of σ = 0.8 since there are no similarity scores available.E.1 Comparing Different Network Setups We present the results for different network setups in Table 3 . We note that as the number of topics approach the number of arms (i.e. N topics = {30, 40}, all algorithms perform in a highly unstable manner, as reflected in the standard deviations presented. We emphasizes here that the performance of EduQate is dependent on the quality of the network it is working on, and tends to thrive in more complex, yet realistic scenarios, such as the Junyi dataset presented in Figure 2 . We present an example of a graph generated when N topics = 40 in Figure 4 , where we notice that many arms do not belong to a group. Under this network, the EdNetRMAB can be approximated to a traditional RMAB, where the arms are independent of each other. We investigate the importance of the Experience Replay buffer in EduQate, as shown in Figure 5 and Table 4 . For the Simulated and Junyi datasets, EduQate without Experience Replay (EduQate-) does not achieve the performance levels of the full EduQate algorithm within 800 episodes, highlighting the importance of methods that aid Q-learning convergence. In real-world applications, slow convergence can result in students experiencing a curriculum similar to a random policy, leading to sub-optimal learning experiences during the early stages. This issue is known as the cold-start problem [3] .Future work in EdNetRMABs should explore methods to overcome cold-start problems and improve convergence in Q-learning-based methods.F Q-Learning Q-learning [25] is a popular reinforcement learning method that enables an agent to learn optimal actions in an environment by iteratively updating its estimate of state-action value, Q(s, a), based on the rewards it receives. The objective, therefore, to learn Q * (s, a) for each state-action pair of an MDP, given by:Q * (s, a) = r(s) + s ′ ∈S P (s, a, s ′ ) • V * (s ′ )where V * (s ′ ) is the optimal expected value of a state, is given by:V * (s) = max a∈A (Q(s, a))Q-learning estimates Q * through repeated interactions with the environment. At each time step t, the agent takes an action a using its current estimate of Q values and current state s, thus received a reward of r(s) and new state s ′ . Q-learning then updates the current estimate using the following:EQUATIONwhere α ∈ [0, 1] is the learning rate that controls updates, and γ is the discount on future rewards associated with the MDP. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Code [17] and datasets [7, 4] were appropriately cited.• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators.Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.http://www.Junyiacademy.org/https://oli.cmu.edu/courses/engineering-statics-open-free/http://david-hu.com/2011/11/02/how-khan-academy-is-using-machine-learning-to-assess-studentmastery.html 


Decision: The reviewers for this paper had a diverging assessment with a huge spread in the ratings (8, 8, 4, 3, 3). On the one hand, several reviewers had a positive assessment, highlighting the paper's strong contributions of introducing a new approach for creating personalized curricula that can account for interdependencies between items. On the other hand, several reviewers had negative assessment, raising concerns related to limited experimental evaluation and modeling that might not capture real-world educational settings in practice. We want to thank the authors for their detailed responses. Based on the raised concerns and follow-up discussions, unfortunately, the final decision is a rejection. Nevertheless, the reviewers have provided detailed and constructive feedback. We hope the authors can incorporate this feedback when preparing future revisions of the paper.

Reviews: review: paper_summary: paper_summary: This paper presents a way to extend RMAB with Q-learning to accommodate the fact that some items/arms belong to a group. RMAB can be considered as a weakened version of contextual bandit CB but also a strong version of CB since it considers state transitions (explicitly defined on arms).
strength_weakness: strength_weakness: 1. Can add some more ablation studies.
2. Some clarifications (see question section).
questions: questions: I have a few questions related to the technical contributions of the paper. None of these questions are reasons for rejection. I just want some clarification from the authors.
1. Equation (1) -- it seems to me that this altered the original definition of Whittle Index by summing over pseudo-actions. Can you explain by doing so, would it change the optimality of Whittle Index as a solution to RMAB? For example, how do we know under such an extension, the Whittle Index can still get the optimal solution? Is Theorem 1 answering this question? If so, does the original Whittle Index algorithm guarantee optimality (I assume over sublinear regret?) for both k=1 and k>1 arm selection strategy? I apologize if my understanding of RMAB is inaccurate.
2. If it's easy to run, can you add an ablation study that's under misspecification of $\phi$? Often in education, the grouping of items/materials is not exact -- and can often be wrong. The membership function $\phi$ has errors. I'm a bit curious about how much the quality of $\phi$ impacts EduQate. I understand it's hard to add experiments given the tight rebuttal period, so don't worry if this is too much of an ask.
3. Eq 4 shows IB as a metric. My question is: if both $\pi$ and $EQ$ are worse than random -- then IB will still be positive. This is not a metric that shows "successful algorithms must demonstrate superiority over random policies" -- this is a metric that says we should have a policy better than the baseline, but it can be worse. Do I understand this correctly? IB is a valid metric to use. Also, why is IB 100% in Table  1? Maybe my intuition is just off, but how is 100% achieved? Can you give an example?
limitations: limitations: The authors discussed limitations in a section.
paper_summary: paper_summary: This paper proposes a solution to generate personalized learning curricula in educational settings, focusing on the challenge of accounting for interdependencies between learning topics. It argues that existing approaches, often based on the Restless Multi-Armed Bandit (RMAB) framework, fall short by assuming independence of learning content which is unrealistic in educational settings. To address this the paper introduces a new model Restless Multi-armed Bandits for Education (EdNetRMABS) enabling the capture of relationships between independent learning items. Building upon this model, the authors propose a new algorithm called EduQate, which leverages Q-learning and the Whittle index to compute an interdependency-aware teacher policy for recommending educational content. Notably, EduQate doesn't require prior knowledge of the transition matrix, unlike traditional Whittle index methods.

The authors provide a theoretical analysis demonstrating the optimality of EduQate for the case of recommending a single item at each time step (k=1). While finding the optimal solution for recommending multiple items (k>1) is proven to be NP-hard, a heuristic greedy algorithm is proposed to find solutions.

Through experiments on synthetic and real-world datasets (Junyi and OLI), the paper demonstrates the superiority of EduQate over baseline policies, including Threshold Whittle (TW), WIQL, Myopic, and Random. EduQate consistently achieves higher intervention benefits and average rewards across all datasets. Further analysis reveals the effectiveness of the replay buffer in EduQate, mitigating the "cold-start problem" common in reinforcement learning applications.
strength_weakness: strength_weakness: The assumption of fully observable knowledge states is a significant limitation. Future work should explore extending the model and algorithm to handle partial observability, a more realistic scenario in education.
While the complexity analysis is provided, further investigation on the scalability of EduQate to larger datasets and more complex networks would strengthen its practical applicability.
The current work primarily focuses on maximizing long-term rewards. Further analysis on balancing exploration and exploitation in EdNetRMABs could offer valuable insights for curriculum design.
questions: questions: Can EduQate's recommendations be easily interpreted?  Understanding the rationale behind recommendations can enhance trust and facilitate pedagogical insights.
Are there any potential ethical implications of using EdNetRMABs and EduQate in education?
limitations: limitations: 1. The experiments are based on simulated students and existing datasets. Real-world classrooms involve numerous factors not accounted for in the model, such as student motivation, engagement or diverse learning styles. This may limit the practical utility of this method.
2. The paper does not address the interpretability of EduQate's recommendations.
3. The assumption that the student states are fully observable is a major limitation of this work. This can result in overconfident recommendations.
paper_summary: paper_summary: The paper proposes EduQate, an innovative framework that leverages EdNetRMABs to achieve interdependence among knowledge points. By using Q-learning, EduQate implements optimal strategies for personalized learning, offering optimality guarantees without needing explicit knowledge of transition functions governing student learning states. This approach dynamically adapts educational content to individual progress, optimizing learning experiences and outcomes. The effectiveness of EduQate is validated using three real-world datasets, demonstrating its capability to identify and implement the most effective teaching strategies.
strength_weakness: strength_weakness: The formatting of the article is messy, and the images are not very clear. The layout on the sixth page is problematic, with formulas and text mixed together, making it confusing. The images on the seventh page and some of the appendix images are blurred. I suggest using a different format to redraw them.

There are too few baselines in the experimental section, and there are very few validation experiments. Merely presenting outstanding performance in one experimental metric is insufficient to demonstrate the superiority of your method. It would be beneficial to include some reinforcement learning baselines for comparison. Other reinforcement learning methods, such as the Bayesian network approach mentioned in the citations, can also accomplish the task. Therefore, the necessity of your method is unclear in the paper.

Your method does not consider the specific circumstances of practical problems. It raises fairness concerns. Ensuring that all knowledge points are fairly selected in a specific educational context is a crucial issue. Although the article presents a good framework, I believe it is not feasible to use it.
questions: questions: Your claim seems to have ambiguity. In equation 5 of the proof, the expression for 

$Q(s_i,a_i=1)-Q(s_i,a_i=0)+\sum_{j∈ϕ_i^-}(Q(s_j,a_j=1)-Q(s_j,a_j=0))$

However, in equation 1, the expression for λ_i is given as:

$Q(s_i,a_i=2)-Q(s_i,a_i=0)+\sum_{j∈ϕ_i^-}(Q(s_j,a_j=1)-Q(s_j,a_j=0))$

Did I miss something?

Regarding equation 9, I am unable to comprehend how $Q(s,a=I_i)$ is derived from the previous step without further information.
limitations: limitations: As mentioned above, there are many factors to consider in the specific field of education. These factors include fairness, mapping each question's knowledge points to the ARMs mentioned above, and the presence of cold-start problems, among others. Time is also a crucial factor to consider during real-time usage, although the paper does not mention specific algorithmic time information in practice.
paper_summary: paper_summary: The paper introduces **EduQate**, a system that generates adaptive educational curricula using restless multi-armed bandits (RMABs). This method aims to efficiently achieve mastery across multiple interdependent educational contents. Unlike traditional methods that assume learning contents are independent, EduQate acknowledges and leverages the interdependencies between different educational concepts.

The paper addresses key challenges in modeling and optimizing the learning process in educational environments where the learning of different topics is interconnected. By considering these interdependencies, the proposed EduQate system ensures more accurate and effective personalized learning.
strength_weakness: strength_weakness: 1. **Clarity of Explanations**: Due to the limited length, many explanations in the main text are not very clear. For instance, the survival and design of the student model and the content and information of the datasets are not thoroughly explained. For example, there is a dataset mentioned later that lacks similarity content, which should have been noted when introducing the dataset.

2. **Inconsistency in Notation**: In Section 4.1, "Analysis of EduQate," the variable k in the second line has a different font from the k mentioned later. Additionally, the content related to k mentioned earlier is too far from this section, making it difficult for readers to understand and potentially leading to misunderstandings.

3. **Simplified Modeling**: The model uses only one pseudo-state, where if any content within a group is learned, all unlearned content in the group is marked as pseudo-state. This approach seems overly simplified. For example, in Case 1, if only one piece of content in a group is learned, all other content is marked as pseudo-state. In Case 2, if many pieces of content in a group are learned, the remaining content is also marked as pseudo-state. Although both cases result in a pseudo-state, their actual significance differs, and it seems the authors did not consider this distinction.

4. **Bidirectional Relationships**: The relationships between knowledge points in the paper are bidirectional, with only grouping relationships. However, in practice, many relationships between knowledge points within the same group are unidirectional. Recommending subsequent courses without recommending prerequisite courses first can lead to students needing to self-study prerequisite courses when learning subsequent courses, increasing their learning burden. The modeling in the paper does not seem to account for such unidirectional structures, instead using groups to link knowledge points, which could lead to this reverse learning issue.

5. **User Experience**: The discussion on user experience for educators and students is insufficient. Including qualitative and quantitative feedback from pilot implementations would be useful. The authors should conduct user research to gather insights on the intuitiveness and user-friendliness of the system for the target users. This could include surveys, interviews, and usability testing, focusing on ease of use, effectiveness, and areas needing improvement.

6. **Scalability of the EduQate Algorithm**: Although the paper mentions the efficiency of the EduQate algorithm, it lacks an in-depth exploration of its scalability in large-scale educational environments. Detailed performance benchmarking of the algorithm, including analysis of computation time and resource utilization, would be beneficial. The authors should provide a comprehensive performance analysis of the algorithm as data scale and complexity increase, including potential optimization techniques for handling large datasets.
questions: questions: 1. **Why not use Deep Q-learning?**: The method used in the paper seems to have poor scalability and may only be suitable for relatively fixed knowledge systems. There is a lack of discussion on its scalability in large-scale educational environments.

2. **Sample Sizes in Datasets**: How were the sample sizes for the datasets set? In Junyi and OLI Statics, the quantity N is 100, but the number of topics varies. Is this N an empirical value or does it represent all the data in the original datasets? What is the actual N in this task, i.e., how many knowledge points should a model handle in practice? These aspects are not discussed in the main text, which is confusing.

3. **E(BI) Result of 100**: The paper reports an E(BI) result of 100 for the proposed method, but it is unclear what this result signifies compared to the random method. The paper also states that other results did not exceed 50, implying that these results are worse than random. Could you clarify why not exceeding 50 indicates worse performance than the random method and explain the significance of the E(BI) result of 100 in this context?
limitations: limitations: The authors have explicitly acknowledged the limitations of their work, providing explanations for these constraints and discussing their impact on the research. The paper does not pose any negative social impact, as it aims to improve educational practices through a more personalized approach to learning.
paper_summary: paper_summary: This paper proposes a variant of Multi-Armed Bandits (MAB) named EdNetRMABs that model the interdependencies between learning contents to meet the real-world education scenarios. Subsequently, the authors introduce EduQate, an interdependency-aware Q-learning algorithm to optimize content recommendation given the EdNetRMABs. The paper demonstrates the theoretical and empirical effectiveness of EduQate with the experimental results of synthetic and real-world data.
strength_weakness: strength_weakness: 1. As this work considers interdependency awareness in content recommendation in educational scenarios, the generated group in the experiment is not clear. The authors need to clarify how to capture relevance between the selected exercises of each topic in different datasets. Furthermore, does the different number of topics influence the experimental results?
2. The definition of state space representing a student's knowledge states a binary value. However, in real-world scenarios, the knowledge states are multi-level. A fine-grained knowledge state estimation result is essential for the content recommendation in adaptive learning.
3. The paper lacks the details of EdNetRMABs. Some illustrations would help to understand.
questions: questions: NA
limitations: limitations: NA

Comments: title: comment from reviewer
Acknowledgement of rebuttal
comment: Thanks for reading our response. It seems the reviewer lowered the score from 5 to 4. Can we please check if there was some issue with our response? Could we help clarify anything?
Thanks for the response.
The new metric seems much more reasonable! Thanks for the explanation.

I read through the rejection reviews as well. Multi-Arm Bandit is still one of the most commonly used algorithms by education platforms such as Duolingo and others. As far as bandit paper goes, this paper's experimental results are sufficient.

I will keep my score, and I'm happy to defend this paper and my decisions.
Thank you for addressing my questions.