# A Knowledge-Fusion Ranking System with an Attention Network for Making Assignment Recommendations

Authors: Canghong Jin; Yuli Zhou; Shengyu Ying; Chi Zhang; Weisong Wang; Minghui Wu
Journal: Computational Intelligence and Neuroscience
 (2020)

---

## Abstract

In recent decades, more teachers are using question generators to provide students with online homework. Learning-to-rank (LTR) methods can partially rank questions to address the needs of individual students and reduce their study burden. Unfortunately, ranking questions for students is not trivial because of three main challenges: (1) discovering students’ latent knowledge and cognitive level is difficult, (2) the content of quizzes can be totally different but the knowledge points of these quizzes may be inherently related, and (3) ranking models based on supervised, semisupervised, or reinforcement learning focus on the current assignment without considering past performance. In this work, we propose KFRank, a knowledge-fusion ranking model based on reinforcement learning, which considers both a student’s assignment history and the relevance of quizzes with their knowledge points. First, we load students’ assignment history, reorganize it using knowledge points, and calculate the effective features for ranking in terms of the relation between a student’s knowledge cognitive and the question. Then, a similarity estimator is built to choose historical questions, and an attention neural network is used to calculate the attention value and update the current study state with knowledge fusion. Finally, a rank algorithm based on a Markov decision process is used to optimize the parameters. Extensive experiments were conducted on a real-life dataset spanning a year and we compared our model with the state-of-the-art ranking models (e.g., ListNET and LambdaMART) and reinforcement-learning methods (such as MDPRank). Based on top-k nDCG values, our model outperforms other methods for groups of average and weak students, whose study abilities are relatively poor and thus their behaviors are more difficult to predict.


---

## Body

## 1. Introduction

Educational data mining is an emerging discipline, concerned with developing methods for exploring the unique and increasingly large-scale data that come from educational settings and using those methods to understand students and the settings which they learn in better. In recent years, physical bricks and mortar classrooms are starting to lose their monopoly as a place of learning. The Internet has made online learning possible, and many researchers and educators are interested in online learning to enhance and improve students’ learning outcomes while mitigating the reduction in resources [1]. Online learning platforms include Coursera, MOOC, and Udacity. Online assignments, such as quizzes, practice exercises, virtual labs, online literature searches, and simulations, play a critical role in online learning [2].One of the most important tasks of an online assignment system in educational data mining is to find suitable questions for students according to their ability. An online assignment system can make learning more efficient. It can evaluate study performance and identify at-risk students, who can be given further help. For example, from a log of assignment results, we can identify topics that a student has poorly mastered and recommend to them related questions to improve their knowledge. Thus, the task is to rank a large number of questions and recommend only relevant questions to students.An automated process for producing an online assignment works as follows:(i)
Several questions, which are organized into an assignment, are assigned by a teacher weekly following the syllabus. Students are expected to finish them on time.(ii)
The system can check the answers automatically, calculate the students’ marks, and generate a report for each assignment.(iii)
Since the number of candidate questions is too large to finish in one assignment, the recommendation system should be able to choose suitable questions and discard those that are too easy or too difficult.Note that the goal of an assignment recommendation task is not whether the question can be answered correctly; that is, it is not a classification problem. Rather, questions should be ranked according to the importance of the topic in improving the student’s ability. The aim is to find the questions with the highest benefit for students.In previous studies, this problem has been tackled by supervised and semisupervised ranking methods [3–5]. In particular, state-of-the-art reinforcement learning has been used, which considers the problem as a process of sequential decision-making and learns model parameters through maximizing the rewards accumulated from all decisions [6]. MDPRank is a ranking model based on a Markov decision process (MDP). It treats documents as states and ranks the position of documents at each iteration. However, the current MDPRank is imperfect if we apply it to our assignment recommendation system directly. It is obvious that a student’s performance in an assignment not only depends on the questions, such as the marks for each question, the types of question, and the difficulty of each question, but also depends on their current knowledge, especially for those questions with similar knowledge points. Other researches [7, 8] are close to our recommend target but both of them are based on study cognition and semantic content of questions, which is more complex than our situationsIn this article, we illustrate our motivation in Figure1, which shows the relations between questions from different assignments. Our intuition is that the ranking is influenced by two dimensions: how questions have been answered in the same assignment and how questions were answered in previous assignments.Figure 1
Motivation and intuition of our solution. For a student, the right-hand box is the target assignment and the dashed circles are previously completed assignments. The geometric forms inside each period are the knowledge points in each question.Based on the above intuition, we propose a knowledge-fusion ranking system using an attention network, KFRank, which can improve the reliability and accuracy of ranking using the relations between knowledge points. Compared with the state-of-the-art learning-to-rank (LTR) and reinforcement-learning methods, our KFRank method has the following advantages:(i)
KFRank considers the ranking problem as having multilevel dimensions and generates effective cognitive features for learning models.(ii)
KFRank utilizes knowledge points and integrates them during training. We build a cluster of questions using the knowledge points and generate an attention network to pretrain the terms in questions. These are represented in vectors of questions in the classification phase.(iii)
KFRank is based on an MDP but rebuilds the environment by considering multiple factors: questions in the current assignment and results from previous similar assignments.The rest of the paper is organized as follows: first, we formulate the problem and introduce the concepts used in the assignment ranking problem in Section2. Section 3 gives the architecture of KFRank and proposes the attention network for training. Next, in Section 4, we describe our experiments with real-life datasets and compare the performance of the proposed model with other methods. Related work is discussed in Section 5, and Section 6 concludes the work.


## 2. Preliminaries

In this section, we first formally formulate the assignment ranking problem and then briefly introduce the LTR and reinforcement-learning methods for solving this problem.


### 2.1. Problem Definition

Definition 1.
An assignmentω=Q,p,t,λ, where Q=q1,q2,…,qq is a set of questions, p is a unique student, t is the assignment time, and λ is the score for q.Definition 2.
The knowledge points of a questionO are a set of knowledge points oq, which belong to question q.Definition 3.
Our assignment ranking problem is to rank questions in an assignment by predicting the performance of each student based on the difficulty of the questions. Questions higher in the list are relatively easier for a particular student than other questions.


### 2.2. Ranking Using Supervised Learning

LTR is a sorting method based on supervised learning. The user-item scoring matrix is produced by a recommendation algorithm after learning from a training set. Here, different sorting techniques, such as pointwise [9], pairwise [10], and listwise [11], can be used to obtain a sorting model. In the test phase, the system generates an ordered list of items for target users using the trained model.LTR can be online or offline. In offline approaches, the training set is produced by human assessors, which is time-consuming and expensive. In contrast, an online LTR system collects data when users interact with the system, such as by clicking, moving a mouse, and entering a query string.


### 2.3. Ranking Using Reinforcement Learning

Reinforcement learning is a branch of artificial intelligence. It is good at controlling an agent who can act autonomously in a specific environment and continuously improve their behavior. Suitable problems for reinforcement learning involve learning how to do tasks and how to map the environment into actions that maximize the rewards. In reinforcement learning, the learner is a decision-making agent who is not told what to do. Instead, they attempt a task repeatedly to find the behavior that gives the greatest reward. By giving each question a reward, reinforcement learning can learn how to rank them in an assignment as a closed-loop control problem.


## 2.1. Problem Definition

Definition 1.
An assignmentω=Q,p,t,λ, where Q=q1,q2,…,qq is a set of questions, p is a unique student, t is the assignment time, and λ is the score for q.Definition 2.
The knowledge points of a questionO are a set of knowledge points oq, which belong to question q.Definition 3.
Our assignment ranking problem is to rank questions in an assignment by predicting the performance of each student based on the difficulty of the questions. Questions higher in the list are relatively easier for a particular student than other questions.


## 2.2. Ranking Using Supervised Learning

LTR is a sorting method based on supervised learning. The user-item scoring matrix is produced by a recommendation algorithm after learning from a training set. Here, different sorting techniques, such as pointwise [9], pairwise [10], and listwise [11], can be used to obtain a sorting model. In the test phase, the system generates an ordered list of items for target users using the trained model.LTR can be online or offline. In offline approaches, the training set is produced by human assessors, which is time-consuming and expensive. In contrast, an online LTR system collects data when users interact with the system, such as by clicking, moving a mouse, and entering a query string.


## 2.3. Ranking Using Reinforcement Learning

Reinforcement learning is a branch of artificial intelligence. It is good at controlling an agent who can act autonomously in a specific environment and continuously improve their behavior. Suitable problems for reinforcement learning involve learning how to do tasks and how to map the environment into actions that maximize the rewards. In reinforcement learning, the learner is a decision-making agent who is not told what to do. Instead, they attempt a task repeatedly to find the behavior that gives the greatest reward. By giving each question a reward, reinforcement learning can learn how to rank them in an assignment as a closed-loop control problem.


## 3. The Kfrank Model

Three aspects of the KFRank model are presented in this section: (1) an MDP for ranking, (2) a knowledge-fusion model for updating the environment, and (3) the architecture and algorithm (Algorithm1) of KFRank.Algorithm 1: KFRank learning.
Input: assignment records ω=Q,p,t,λn=1N, learning rate η, discount factor γ, and reward function ROutput:w(1)
Initializew with random values(2)
s⟵ updated state in an episode (ω, G), Algorithm 2(3)
Δw=0(4)
forX,Y∈Ddo(5)
s0,a0,r1,…,sM−1,aM−1,rM⟵ sample ranking in an episode(6)
fort=0toM−1do(7)
Sample an actionat∈Ast∼πat|st;w(8)
rt+1⟵ℛst,at(9)
Appendst,at,rt+1 at the end of E(10)
State transitionst+1⟵t+1,Qt\qmqt(11)
end for(12)
fort=0toM−1do(13)
Gt⟵∑k=1M−tγk−1rt+k(14)
Δw⟵Δw+γtGt∇wlogπat|st;w(15)
end for(16)
w⟵w+ηΔw(17)
end for


### 3.1. Ranking Using an MDP

Analyzing study performance can be formalized as an MDP, in which the construction of a list of ranked questions can be considered as a sequential decision-making process in which each time step corresponds to selecting a question for a corresponding position. We propose a tupleS,A,T,ℛ,π,ℒ,χ to illustrate KFRank by states, actions, transition, reward, policy, history, and rebuilder, which are defined as follows.


#### 3.1.1. States

S is a set of states that represent the environment of the current assignments. In ranking, the agent should know the current positions as well as the remaining questions. Thus, state St for step t is t,Xt, where in our model X is initially treated as a given assignment ω, and Xt are the questions still to be ranked Qt.


#### 3.1.2. Actions

A are a discrete set of actions that an agent can take in which available actions can depend on the state S, denoted as Ast. At step t, at∈Ast is used to calculate the value of each q in ω and to select a question qmat for the ranking position t+1, where mat is the index of the question selected by action at.


#### 3.1.3. Transition

TS,A is a function that maps state St and action At to a new state St+1 as S×A⟶S. At step t, action at selects qmat and removes it from Qt as follows:(1)st+1=Tst,at=t+1,Qt\qmat.


#### 3.1.4. Reward

The state value functionV:S⟶ℛ is a scalar evaluation, estimating the quality of the entire list of ranked questions (an assignment) based on the input state S. Here, we define the value function as DCG:(2)ℛDCGst,at=2ymat−1,ift=0,2ymat−1log2t+1,ift=1,where ymat is a relevance label for the current selected question qmat. In our model, we calculate ymat according to a student’s performance and the difficulty of a question. The difficulty θ of q is defined as(3)qθ=∑k=1Nqr=0∑k=1Nqr=1+∑k=1Nqr=0,where r=0 means the result is wrong and r=1 means the result is right. Thus, ymat is defined as(4)ymat=Vqr×qθ.


#### 3.1.5. Policy

π is a function that takes a state as input and outputs a distribution over all possible actions a∈As. KFRank calculates the probability of selecting each question based on its current rank:(5)πat|st;w=expwTqmat∑a∈AstexpwTqmat,where w∈ℝK are the model parameters, whose dimension is the same as that of the ranking feature.In our case, the policy is an agent’s strategy to rank the assignment by predicting the study results and measure the rewards by nDCG. Obviously, some policies are better than others, and there are multiple ways to assess them.


#### 3.1.6. History

ℒ is a set of historical assignment results for a group of students. Here, ℒ=lpq, where l is a previous result, q is a question, and p is a student.ℒ is designed as an information retrieval system and contains student IDs, questions, knowledge points, results, and operating time. Several metrics related to information retrieval can be used to compare the similarity of given questions and archived questions as follows:(6)Dℒ,q,ℳ,k=ℳoqi∈ℒ,oq,k,where q is a question in ω, ℳ is an information retrieval search function, and k is the number of output results.


#### 3.1.7. Rebuilder

χ is a module that updates current states s to s^t with D as input. We extract a student’s performance on special knowledge points o from ℒ and predict their current ability using a recurrent neural network.


### 3.2. Basic Study State

We calculate the effective features for ranking in terms of the relation between a student’s knowledge cognitive and the question. In this article, we first estimate the difficulty of each question using a correctness ratio. Then, for each student, their knowledge cognitive is measured as their average score for all completed assignments based on knowledge points. Here, a student’s knowledge cognitive is dynamic and updated during the study process. Thus, if a student does very well on an assignment, all the related cognitive levels increase rapidly. On the other hand, the difficulty of a question is fixed or not easy to change, because it depends on the performance of all students.Based on the student’s knowledge cognitive and the difficulty of the questions, we utilize several similarity functions to construct the representation of the student’s states for ranking according to the traditional learning-to-rank method, including Euclidean distance, Pearson’s similarity, Manhattan distance, cosine similarity, and so on.As shown in Figures2 and 3, one question can contain several knowledge points and one knowledge point may have many related questions. Thus, for a given question with several knowledge points, we first calculate the state for each knowledge point and then merge all the related states. According to the relationship of knowledge points in the question, we can obtain many basic triples in the knowledge graph. We will introduce how to build a reasonable representation of knowledge points in Section 3.3.Figure 2
Typical question. In this question, there are two separate knowledge points: an unary equation and an absolute value. If a student answers the question right, both cognitive levels would be increased.Figure 3
Relations between questions and knowledge points. (a) Each circle represents a question. (b) The correlations between questions. Knowledge points with the same color are from the same question; for example,o1 and o2 are two different knowledge points in q2, whereas q1 connects to q2 with knowledge point o1.
(a)(b)


### 3.3. Knowledge Representation with TransR

Since the content of math questions can vary considerably, we use knowledge points to illustrate the relations between questions, as Figure3 shows. From the knowledge points and the relations between them, which are manually marked, we can construct the triples in the knowledge graphs. In this project, the relations between knowledge points are classed as contains, belongs, and equals. The following is an example.In triangleA, angle B is equal to 90° and angle B is a right angle.Here,triangle is a knowledge point, the relation between A and B is contains, the relation between B and A is belongs, and the relation between right angle and 90° is equals. A triple is defined as h,r,t, where h,t represent the embedding of knowledge points and r is the set of relations. From the knowledge graph, we can obtain the vectors of knowledge points using TransR [12].In TransR, the score function is defined as(7)frh,t=hr+r−tr22,and the model convergence is based on minimizing(8)L=∑h,r,t∈S∑h′,r,t′∈S′max0,frh,t+γ−frh′,t′,where γ is the margin, S is the set of correct triples, and S′ is the set of incorrect triples.


### 3.4. Attention-Based Knowledge-Fusion Model

To tackle the various knowledge points, we design an attention-based model using knowledge fusion. As Figure4 shows, Q is the representation of a student’s current knowledge, as noted in Section 3.2, and k is the embedding of concepts trained from TransR. In the triple (query, key, value) in the attention mechanism, we set Q as k, the knowledge point as q, and the reward corresponding to the question as v. The model is trained in the same way as the encoder part in the transformer (Figure 4), the vectors of multiple knowledge points were incorporated into the basic study state with the way of attention. The output is the latest performance. It is equivalent to integrating the information of different knowledge points into the original state and getting a new vector to represent the current state. R in the figure represent the student’s current knowledge state for each question. The status update process is shown in Algorithm 2.Figure 4
Updating state using knowledge fusion.Q is basic study state, K is the representation of knowledge points, which is calculated from knowledge graph with TransR. R is the representation of output status.Algorithm 2: State updating in an episode.
Input: state s∈ω=Q,p,t,λ, knowledge graph GOutput: updated state s^(1)
forqm∈Qdo(2)
k⟵G //get embedding of knowledge points from G(3)
q⟵s, v⟵λ(4)
att=softmaxq∗k/dk(5)
s^=denseatt∗v(6)
end forWe show the study state vectors before and after the attention model in Figure5. We select ten different knowledge points and related study states and then use the attention model to pretrain input vectors. Then, we use t-SNE to show the latent space representations of two states. Note that the study state after attention training is simple and is closely surrounded by knowledge points with the same color.Figure 5
Visualization of state vectors. (a) Before attention model. (b) After attention model.
(a)(b)


### 3.5. Overview of KFRank

Figures6 and 7 illustrate the construction of the question ranking. For each question in an assignment, first, we extract related performance records. In each episode, the environment is updated with the current status from the knowledge-fusion model. Based on the policy and value function, the agent chooses the optimal action that gives the greatest long-term return. After taking action, the environment is updated.Figure 6
The agent-environment interaction in KFRank.Figure 7
There are four steps in KFRank, from left to right: (1) data collection, knowledge tagging, and organization; (2) generating knowledge representations; (3) updating the current study performance environment with knowledge fusion; and (4) reinforcement-learning ranking.The sorting construct for a given training document can be formalized as follows. A student’s assignment is a queryω, which is a set of questions Q with length M. The initial state is s0=0,Q. At each step t=0,…,M−1, the agent chooses the optimal action at to select qmqt from the set of questions Q as the rank t (lines 7 and 8 in Algorithm 1). The action is removed from Qt, as in equation (1) (lines 9 and 10 in Algorithm 1). We calculate ymat using equation (3) and calculate the long-term return reward. The process is repeated until all of the M questions have been selected.We propose to learn the parametersw in KFRank using a policy-gradient algorithm based on reinforcement learning [13]. The goal of this algorithm is to maximize the long-term return Gt:(9)Gt=∑k=1M−tγk−1rt+k.In the algorithm, the gradientΔwJw is calculated as(10)ΔwJw=γtGt∇wlogπat|st;w.At each iteration, an episode is sampled with the current policy. At each stept, the parameters w are adjusted according to ∇wlogπat|st;w, which maximizes the increase in the probability of repeating the action at for state st. In this way, Gt moves the parameters in the direction that gives the greatest return for the action.


## 3.1. Ranking Using an MDP

Analyzing study performance can be formalized as an MDP, in which the construction of a list of ranked questions can be considered as a sequential decision-making process in which each time step corresponds to selecting a question for a corresponding position. We propose a tupleS,A,T,ℛ,π,ℒ,χ to illustrate KFRank by states, actions, transition, reward, policy, history, and rebuilder, which are defined as follows.


### 3.1.1. States

S is a set of states that represent the environment of the current assignments. In ranking, the agent should know the current positions as well as the remaining questions. Thus, state St for step t is t,Xt, where in our model X is initially treated as a given assignment ω, and Xt are the questions still to be ranked Qt.


### 3.1.2. Actions

A are a discrete set of actions that an agent can take in which available actions can depend on the state S, denoted as Ast. At step t, at∈Ast is used to calculate the value of each q in ω and to select a question qmat for the ranking position t+1, where mat is the index of the question selected by action at.


### 3.1.3. Transition

TS,A is a function that maps state St and action At to a new state St+1 as S×A⟶S. At step t, action at selects qmat and removes it from Qt as follows:(1)st+1=Tst,at=t+1,Qt\qmat.


### 3.1.4. Reward

The state value functionV:S⟶ℛ is a scalar evaluation, estimating the quality of the entire list of ranked questions (an assignment) based on the input state S. Here, we define the value function as DCG:(2)ℛDCGst,at=2ymat−1,ift=0,2ymat−1log2t+1,ift=1,where ymat is a relevance label for the current selected question qmat. In our model, we calculate ymat according to a student’s performance and the difficulty of a question. The difficulty θ of q is defined as(3)qθ=∑k=1Nqr=0∑k=1Nqr=1+∑k=1Nqr=0,where r=0 means the result is wrong and r=1 means the result is right. Thus, ymat is defined as(4)ymat=Vqr×qθ.


### 3.1.5. Policy

π is a function that takes a state as input and outputs a distribution over all possible actions a∈As. KFRank calculates the probability of selecting each question based on its current rank:(5)πat|st;w=expwTqmat∑a∈AstexpwTqmat,where w∈ℝK are the model parameters, whose dimension is the same as that of the ranking feature.In our case, the policy is an agent’s strategy to rank the assignment by predicting the study results and measure the rewards by nDCG. Obviously, some policies are better than others, and there are multiple ways to assess them.


### 3.1.6. History

ℒ is a set of historical assignment results for a group of students. Here, ℒ=lpq, where l is a previous result, q is a question, and p is a student.ℒ is designed as an information retrieval system and contains student IDs, questions, knowledge points, results, and operating time. Several metrics related to information retrieval can be used to compare the similarity of given questions and archived questions as follows:(6)Dℒ,q,ℳ,k=ℳoqi∈ℒ,oq,k,where q is a question in ω, ℳ is an information retrieval search function, and k is the number of output results.


### 3.1.7. Rebuilder

χ is a module that updates current states s to s^t with D as input. We extract a student’s performance on special knowledge points o from ℒ and predict their current ability using a recurrent neural network.


## 3.1.1. States

S is a set of states that represent the environment of the current assignments. In ranking, the agent should know the current positions as well as the remaining questions. Thus, state St for step t is t,Xt, where in our model X is initially treated as a given assignment ω, and Xt are the questions still to be ranked Qt.


## 3.1.2. Actions

A are a discrete set of actions that an agent can take in which available actions can depend on the state S, denoted as Ast. At step t, at∈Ast is used to calculate the value of each q in ω and to select a question qmat for the ranking position t+1, where mat is the index of the question selected by action at.


## 3.1.3. Transition

TS,A is a function that maps state St and action At to a new state St+1 as S×A⟶S. At step t, action at selects qmat and removes it from Qt as follows:(1)st+1=Tst,at=t+1,Qt\qmat.


## 3.1.4. Reward

The state value functionV:S⟶ℛ is a scalar evaluation, estimating the quality of the entire list of ranked questions (an assignment) based on the input state S. Here, we define the value function as DCG:(2)ℛDCGst,at=2ymat−1,ift=0,2ymat−1log2t+1,ift=1,where ymat is a relevance label for the current selected question qmat. In our model, we calculate ymat according to a student’s performance and the difficulty of a question. The difficulty θ of q is defined as(3)qθ=∑k=1Nqr=0∑k=1Nqr=1+∑k=1Nqr=0,where r=0 means the result is wrong and r=1 means the result is right. Thus, ymat is defined as(4)ymat=Vqr×qθ.


## 3.1.5. Policy

π is a function that takes a state as input and outputs a distribution over all possible actions a∈As. KFRank calculates the probability of selecting each question based on its current rank:(5)πat|st;w=expwTqmat∑a∈AstexpwTqmat,where w∈ℝK are the model parameters, whose dimension is the same as that of the ranking feature.In our case, the policy is an agent’s strategy to rank the assignment by predicting the study results and measure the rewards by nDCG. Obviously, some policies are better than others, and there are multiple ways to assess them.


## 3.1.6. History

ℒ is a set of historical assignment results for a group of students. Here, ℒ=lpq, where l is a previous result, q is a question, and p is a student.ℒ is designed as an information retrieval system and contains student IDs, questions, knowledge points, results, and operating time. Several metrics related to information retrieval can be used to compare the similarity of given questions and archived questions as follows:(6)Dℒ,q,ℳ,k=ℳoqi∈ℒ,oq,k,where q is a question in ω, ℳ is an information retrieval search function, and k is the number of output results.


## 3.1.7. Rebuilder

χ is a module that updates current states s to s^t with D as input. We extract a student’s performance on special knowledge points o from ℒ and predict their current ability using a recurrent neural network.


## 3.2. Basic Study State

We calculate the effective features for ranking in terms of the relation between a student’s knowledge cognitive and the question. In this article, we first estimate the difficulty of each question using a correctness ratio. Then, for each student, their knowledge cognitive is measured as their average score for all completed assignments based on knowledge points. Here, a student’s knowledge cognitive is dynamic and updated during the study process. Thus, if a student does very well on an assignment, all the related cognitive levels increase rapidly. On the other hand, the difficulty of a question is fixed or not easy to change, because it depends on the performance of all students.Based on the student’s knowledge cognitive and the difficulty of the questions, we utilize several similarity functions to construct the representation of the student’s states for ranking according to the traditional learning-to-rank method, including Euclidean distance, Pearson’s similarity, Manhattan distance, cosine similarity, and so on.As shown in Figures2 and 3, one question can contain several knowledge points and one knowledge point may have many related questions. Thus, for a given question with several knowledge points, we first calculate the state for each knowledge point and then merge all the related states. According to the relationship of knowledge points in the question, we can obtain many basic triples in the knowledge graph. We will introduce how to build a reasonable representation of knowledge points in Section 3.3.Figure 2
Typical question. In this question, there are two separate knowledge points: an unary equation and an absolute value. If a student answers the question right, both cognitive levels would be increased.Figure 3
Relations between questions and knowledge points. (a) Each circle represents a question. (b) The correlations between questions. Knowledge points with the same color are from the same question; for example,o1 and o2 are two different knowledge points in q2, whereas q1 connects to q2 with knowledge point o1.
(a)(b)


## 3.3. Knowledge Representation with TransR

Since the content of math questions can vary considerably, we use knowledge points to illustrate the relations between questions, as Figure3 shows. From the knowledge points and the relations between them, which are manually marked, we can construct the triples in the knowledge graphs. In this project, the relations between knowledge points are classed as contains, belongs, and equals. The following is an example.In triangleA, angle B is equal to 90° and angle B is a right angle.Here,triangle is a knowledge point, the relation between A and B is contains, the relation between B and A is belongs, and the relation between right angle and 90° is equals. A triple is defined as h,r,t, where h,t represent the embedding of knowledge points and r is the set of relations. From the knowledge graph, we can obtain the vectors of knowledge points using TransR [12].In TransR, the score function is defined as(7)frh,t=hr+r−tr22,and the model convergence is based on minimizing(8)L=∑h,r,t∈S∑h′,r,t′∈S′max0,frh,t+γ−frh′,t′,where γ is the margin, S is the set of correct triples, and S′ is the set of incorrect triples.


## 3.4. Attention-Based Knowledge-Fusion Model

To tackle the various knowledge points, we design an attention-based model using knowledge fusion. As Figure4 shows, Q is the representation of a student’s current knowledge, as noted in Section 3.2, and k is the embedding of concepts trained from TransR. In the triple (query, key, value) in the attention mechanism, we set Q as k, the knowledge point as q, and the reward corresponding to the question as v. The model is trained in the same way as the encoder part in the transformer (Figure 4), the vectors of multiple knowledge points were incorporated into the basic study state with the way of attention. The output is the latest performance. It is equivalent to integrating the information of different knowledge points into the original state and getting a new vector to represent the current state. R in the figure represent the student’s current knowledge state for each question. The status update process is shown in Algorithm 2.Figure 4
Updating state using knowledge fusion.Q is basic study state, K is the representation of knowledge points, which is calculated from knowledge graph with TransR. R is the representation of output status.Algorithm 2: State updating in an episode.
Input: state s∈ω=Q,p,t,λ, knowledge graph GOutput: updated state s^(1)
forqm∈Qdo(2)
k⟵G //get embedding of knowledge points from G(3)
q⟵s, v⟵λ(4)
att=softmaxq∗k/dk(5)
s^=denseatt∗v(6)
end forWe show the study state vectors before and after the attention model in Figure5. We select ten different knowledge points and related study states and then use the attention model to pretrain input vectors. Then, we use t-SNE to show the latent space representations of two states. Note that the study state after attention training is simple and is closely surrounded by knowledge points with the same color.Figure 5
Visualization of state vectors. (a) Before attention model. (b) After attention model.
(a)(b)


## 3.5. Overview of KFRank

Figures6 and 7 illustrate the construction of the question ranking. For each question in an assignment, first, we extract related performance records. In each episode, the environment is updated with the current status from the knowledge-fusion model. Based on the policy and value function, the agent chooses the optimal action that gives the greatest long-term return. After taking action, the environment is updated.Figure 6
The agent-environment interaction in KFRank.Figure 7
There are four steps in KFRank, from left to right: (1) data collection, knowledge tagging, and organization; (2) generating knowledge representations; (3) updating the current study performance environment with knowledge fusion; and (4) reinforcement-learning ranking.The sorting construct for a given training document can be formalized as follows. A student’s assignment is a queryω, which is a set of questions Q with length M. The initial state is s0=0,Q. At each step t=0,…,M−1, the agent chooses the optimal action at to select qmqt from the set of questions Q as the rank t (lines 7 and 8 in Algorithm 1). The action is removed from Qt, as in equation (1) (lines 9 and 10 in Algorithm 1). We calculate ymat using equation (3) and calculate the long-term return reward. The process is repeated until all of the M questions have been selected.We propose to learn the parametersw in KFRank using a policy-gradient algorithm based on reinforcement learning [13]. The goal of this algorithm is to maximize the long-term return Gt:(9)Gt=∑k=1M−tγk−1rt+k.In the algorithm, the gradientΔwJw is calculated as(10)ΔwJw=γtGt∇wlogπat|st;w.At each iteration, an episode is sampled with the current policy. At each stept, the parameters w are adjusted according to ∇wlogπat|st;w, which maximizes the increase in the probability of repeating the action at for state st. In this way, Gt moves the parameters in the direction that gives the greatest return for the action.


## 4. Performance Evaluation

### 4.1. Datasets

We conducted experiments to validate the performance of our method using a real-life dataset (Table1), from two applications in pad: teacher client and student client. The data spanned the period from September 2017 to June 2018. There were nearly 70 million records for 40,000 students. Based on the student’s historical activity and correctness rate, we selected a record of 300 students with relatively high quality of records. Then, we apportion the data into training and test sets, with a 70-30 split. After that, excellent teachers create the mathematics knowledge graph of middle school, which contains more than 700 nodes and 2000 relationship edges. Finally, we use TransR approach to embed knowledge topic for learning.Table 1
Summary of the dataset.
MetricMeritAverageWeakNumber of students100100100Number of quizzes16,25121,68424,418Number of knowledge points354440401Number of records65,54668,01173,232Ratio correct0.920.610.21To evaluate the effectiveness of the ranking, we split the students into three groups:merit students, average students, and weak students. In the dataset, almost 72 percent of the questions covered three or more knowledge points and less than 14 percent of the questions had only one knowledge point. The distributions of rewards for the three groups of students are shown in Figure 8.Figure 8
Distribution of rewards for three groups of students. (a) Merit student. (b) Average student. (c) Weak student.
(a)(b)(c)


### 4.2. Evaluation Criteria

We first calculate the score of each question for ranking by the performance result of students and difficulty of question. For example, for a given question, only 60 percent of students could choose the right answer, the difficulty of this question is 0.4. Then, we also give the correct and wrong answer with 5 and 1 score. Finally, students will get 2 points if they do the right questions, and only 0.4 if they make a mistake. For each assignment, according to the student’s answer to each question, the rank score is calculated as the true value.We usenDCG@k to measure the performance. To get nDCG@k, we first calculate DCG@k:(11)DCGf=∑r=1kvr1log1+r,where r is the rank of items in the recommendation list, k is the length of the recommendation list, f is the ranking function or algorithm, vr is the value of the rth item, and 1/log1+r is the discount. iDCG is an ideal discounted cumulative gain, iDCG@k is also needed and calculated in a similar way. In iDCG@k, the questions in the recommendation list are ranked by their original values instead of by the ranking algorithm.


### 4.3. Experimental Results

Table2 compares the performance of our model with other methods using nDCG@5 and nDCG@10. The higher the score, the better the performance. Of the LTR methods, CoordAscent, LambdaMART, and ListNet perform relatively well and better than the original reinforcement-learning method, MDPRank. KFRank with updated environments has the best nDCG value for nearly all groups of students.Table 2
nDCG of ranking methods.
MethodMerit studentAverage studentWeak student@5@10@5@10@5@10MART [14]0.7010.7510.3090.3190.2720.384RankNet [15]0.5070.5470.2980.3020.4330.509RankBoost [16]0.3730.3180.2080.2560.3690.403AdaRank [17]0.2880.3620.1730.2280.1960.391CoordAscent [18]0.4660.6810.2410.2930.4940.539LambdaRank [19]0.1560.1550.2350.2130.4290.495LambdaMART [20]0.6460.7490.2790.3920.5150.569ListNET [11]0.4330.6260.1910.2400.4700.464RandForest [21]0.7310.7810.3830.3660.2980.431MDPRank [6]0.3440.4410.4130.5410.5680.642KFRank0.4250.4440.5990.5650.6360.686Bold values represent the best performance of nDCG@k.From the results, it can be seen that the different methods have similar trends for performance for the three groups. For example, the nDCG scores for weak students are always higher than those for merit students, which means it is easier to make predictions for weak students. Students whose performance varies depending on the difficulty of the questions will have a better score.Then, we ran the experiment again and evaluated the stability of our method for various topk results. As shown in Figure 9, the performance of KFRank is generally better and more stable than Random Forest and LambdaMART in traditional learning-to-rank methods. From the results of KFRank, KFRank is superior in the performance of average and weak students, indicating that KFRank has greater help for students with poor knowledge.Figure 9
Comparison of LambdaMART, Random Forest, and KFRank. (a) Merit student. (b) Average student. (c) Weak student.
(a)(b)(c)


## 4.1. Datasets

We conducted experiments to validate the performance of our method using a real-life dataset (Table1), from two applications in pad: teacher client and student client. The data spanned the period from September 2017 to June 2018. There were nearly 70 million records for 40,000 students. Based on the student’s historical activity and correctness rate, we selected a record of 300 students with relatively high quality of records. Then, we apportion the data into training and test sets, with a 70-30 split. After that, excellent teachers create the mathematics knowledge graph of middle school, which contains more than 700 nodes and 2000 relationship edges. Finally, we use TransR approach to embed knowledge topic for learning.Table 1
Summary of the dataset.
MetricMeritAverageWeakNumber of students100100100Number of quizzes16,25121,68424,418Number of knowledge points354440401Number of records65,54668,01173,232Ratio correct0.920.610.21To evaluate the effectiveness of the ranking, we split the students into three groups:merit students, average students, and weak students. In the dataset, almost 72 percent of the questions covered three or more knowledge points and less than 14 percent of the questions had only one knowledge point. The distributions of rewards for the three groups of students are shown in Figure 8.Figure 8
Distribution of rewards for three groups of students. (a) Merit student. (b) Average student. (c) Weak student.
(a)(b)(c)


## 4.2. Evaluation Criteria

We first calculate the score of each question for ranking by the performance result of students and difficulty of question. For example, for a given question, only 60 percent of students could choose the right answer, the difficulty of this question is 0.4. Then, we also give the correct and wrong answer with 5 and 1 score. Finally, students will get 2 points if they do the right questions, and only 0.4 if they make a mistake. For each assignment, according to the student’s answer to each question, the rank score is calculated as the true value.We usenDCG@k to measure the performance. To get nDCG@k, we first calculate DCG@k:(11)DCGf=∑r=1kvr1log1+r,where r is the rank of items in the recommendation list, k is the length of the recommendation list, f is the ranking function or algorithm, vr is the value of the rth item, and 1/log1+r is the discount. iDCG is an ideal discounted cumulative gain, iDCG@k is also needed and calculated in a similar way. In iDCG@k, the questions in the recommendation list are ranked by their original values instead of by the ranking algorithm.


## 4.3. Experimental Results

Table2 compares the performance of our model with other methods using nDCG@5 and nDCG@10. The higher the score, the better the performance. Of the LTR methods, CoordAscent, LambdaMART, and ListNet perform relatively well and better than the original reinforcement-learning method, MDPRank. KFRank with updated environments has the best nDCG value for nearly all groups of students.Table 2
nDCG of ranking methods.
MethodMerit studentAverage studentWeak student@5@10@5@10@5@10MART [14]0.7010.7510.3090.3190.2720.384RankNet [15]0.5070.5470.2980.3020.4330.509RankBoost [16]0.3730.3180.2080.2560.3690.403AdaRank [17]0.2880.3620.1730.2280.1960.391CoordAscent [18]0.4660.6810.2410.2930.4940.539LambdaRank [19]0.1560.1550.2350.2130.4290.495LambdaMART [20]0.6460.7490.2790.3920.5150.569ListNET [11]0.4330.6260.1910.2400.4700.464RandForest [21]0.7310.7810.3830.3660.2980.431MDPRank [6]0.3440.4410.4130.5410.5680.642KFRank0.4250.4440.5990.5650.6360.686Bold values represent the best performance of nDCG@k.From the results, it can be seen that the different methods have similar trends for performance for the three groups. For example, the nDCG scores for weak students are always higher than those for merit students, which means it is easier to make predictions for weak students. Students whose performance varies depending on the difficulty of the questions will have a better score.Then, we ran the experiment again and evaluated the stability of our method for various topk results. As shown in Figure 9, the performance of KFRank is generally better and more stable than Random Forest and LambdaMART in traditional learning-to-rank methods. From the results of KFRank, KFRank is superior in the performance of average and weak students, indicating that KFRank has greater help for students with poor knowledge.Figure 9
Comparison of LambdaMART, Random Forest, and KFRank. (a) Merit student. (b) Average student. (c) Weak student.
(a)(b)(c)


## 5. Related Work

Related work can be classified into two categories, those based on LTR methods and those based on reinforcement learning.


### 5.1. Making Recommendations with LTR

Karatzoglou et al. [3] explained how the key ideas of different LTR methods can be applied to specific collaborative filtering methods. Sun et al. [4] applied the LTR method RankSVM to generate a list of recommended items for users. Yao et al. [5] applied the LTR method for item recommendation and integrated social information between users in the training of the Listwise model to improve the quality of a sorted list of items. Canuto et al. [22] applied the LTR method to learn automatically how to sort labels. They compared the performance of eight different methods of recommending labels. Ifada et al. [23] developed a novel LTR method Go-Rank for a label-based project recommendation system. They directly optimized the graded average precision, resulting in an optimized list of recommended items. Huang et al. [24] reviewed recent research into recommendation algorithms based on LTR. They generalized, compared, and analyzed problem definitions, key technologies, utility evaluations, and progress. Finally, they discussed and forecast the trends for recommendation algorithms based on LTR.


### 5.2. Making Recommendations with Advanced Reinforcement Learning

Shani et al. [25] proposed an MDP-based collaborative filtering model, which uses a finite window for history, instead of an unbounded one, to define the current state. It can be regarded as approximating a partial observable MDP (POMDP). Since POMDP has high computational and representational complexity, various strategies have been suggested for simplifying it, such as policy-based optimization [26], value function approximation [27], and stochastic sampling [28]. Regarding sequential decision problems, Sunehag et al. [29] designed a reinforcement-learning agent using high-dimensional combinatorial slate-action spaces and achieved remarkable results. As ranking is a key issue in practical recommendation problems, any improvements in ranking contribute significantly to reinforcement recommendation systems. Zhang et al. [30] used a log-based document reranking modeled as a POMDP. Wei et al. [6] proposed a novel LTR model based on a MDP, referred to as MDPRank, which directly optimizes a ranking using a MDP. FAIR-PG-Rank recommends items via a policy-gradient approach which could satisfy fairness of exposure constraints with respect to items [31]. A similar idea in the article is generating unified term impact (UTI) during the indexing time and combining into a hybrid model to improve the accuracy [32]. Since the relationship between study performance and exam results is much complicated, article [33] finds the correspondence of input values and predicts targets which is not a one-to-one relation, treats the classification task as a fuzzy geometrical problem, and proposes a fuzzy similarity approach to solve the problem [34, 35].


## 5.1. Making Recommendations with LTR

Karatzoglou et al. [3] explained how the key ideas of different LTR methods can be applied to specific collaborative filtering methods. Sun et al. [4] applied the LTR method RankSVM to generate a list of recommended items for users. Yao et al. [5] applied the LTR method for item recommendation and integrated social information between users in the training of the Listwise model to improve the quality of a sorted list of items. Canuto et al. [22] applied the LTR method to learn automatically how to sort labels. They compared the performance of eight different methods of recommending labels. Ifada et al. [23] developed a novel LTR method Go-Rank for a label-based project recommendation system. They directly optimized the graded average precision, resulting in an optimized list of recommended items. Huang et al. [24] reviewed recent research into recommendation algorithms based on LTR. They generalized, compared, and analyzed problem definitions, key technologies, utility evaluations, and progress. Finally, they discussed and forecast the trends for recommendation algorithms based on LTR.


## 5.2. Making Recommendations with Advanced Reinforcement Learning

Shani et al. [25] proposed an MDP-based collaborative filtering model, which uses a finite window for history, instead of an unbounded one, to define the current state. It can be regarded as approximating a partial observable MDP (POMDP). Since POMDP has high computational and representational complexity, various strategies have been suggested for simplifying it, such as policy-based optimization [26], value function approximation [27], and stochastic sampling [28]. Regarding sequential decision problems, Sunehag et al. [29] designed a reinforcement-learning agent using high-dimensional combinatorial slate-action spaces and achieved remarkable results. As ranking is a key issue in practical recommendation problems, any improvements in ranking contribute significantly to reinforcement recommendation systems. Zhang et al. [30] used a log-based document reranking modeled as a POMDP. Wei et al. [6] proposed a novel LTR model based on a MDP, referred to as MDPRank, which directly optimizes a ranking using a MDP. FAIR-PG-Rank recommends items via a policy-gradient approach which could satisfy fairness of exposure constraints with respect to items [31]. A similar idea in the article is generating unified term impact (UTI) during the indexing time and combining into a hybrid model to improve the accuracy [32]. Since the relationship between study performance and exam results is much complicated, article [33] finds the correspondence of input values and predicts targets which is not a one-to-one relation, treats the classification task as a fuzzy geometrical problem, and proposes a fuzzy similarity approach to solve the problem [34, 35].


## 6. Conclusions

Assignment recommendation is an essential and trick task in online study research. In this paper, we investigated how to predict the performance of students by using assignment ranking mechanisms. Based on traditional learning-to-rank models, we proposed a knowledge-fusion model with an attention network named KFRank, which employs two novel features compared to previous methods: (1) an attention network for capturing multiple knowledge factors in human behavior and (2) a reinforcement-learning module for ranking questions by their predictable reward. Our model could capture both current study status and previous study performance of similar math topics. Extensive experiments on a real-world dataset with three different levels of students showed that KFRank significantly outperforms other methods in most cases.In the future, there are still some directions for further studies. First, besides the historical log, we would like to measure the study performance in more aspects, for example, by studying cognitive model. Second, for the policy strategy, some network optimization [36] and fuzzy theories could be introduced in our model [34]. Finally, as our KFRank is a general framework, we will test its performance on other disciplines (e.g., click-through rate prediction) and, meanwhile, on the similar applications in other domains, such as the user behavior of customers in e-commerce.



---

*Source: 6748430-2020-12-23.xml*