## 4 Methodology
### Multiband Attention Model  
The forgetting mechanism effectively constrains interactions within a relatively narrow range, thereby allowing the current knowledge concept to consider more recent exercise results. However, due to the varying forgetting rates for different knowledge concepts among learners, a fixed-parameter forgetting model may not necessarily fit it well. Therefore, we propose a multiband attention model, which possesses multiple effective temporal ranges of interaction to accommodate forgetting patterns with different rates. This multiband attention model naturally adapts to the multihead attention mechanism of transformer models, where different heads utilize different bands.  
The original input is, respectively, processed through a knowledge concept embedding layer and quantized question embedding layer, and it is then summed up to obtain the question representation feature \(\hat{x}_{t}\) and the response representation feature \(\hat{y}_{t}\). Next, we used a multiband attention model and masked multiband attention layers to capture the interactions between features.  
We obtained the query, key, and value for the question embeddings \(\{\hat{x}_{1},\dots,\hat{x}_{t}\}\) with the following equations:  
\[Q=\hat{X}W_{q},K=\hat{X}W_{k},V=\hat{X}W_{v}, \tag{8}\]  
where \(\hat{X}\) is the question feature matrix that is composed of vectors \(\{\hat{x}_{1},\dots,\hat{x}_{t}\}\), and \(W^{q}\), \(W^{k}\), and \(W^{v}\) are the linear projection matrices for query, key, and value, respectively.  
Figure 2: The process of quantized question embedding. We implemented quantized question embedding by creating a codebook to store quantized features. For each question embedding \(z_{t}\), the model searched for the most similar feature \(x_{t}\) in the codebook. A stop gradient operation was introduced to solve the nondifferentiable process from \(z_{t}\) to \(x_{t}\). For this module, during forward propagation, we replaced \(z_{t}\) with \(x_{t}\). During backward propagation, we replaced \(x_{t}\) with \(z_{t}\).  
The relevance of each of the previous interactions with the current question is determined by the \(m\)th band attention weights, which are defined as follows:  
\[Attention_{m}(Q,K,V)=\text{Softmax}\bigg{(}\frac{QK^{T}exp(-m\theta d(\Delta t))}{ \sqrt{D}}\bigg{)}V, \tag{9}\]  
where \(d(\Delta t)\) denotes the distance between the temporal interval \(\Delta t\), and the parameters \(\theta\) control the decay rate of the distance.  
To effectively attend to the information from various representative subspaces, we utilized the projection matrices \(W_{q}\)s, \(W_{k}\)s, and \(W_{v}\)s to linearly project the queries, keys, and values with different \(m\)s. Figure 3 illustrates the combination of multiband attention matrices with different \(m\)s from 1 to 8.  
\[Multiband(x)=(band_{1}\oplus\ldots\oplus band_{h})W_{o}, \tag{10}\]  
where \(band_{m}=Attention_{m}(\hat{X}W_{q}^{(m)},\hat{X}W_{k}^{(m)},\hat{X}W_{v}^{(m)})\), and \(W_{o}\) is a linear projection matrix.  
Similarly, the query, key, and value for the response embeddings \(\{0,\hat{y}_{1},\ldots,\hat{y}_{t-1}\}\) were expressed with the following equations:  
\[Q=\hat{Y}\hat{W}_{q},K=\hat{Y}\hat{W}_{k},V=\hat{X}\hat{W}_{v}, \tag{11}\]  
where \(\hat{Y}\) is the response feature matrix composed of vectors \(\{0,\hat{y}_{1},\ldots,\hat{y}_{t-1}\}\).  
The masked attention operation was adopted to prevent the model from seeing future answers.  
\[MaskedAttention_{m}(Q,K,V)=\text{Softmax}\bigg{(}\frac{QK^{T}exp(-m\theta d( \Delta t))\odot M}{\sqrt{D}}\bigg{)}V, \tag{12}\]  
where \(M\) is a lower triangular unit matrix, and \(\odot\) denotes element-wise multiplication.