# Student Behavior Prediction of Mental Health Based on Two-Stream Informer Network
## 3 Proposed Method
### Encoding Layer  
In the encoding layer, a new backbone network is proposed that is composed of the Time Encoder, Behavior Encoder and its Intermediate Fusion Module. The Time Encoder extracts the sequence feature information from the time dimension and uses the informer encoder using a mask to obtain the time cycle trend. The Behavior Encoder extracts the behavior feature information from the behavior dimension and uses the informer encoder to obtain the behavior dependency. In the Intermediate Fusion Module, the features extracted by each Time Encoder and the features extracted by the Behavior Encoder are fused, and  
Figure 1: Model architecture of the two-stream informer network.  
then a convolution layer with convolution kernel of 1 is passed through. After the whole process is repeated four times, the encoder features of the two channels are fused as the output of the backbone network.  
The key part of the Time Encoder and Behavior Encoder is the self-attention module. ProbSparse self-attention [34] is a kind of improved self-attention that can reduce the quadratic computational complexity of the self-attention mechanism and solve the classification prediction problem of long sequence data. Its overall architecture is shown in Figure 2a.  
In the following, \(q_{i}\), \(k_{i}\) and \(v_{i}\) respectively represent the \(i\)-th line in \(Q\), \(K\) and \(V\). According to the equation of [PERSON] et al. [35], the attention of the \(i\)-th query is defined as a kernel smoother in probabilistic form:  
\[P\big{(}k_{j}|q_{i}\big{)}=\frac{k\big{(}q_{i},\ k_{j}\big{)}}{\sum_{l}k(q_{i},\ k_{l})} \tag{2}\]  
According to probability \(P\big{(}k_{j}|q_{i}\big{)}\) to combine values \(v\) and obtain the final output,  
\[\pi(q_{i},\ K,\ V)=\sum_{j}\frac{k\big{(}q_{i},\ k_{j}\big{)}}{\sum_{l}k(q_{i},\ k_{l})}v_{j}=\mathbb{E}_{p(k_{j}|q_{i})}\big{[}v_{j}\big{]} \tag{3}\]  
Based on the long tail property, the key part is to calculate the distance between the probability distribution and the discrete uniform distribution of the query and key dot product pairs, so as to screen out the dominant dot product pairs. The Kullback-Leibler divergence attention is used to calculate the probability distribution \(P\big{(}k_{j}|q_{i}\big{)}=\frac{k\big{(}q_{i},\ k_{j}\big{)}}{\sum_{l}k(q_{i},\ k_{l})}\) with uniform distribution \(P\big{(}k_{j}|q_{i}\big{)}=\frac{1}{L_{k}}\) similarity.  
\[\overline{M}(q_{i},\ K) = max\Bigg{\{}\frac{q_{i}k_{j}^{T}}{\sqrt{d}}\Bigg{\}}-\frac{1}{L _{k}}\sum_{j=1}^{L_{k}}\frac{q_{i}k_{j}^{T}}{\sqrt{d}} \tag{4}\]  
Figure 2: (**a**) ProbSparse self-attention; (**b**) multi-head attention layer.  
If the case of \(i\)-th query increases \(\overline{M}(q_{i},\ K)\), its probability distribution and attention should be more diverse, and have a higher probability in the long tail distribution header fields contained in the edge of the dot product. For each \(k\) value, only the leading query of top-\(u\) is selected:  
\[\pi(\text{Q, K, V})=Softmax\left(\frac{\overline{Q}K^{T}}{\sqrt{d}}\right)V \tag{5}\]  
where \(\overline{Q}\) is a sparse matrix containing only \(u\) queries.  
Compared with single-head self-attention, multi-head self-attention can capture different information through different projection spaces, which enhances the feature extraction ability of the encoder. The structure of multiple self-attention layers is shown in Figure 2b. It is sent to each self-attention layer through a linear projection. The information obtained from the model input via linear projection is sent to the attention layer with the following output:  
\[\begin{split}& h_{i}=\pi(Q,\ K,\ V)\\ &\text{MultiHead}(\text{Q, K, V})=W_{proj}concat(h_{1},\ h_{2},\ \dots,\ h_{h})+b_{proj}\end{split} \tag{6}\]