SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shangshang Yang $^{1,3}$  Xiaoshan Yu $^{1,3}$  Ye Tian $^{1,3}$  Xueming Yan $^{2}$  Haiping Ma $^{1,3,4*}$
Xingyi Zhang $^{1,3*}$
$^{1}$ Anhui University  $^{2}$ Guangdong University of Foreign Studies
<sup>3</sup>Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education
$^{4}$ Department of Information Materials and Intelligent Sensing Laboratory of Anhui Province
{yangshang0308，yxsleo，field910921}@gmail.com xueming126@126.com
hpma@ahu.edu.cn xyzhanghust@gmail.com
Transformer has achieved excellent performance in the knowledge tracing (KT) task, but they are criticized for the manually selected input features for fusion and the defect of single global context modelling to directly capture students' forgetting behavior in KT, when the related records are distant from the current record in terms of time. To address the issues, this paper first considers adding convolution operations to the Transformer to enhance its local context modelling ability used for students' forgetting behavior, then proposes an evolutionary neural architecture search approach to automate the input feature selection and automatically determine where to apply which operation for achieving the balancing of the local/global context modelling. In the search space design, the original global path containing the attention module in Transformer is replaced with the sum of a global path and a local path that could contain different convolutions, and the selection of input features is also considered. To search the best architecture, we employ an effective evolutionary algorithm to explore the search space and also suggest a search space reduction strategy to accelerate the convergence of the algorithm. Experimental results on the two largest and most challenging education datasets demonstrate the effectiveness of the architecture found by the proposed approach.
With the rapid development of online education systems, such as Coursera and ASSISTment [10], there have been a massive amount of available data about student-system interactions recorded in these systems. Based on the students' historical learning activities over time, knowledge tracing (KT), as a fundamental task in intelligent education, aims to trace students' knowledge states on knowledge concepts, which plays an important role in both personalized learning and adaptive learning [28, 12] and thus has received growing attention from the scientific community.
Since the concept of KT was first proposed by Corbett and Anderson in Bayesian knowledge tracing (BKT) [9], considerable efforts have been devoted to developing various KT approaches, which can mainly be split into two categories. The first type of KT approaches are traditional approaches based on probabilistic models or logistic models, such as dynamic BKT [17], LFA [3], PFA [30], and KTM [42]. The second type of approaches [21] have shown significantly better performance and generalization since they utilize various deep neural networks (DNNs) to solve the KT task, such as
recurrent neural networks (RNNs) as adopted in [53], long short-term memory networks (LSTMs) as adopted in [1], graph neural networks as adopted in [26], and Transformer as adopted in [7, 33, 36].
Among these DNN-based approaches, Transformer-based KT approaches have exhibited significantly better performance due to the better ability to tackle sequence tasks. However, as demonstrated in [16, 36], when there is no extra specific information introduced, the attention mechanism in the vanilla Transformer [41] cannot capture students' forgetting behavior [24], since its providing single global context modelling usually assigns high importance to the highly related records to predict the current record even with position information. However, those related records distant in time commonly pose little influence on the current prediction. Moreover, the input features in these approaches are manually selected based on domain knowledge and simply fused as the inputs to the Transformer encoder and decoder, but the manually selected features and the simple feature fusion method may miss some valuable features and complex feature interaction information.
To address the issues for further improving the performance, we first equip the Transformer with convolutions to boost its local context modelling ability used for students' forgetting behavior, then propose an evolutionary neural architecture search (NAS) [54] approach for KT (termed ENAS-KT), which aims to automatically select input features and determine where and which operation is applied to strike the balancing of local/global context modelling. Our main contributions are as:
- To the best of our knowledge, we are the first to apply NAS to search the Transformer architecture for the KT task. Here, the proposed ENAS-KT aims to find the best architecture that holds the optimal selected input features and the ideal balancing of the local/global context modelling, which can model students' forgetting behavior well and thus hold significantly good prediction performance on KT.
- In the proposed ENAS-KT, we design a novel search space for the Transformer and devise an effective evolutionary algorithm (EA) as the search approach to explore the search space. The search space not only considers the input feature selection of the Transformer but also equips it with local context modelling by replacing the original global path containing the attention module with the sum of a global path and a local path that could contain different convolutions. Besides, an effective hierarchical fusion method is suggested to fuse selected features. The proposed ENAS-KT not only employs a trained super-net for fast solution evaluation but also suggests a search space reduction strategy to accelerate the convergence.
- Experimental results validated on two largest and most challenging datasets show that the architecture found by the proposed approach holds significantly better performance than state-of-the-art (SOTA) KT approaches.
2.1 Transformer-based Knowledge Tracing
Existing Transformer-based KT approaches focus on taking different features as the inputs to the Transformer encoder and decoder based on human expertise without changing the model architecture. In [7], Choi et al. proposed to directly employ the vanilla Transformer for KT, namely SAINT, where the exercise embedding and knowledge concept embedding are taken as the input of the Transformer encoder part and the response embedding is taken as the input to the Transformer decoder part. As a result, the SAINT achieves significantly better performance than other KT approaches. To model the forgetting behavior in students' learning, Pu et al. [33] further fed the timestamp embedding of each response record to the Transformer model. Besides, to model the structure of interactions, the exercise to be answered at the next time is also fed to the model. As the successor of the SAINT, due to the problem in SAINT that students' forgetting behavior cannot be modelled well by the global context provided by the attention mechanism, the SAINT+ [36] aims to further improve the performance of KT by mitigating the problem. To this end, two temporal feature embedding, the elapsed time and the lag time, are combined with response embedding as input to the decoder part.
Different from above two approaches that only focus on how to manually select input features to alleviate the forgetting behavior problem, this paper focuses on not only realizing the input feature automatic selection of the Transformer but also the Transformer architecture automatic design by proposing an EA-based NAS approach, where convolution operations are considered in the Transformer to enhance its local context ability used for students' forgetting behavior.
2.2 Neural Architecture Search for Transformer
NAS has been applied to many domains [13], such as convolution neural networks in computer vision [54], RNNs in natural language processing [46] or automatic speech recognition [2].
So et al. [38] are the first to apply NAS to design the Transformer architecture, where an EA is employed to explore the devised NASNet-like search space consisting of two stackable cells for translation tasks. To save computation resources, a progressive dynamic hurdles method is employed to early stop the training of poor performance architectures. Inspired by this work, there have been a series of NAS work proposed to search the Transformer architecture in different research fields. For NLP, the representative approaches include NAS-BERT [49] and HAT [45]. For ASR, the representative approaches include the LightSpeech [23], and DARTS-CONFORMER [35]. For CV, the representative approaches include the AutoFormer [6], GLiT [5], and ViTAS [39]. The Transformer architectures found by these approaches have demonstrated significantly better performance than the SOTA competitors in their own domains.
Despite the emergence of these NAS approaches for the Transformer, their designed search spaces for different domains cannot be directly used for solving the feature selection problem and the problem of balancing local context and global context in this paper. To the best of our knowledge, we are the first to apply the NAS to search the Transformer architecture for the KT task. Different from NAS-Cell [11] that directly applies existing NAS methods used for general LSTMs to KT, our work designs effective search space for the Transformer to strengthen its modeling capability in KT.
3.1 Knowledge Tracing Problem
Given a student's response records on a set of exercises over time, i.e., a sequence of interactions denoted by  $I = \{I_1, I_2, \dots, I_n\}$ , the KT task [21] aims to predict the probability that student answers correctly on the exercise in the next interaction  $I_{n+1}$ .  $I_i = (e_i, o_i, dt_i, r_i)$  is the  $i$ -th interaction of the student,  $e_i$  denotes the  $i$ -th exercise assigned to the student, and  $o_i$  contains other related information about the interaction  $I_i$ , such as the timestamp, the type of exercise  $e_i$ , knowledge concepts in  $e_i$ , and so on.  $dt_i$  represents the elapsed time the student took to answer,  $r_i \in \{0, 1\}$  is the student's response/answer on exercise  $e_i$ , where  $r_i = 1$  means the answer is correct otherwise the answer is incorrect.
Based on the interactions before the  $t + 1$ -th interaction  $I_{t + 1}$ , the KT focuses on predicting the probability of the student answering correctly on exercise  $e_{t + 1}$ , formally:
$$
P \left(r _ {t + 1} \mid I _ {1}, I _ {2}, \dots , I _ {t}, \left\{e _ {t + 1}, o _ {t + 1} \right\}\right), \tag {1}
$$
where the response  $r_{t + 1}$  and the elapsed time  $dt_{t + 1}$  cannot be accessed for the prediction of  $I_{t + 1}$ .
3.2 Transformer-based KT
Transformer [41] consists of an encoder part and a decoder part, where each part is composed of a stack of  $N$  identical blocks, and each block has a multi-head self-attention (MHSA) module and a position-wise feed-forward network (FFN) module. Then, each encoder block can be expressed as
$$
\operatorname {b l o c k} _ {E n}: h = \mathrm {L N} (\mathrm {F F N} (h) + h), h = \mathrm {L N} (\mathrm {M H S A} (X, X, X) + X), \tag {2}
$$
where  $X$  is the input of the encoder block, and  $\mathrm{LN}(\cdot)$  is layer normalization [48]. Each decoder block can be denoted by
$$
\operatorname {b l o c k} _ {D e} \left\{ \begin{array}{l} h = \operatorname {L N} (\text {M a s k e d M H S A} (X, X, X) + X) \\ h = \operatorname {L N} (\operatorname {F F N} (h) + h), h = \operatorname {L N} (\operatorname {M H S A} (h, O _ {E n}, O _ {E n}) + h) \end{array} , \right. \tag {3}
$$
where Masked_MHSA(\cdot) is the MHSA(\cdot) with the mask operation to prevent current position to attend the later positions, and  $O_{En}$  is the last encoder block's output.
For the Transformer-based KT approaches, the following sequential inputs  $X_{input}$  with length of  $L$ :
$$
X _ {i n p u t} = \{E, O, D T, R \} = \{E = \left\{e _ {1}, e _ {2}, \dots , e _ {L} \right\}, \tag {4}
$$
$$
O = \left\{o _ {1}, o _ {2}, \dots , o _ {L} \right\}, D T = \left\{d t _ {\text {s t a r t}}, d t _ {1}, \dots , d t _ {L - 1} \right\}, R = \left\{r _ {\text {s t a r t}}, r _ {1}, \dots , r _ {L - 1} \right\}
$$
Figure 1: The proposed search space. There are five candidate operations for the local operation and three operations for the global operation.
will be first mapped into a set of embedding  $X_{embed} = \{ \text{embed}_i \in R^{L \times D} | 1 \leq i \leq Num \}$ , where  $D$  is the embedding dimension,  $Num$  is the number of available features in a dataset,  $dt_{start}$  and  $r_{start}$  are the start tokens.  $O$  could contain multiple features and thus produce multiple embedding. Then, two sets of embedding  $X_{En}$  and  $X_{De}$  will be selected from  $X_{embed}$  and fed to the encoder part and decoder part, respectively. Finally, the decoder will output the predicted response  $\hat{r} = \{ \hat{r}_1, \hat{r}_2, \dots, \hat{r}_L \}$  as
$$
\hat {r} = \operatorname {D e c o d e r} \left(\operatorname {F u s i o n} \left(X _ {D e}\right), O _ {E n}, O _ {E n}\right), O _ {E n} = \operatorname {E n c o d e r} \left(\operatorname {F u s i o n} \left(X _ {E n}\right)\right), \tag {5}
$$
where  $\text{Fusion}(\cdot)$  is a fusion method to get a  $L \times D$  tensor.  $\text{Encoder}(\cdot)$  and  $\text{Decoder}(\cdot)$  denote the forward passes of the encoder part and decoder part, whose each block's forward pass is as shown in (2) or (3). Due to the nature of KT task, a mask operation is necessary for each MHSA module to prevent current position from accessing the information in later positions. Thus the MHSA module mentioned in subsequent sections refers to Masked_MHSA( $\cdot$ ).
Existing Transformer-based KT approaches empirically fuse the exercise-related embedding and response-related embedding as the inputs to encoder and decoder parts, respectively. However, the adopted feature fusion method is simple and the selected feature embedding sets may be not the best choice. Therefore, we further employ NAS to realize the input feature selections of both encoder and decoder parts and suggest an effective hierarchical feature fusion.
4 The Proposed ENAS-KT
To achieve the automatic input feature selection and automate the balancing of local/global modelling, we design a novel search space for the Transformer, and propose an evolutionary algorithm to search the optimal architecture. The following two subsections will present the details.
4.1 Search Space Design
We first replace post-LN [48] with the pre-LN [19] to stabilize the training of the Transformer. Then, we design a novel search space for the Transformer, which has been presented in Fig. 1 and there exist three important designs: (1) A selective hierarchical input module; (2) The sum of a global path and a local path to replace the original global path; (3) The global operations containing MHSA module, FFN module, and zero operation.
4.1.1 Selective Hierarchical Input Module
Given all available input features  $X_{input} = \{E, O, DT, R\}$  in a student's response records, we can obtain each candidate embedding in  $X_{embed} = \{embed_i \in R^{L \times D} | 1 \leq i \leq Num\}$  by categorical embedding or continuous embedding. For example, the categorical embedding of  $E \in R^{1 \times L}$  can be computed by
$$
\operatorname {e m b e d} _ {\text {c a t e}} = \operatorname {o n e - h o t} (E) * \mathbf {W} _ {\text {c a t e}}, \mathbf {W} _ {\text {c a t e}} \in R ^ {\text {n u m} _ {E} \times D}, \tag {6}
$$
where  $num_E$  is the number of all exercises, one-hot(\cdot) outputs a  $L \times num_E$  one-hot-vector matrix, and  $\mathbf{W}_{\text{cate}}$  denotes the trainable parameters of the embedding layer. And the continuous embedding of  $DT \in R^{1 \times L}$  can be computed by
$$
e m b e d _ {c o n t} = D T ^ {T} * \mathbf {W} _ {c o n t}, \mathbf {W} _ {c o n t} \in R ^ {1 \times D}. \tag {7}
$$
Based on all candidate embedding  $X_{embed}$ , a selective hierarchical input module is proposed to first select two sets of suitable embedding from  $X_{embed}$  and then fuse the selected embedding as the inputs to encoder and decoder, respectively. To this end, we employ two binary vectors to determine which candidate embedding is selected:
$$
\mathbf {b} _ {\mathbf {E n}} = \left(b _ {i} \in \{0, 1 \}\right) ^ {1 \times N u m}, 1 \leq s u m \left(\mathbf {b} _ {\mathbf {E n}}\right), \mathbf {b} _ {\mathbf {D e}} = \left(b _ {i} \in \{0, 1 \}\right) ^ {1 \times N u m}, 1 \leq s u m \left(\mathbf {b} _ {\mathbf {D e}}\right), \tag {8}
$$
where  $1 \leq i \leq Num$  and  $1 \leq sum(\mathbf{b_{En}})$  ensures at least one embedding is selected.  $b_i = 1$  means embedding  $embed_i$  in  $X_{embed}$  is selected and  $b_i = 0$  otherwise. By doing so, two sets of selected embedding  $X_{embed}^{En}$  and  $X_{embed}^{De}$  used for the encoder and decoder parts can be obtained by
$$
X _ {E n} = \left\{e m b e d _ {i} \in X _ {e m b e d} \mid b _ {i} = 1, b _ {i} \in \mathbf {b} _ {\mathbf {E n}} \right\}, X _ {D e} = \left\{e m b e d _ {i} \in X _ {e m b e d} \mid b _ {i} = 1, b _ {i} \in \mathbf {b} _ {\mathbf {D e}} \right\}, \tag {9}
$$
Instead of directly applying the simple sum or concatenation operation to  $X_{En}$ , we propose a hierarchical fusion method to fully aggregate the selected embedding  $X_{En}$  as
$$
\operatorname {H i e r} \left(X _ {E n}\right): \operatorname {i n p u t} _ {E n} = \operatorname {c o n c a t} (\text {t e m p}) * \mathbf {W} _ {\text {o u t}}, \operatorname {t e m p} = \cup_ {x _ {i}, x _ {j} \in X _ {E n}} ^ {x _ {i} \neq x _ {j}} \sigma \left(\operatorname {c o n c a t} ([ x _ {i}, x _ {j} ]) * \mathbf {W} _ {i j}\right), \tag {10}
$$
where  $\text{concat}(\cdot)$  is concatenation operation,  $\sigma(\cdot)$  is the tanh [14] activation function,  $\mathbf{W}_{ij} \in R^{2D \times D}$  and  $\mathbf{W}_{out} \in R^{|temp|*D \times D}$  are trainable matrices, and  $|temp|$  denotes the number of temporary features in temp. The first item in equation (10) is to combine the selected embedding in pairs to fully fuse a pair of embedding without the influence of other embedding, which is inspired by the feature fusion in multi-modal tasks [25]. After obtaining temp, the second item in equation (10) is to directly apply the concatenation operation to temp followed by a linear transformation to get the final input  $\text{input}_{En}$  to the encoder part. The input  $\text{input}_{De}$  to the decoder part can be obtained in the same way.
As shown in Fig. 1, the size of  $\mathbf{W}_{out}$  is fixed as  $\frac{\text{Num}*(\text{Num}-1)*D}{2} \times D$  for easily training superTransformer, but the temporary features related to non-selected embedding are set to zeros. For example, feature 2 is not selected, thus the 1st, 4th, and 5th temporary features are set to zeros.
4.1.2 The Global Path and The Local Path
Since students' forgetting behaviors in the KT cannot be directly modelled well by the single MHSA module, we aim to incorporate the Transformer with the local context modelling. Thus, we split the original global path containing the MHSA module in each encoder block into the sum of two paths: one path is the original global path, and another path is the newly added local path containing convolution operations. Each decoder block is also set to be composed of these two paths. After applying the pre-LN to the input  $X$  of each block as  $x_{1} = \mathrm{LN}(X)$ , each encoder and decoder block's forward pass in (2) and (3) can be rewritten as
$$
b l o c k _ {E n} \left\{\begin{array}{l}g p = \mathrm {L N} \left(x _ {1} + \mathrm {M H S A} \left(x _ {1}, x _ {1}, x _ {1}\right)\right)\\l p = \mathrm {L N} \left(x _ {1} + L O \left(x _ {1}\right)\right)\\h = h + \operatorname {F F N} (h), h = l p + g p\end{array}\quad \text {a n d} b l o c k _ {D e} \left\{\begin{array}{l}g p = \mathrm {L N} \left(x _ {1} + \mathrm {M H S A} \left(x _ {1}, x _ {1}, x _ {1}\right)\right)\\g p = g p + \mathrm {M H S A} \left(g p, O _ {E n}, O _ {E n}\right)\\l p = \mathrm {L N} \left(x _ {1} + L O \left(x _ {1}\right)\right)\\h = h + \operatorname {F F N} (h), h = l p + g p\end{array}, \right.\right.\left. \right.
$$
where  $h$  refers to the output,  $gp$  is the latent representation obtained by the global path,  $lp$  is the latent representation obtained by the local path, and  $LO(\cdot)$  denotes a local operation used for the local path.
As shown in Fig. 1, there are five candidate operations for  $LO(\cdot)$ , including four one-dimension convolution operations with kernel sizes in  $\{3,5,7,11\}$  and a zero operation that returns a zero tensor. When the zero operation is used for  $LO(\cdot)$ , the block $_{En}$  and block $_{De}$  will degenerate to the encoder and decoder block in the vanilla Transformer.
4.1.3 The Global Operation
As demonstrated in the Sandwich Transformer [32], reordering the MHSA model and FFN module may lead to better performance. For this aim, a global operation is used to replace the first MHSA
Figure 2: The overview of the proposed ENAS-KT.
module and the FFN module of each encoder and decoder block in (11), where  $\mathrm{MHSA}(\cdot)$  and  $\mathrm{FFN}(\cdot)$  are replaced by  $GO_{1}(\cdot)$  and  $GO_{2}(\cdot)$ . Here,  $GO_{1}(\cdot)$  and  $GO_{2}(\cdot)$  represent the adopted global operation. As shown in the light orange area on the right of Fig. 1, there are three candidate operations for  $GO_{1/2}(\cdot)$ , including the MHSA module, FFN module, and a zero operation.
With the above modifications, the proposed Transformer architecture predicts the response  $\hat{r} = \{\hat{r}_1,\hat{r}_2,\dots ,\hat{r}_L\}$  as
$$
\hat {r} = \operatorname {D e c o d e r} \left(\operatorname {H i e r} \left(X _ {D e}\right), O _ {E n}, O _ {E n}\right), O _ {E n} = \operatorname {E n c o d e r} \left(\operatorname {H i e r} \left(X _ {E n}\right)\right), \tag {12}
$$
where  $X_{En}$  and  $X_{De}$  are obtained based on  $\mathbf{b}_{\mathbf{En}}$ ,  $\mathbf{b}_{\mathbf{De}}$ , and  $X_{embed}$  as shown in (8) and (9).
4.1.4 Encoding Scheme
Under the proposed search space, there theoretically exist  $Num^2 * 2^{2(Num - 1)} * (3 * 3 * 5)^{2N}$  candidate Transformer architectures in total, where each candidate Transformer architecture  $\mathcal{A}$  can be encoded by  $\mathcal{A} = [\mathbf{b_{En}}, \mathbf{b_{De}}, (lo \in \{0, \dots, 4\}, go_1, go_2 \in \{0, 1, 2\})^{1 \times 2N}]$ .
Here  $\mathbf{b_{En}}$  and  $\mathbf{b_{De}}$  denote which embedding is selected for the inputs to encoder and decoder parts, and  $2N$  triplets  $(lo,go_1,go_2)$  determine which operations are adopted for each block, where  $lo,go_1,$  and  $go_2$  determine which operation is used for  $LO(\cdot),GO_1(\cdot)$  ,and  $GO_2(\cdot)$
Next, we search the optimal Transformer architecture by solving the following optimization problem:
$$
\max  _ {\mathcal {A}} \mathbf {F} (\mathcal {A}) = f _ {\text {v a l} _ {\text {a u c}}} (\mathcal {A}, D _ {\text {v a l}}), \tag {13}
$$
where  $f_{val\_auc}(\mathcal{A}, D_{val})$  denotes the AUC (area under the curve) value of  $\mathcal{A}$  on dataset  $D_{val}$ .
4.2 Search Approach Design
Overview. An overview of the proposed approach is shown in Fig. 2, where a super-Transformer is first trained and then an EA with  $Pop$  individuals and  $Gen$  generations is used for searching.
4.2.1 Super-Transformer Training
Inspired by the sandwich rule [51], we propose to make the training rule adapt to our superTransformer by utilizing three sub-models in one batch of training of the super-Transformer, including a global sub-model  $sub_{g}$ , a local sub-model  $sub_{l}$ , and a randomly sampled sub-model  $sub_{r}$ .  $sub_{g}$  consists of pure global operations but no local operations, whose  $\mathbf{b}_{\mathrm{En}}$  and  $\mathbf{b}_{\mathrm{De}}$  are randomly generated; similarly,  $sub_{l}$  consists of only convolution operations with the smallest kernel size.
4.2.2 Search Space Reduction-based Evolutionary Search