SlowGuess/NeurIPS_2023_OCR huggingface dataset:
After training the supernet, a tailored EA is employed for searching by solving the problem in (13). The proposed EA takes the binary tournament selection [37] and single-point crossover together with the bit-wise mutation [40] to mate and generate individuals, where a search space reduction strategy is suggested to reduce the search space for accelerating the convergence at each generation.
Search Space Reduction Strategy. This strategy aims to iteratively remove some worse operations from the search space based on all individuals' fitness statistics to speed up the convergence. The item on each bit of  $\mathcal{A}$  is an operation set, such as the item on first bit is  $\{0,1\}$ , and the items on
Table 1: Statistics of the two largest education datasets for knowledge tracing, EdNet and RAIEd2020.
Datasets/Statistics	# of interactions (# of tags)	# of students (# of tag-sets)	# of exercises (# of bundles)	# of skills (# of explanation)
EdNet/RAIEd2020	95,293,926/99,271,300 (302)/(189)	84,309/393,656 (1,792)/(1,520)	13,169/13,523 (9,534)/(-)	7/7 (-)/(2)
Table 2: The comparison between ENAS-KT and compared approaches in term of model performance and parameters. The results under splitting setting of  $80\%$  and  $20\%$  on EdNet and RAIEd2020 are reported in terms of RMSE, ACC, and AUC, averaged over 5-fold cross-validation. The Wilcoxon rank sum test with a significance level  $\alpha = 0.05$  is used for analyzing the significance, where  $+,$ ,  $-$ , and  $^{\prime}\approx$  denote Ours is significantly better than, worse than, and similar to the compared approach, respectively. "M" is short for "million" in "Param.(M)", counting the number of model parameters.
Dataset	Metric	DKT	HawkesKT	CT-NCM	SAKT	AKT	SAINT	SAINT+	NAS-Cell	Ours
Param.(M)	EdNet	0.13495	0.019578	1.9974	2.0864	1.2330	2.7492	3.1862	1.8692	3.8232
RAIEd2020	0.13531	0.019932	2.0431	2.1317	1.2335	2.7945	3.2315	1.9145	4.1262
EdNet	RMSE ↓	0.4653	0.4475	0.4364	0.4405	0.4399	0.4322	0.4285	0.4345	0.4209
ACC ↑	0.6537	0.6888	0.7063	0.6998	0.7016	0.7132	0.7188	0.7143	0.7295
AUC ↑	0.6952	0.7487	0.7743	0.7650	0.7686	0.7825	0.7916	0.7796	0.8062
RAIEd2020	RMSE ↓	0.4632	0.4453	0.4355	0.4381	0.4368	0.4310	0.4272	0.4309	0.4196
ACC ↑	0.6622	0.6928	0.7079	0.7035	0.7076	0.7143	0.7192	0.7167	0.7313
AUC ↑	0.7108	0.7525	0.7771	0.7693	0.7752	0.7862	0.7934	0.7839	0.8089
+/-/≈ (six results totally)	6/0/0	6/0/0	6/0/0	6/0/0	6/0/0	6/0/0	6/0/0	6/0/0	-
the bits related to  $lo$  are  $\{0,1,2,3,4\}$ . To start with, combine individuals in the population and offspring population; secondly, record each operation fitness in list  $SF$ , where each operation fitness is obtained by averaging the fitness of the individuals containing the corresponding operation; thirdly, compute fitness standard deviation of each each operation set as list  $SF_{std}$ ; next, based on  $SF$  and  $SF_{std}$ , the worst two operations will be deleted, and the worst operation in the operation set with the largest standard deviation will be also removed, where there is at least one operation kept for each bit. By doing so, the ever-reducing search space makes the search process easier. More details (about the super-net training, the EA, and the reduction strategy) can be found in Appendix A.
Datasets. For convincing validation, two largest and most challenging real-world education datasets EdNet [8] and RAIEd2020 [34] were used in experiments, whose statistics are summarized in Table 1.
In addition to the above five feature embedding in Table 1: exercise exerc, skill sk, tag tag, tag-set tagset, and bundle/explanation bund/expla, there are other seven feature embedding in candidate input embedding  $X_{embed}$  ( $Num = 12$ ), including answer embedding ans, continuous embedding for elapsed time cont_ela, categorical embedding for elapsed time in terms of seconds cate_ela_s, continuous embedding for lag time cont_lag, categorical embedding for lag time in terms of seconds, minutes, and days (termed cate_lag_s, cate_lag_m, and cate_lag_d).
Compared Approaches. The following eight SOTA KT approaches were used for comparison:
- LSTM-based and Hawkes-based methods: DKT [31], HawkesKT [44], CT-NCM [24], and NAS-Cell [11]. HawkesKT and CT-NCM consider extra students' forgetting behaviors in modelling, and NAS-Cell is NAS-based method for searching LSTMs' cells.
- Attention mechanism-based neural network methods: SAKT [27] and AKT [16]. Here AKT considers extra students' forgetting behavior in its attention weights.
- Transformer-based methods: SAINT [7] and SAINT+ [36]. But SAINT+ considers extra features of elapsed time and lag time in the decoder input to mitigate the forgetting behavior.
Parameter Settings. According to [7, 36], all students were randomly split into  $70\% /10\% /20\%$  for training/validation/testing. The maximal length of input sequences was set to 100  $(L = 100)$ , we truncated student learning interactions longer than 100 to several sub-sequences, and 5-fold cross-validation was used. The number of blocks  $N = 4$ , embedding size  $D = 128$ , the hidden dimension of FFN was set to 128. To train the supernet, epoch number, learning rate, dropout rate, and batch size was set to 60, 1e-3, 0.1, and 128, and the number of warm-up steps in Noam scheme [7] was set to 8,000. To train the best architecture, epoch number and the number of warm-up steps was set to 30 and 4,000, and others kept same as above. Pop and Gen was set to 20 and 30. For a fair comparison, parameter settings of compared approaches are same as those in their original papers. We adopted AUC, accuracy (ACC), and root mean squared error (RMSE) as evaluation metrics. All experiments were implemented with PyTorch and run under NVIDIA 3080 GPU. The source code of the proposed approach is publicly available at https://github.com/DevilYangS/ENAS-KT.
5.2 Overall Results
To verify the effectiveness of the proposed approach, Table 2 summarizes the performance of the proposed ENAS-KT and all comparison KT approaches on predicting students' responses across EdNet and RAIEd2020 datasets, where the average RMSE, ACC, and AUC values over 5-fold cross-validation are reported and the best results are in bold. For more convincing, the Wilcoxon rank sum test [47] with a significance level  $\alpha = 0.05$  is used for analyzing the significance. In addition, the number of model parameters for each approach is reported for profound observations.
From Table 2, we can obtain the following five observations: (1) The performance of Transformer-based KT approaches is significantly better than LSTM-based KT approaches and attention mechanism-based KT approaches. (2) From the comparisons between DKT and CT-NCM, SAKT and AKT, SAINT and SAINT+, it can be found that considering the forgetting behavior in the KT can indeed improve the prediction performance to some extent under whichever type of neural architecture. (3) The RMSE, ACC, and AUC values achieved by the architecture obtained by the proposed approach on both two datasets are significantly better than that achieved by comparison approaches, which indicates the architecture found by the proposed approach is more effective in tracing students' knowledge states. (4) Compared to NAS-Cell, the architectures found by the proposed approach are more effective, which demonstrates the effectiveness of the proposed search space and search algorithm. (5) In terms of model parameters, HawkesKT is the best, DKT is the second-best, while the architectures found by the proposed ENAS-KT are worse than other KT models. Actually, high model parameters of Ours are attributed to its input part, which is used to get the input embeddings and aggregate these embeddings.
To analyze the influence of model parameters on the performance of the best-found architecture, four parameter constraints (including 2M, 2.5M, 3M, and 3.5M) are added to the proposed ENAS-KT, respectively, where the parameters of all generated architectures cannot exceed the used parameter constraint. As a result, four architectures are found on EdNet, denoted as ENAS-KT(2M), ENAS-KT(2.5M), ENAS-KT(3M), and ENAS-KT(3.5M), and Table 3 gives a brief comparison between these found architectures some KT models. As can be seen, the proposed ENAS-KT can still find good potential architecture under the parameter constraint, and the found architecture's performance increases with model parameters increasing, but the increasing in model parameters is caused by the change of the found model architecture, not the embedding or hidden size. Therefore, we can conclude that the effectiveness of the proposed approach is mainly not attributed to its high model parameters, to some extent, which can be mainly attributed to the suggested search space. Note that the comparison between SAINT/SAINT+ and SAINT(more)/SAINT+(more) can indirectly validate the above conclusion that only increasing the model parameters to improve model performance does not work and is unworthy, which further demonstrates the effectiveness of the proposed search space. For more convincing, more results under other splitting settings can be found in Appendix B.
5.3 Search Result Analysis
Architecture Visualization. The best-found architectures are plotted in Fig. 3. We can find that these two best-found architectures have very similar structures, their encoder parts prefer the convolution operations and FFN modules, especially for their first encoder blocks, while their decoder parts prefer the global operation: the MSHA module, especially for the decoder blocks close to the output, where the convolution operations with large kernel sizes are also preferred. To investigate the observation,
Table 3: The comparison on EdNet dataset in terms of AUC and the number of model parameters: some KT approaches and best architectures found by ENAS-KT with different parameter constraints. SAINT(more) and SAINT+(more) refer to two models with larger hidden and embedding sizes (192).
Model	Param.(M)	AUC	Model	Param.(M)	AUC	Model	Param.(M)	AUC
ENAS-KT(2M)	1.938	0.8021	CT-NCM	1.997	0.7743	SAINT	2.749	0.7825
ENAS-KT(2.5M)	2.389	0.8047	SAKT	2.086	0.7650	SAINT+	3.186	0.7916
ENAS-KT(3M)	2.918	0.8056	AKT	1.233	0.7686	SAINT(more)	4.910	0.7828
ENAS-KT(3.5M)	3.489	0.8059	NAS-Cell	1.869	0.7796	SAINT+(more)	5.627	0.7921
(a) Searched on EdNet.
(b) Searched on RAIEd2020.
Figure 3: The best architectures found on EdNet (a) and RAIEd2020 (b), here we simplify the connections in each encoder/decoder block for better observation.
we further transfer the best architecture found on one dataset to another datasets. The best architecture searched on EdNet dataset achieves an AUC value of 0.8078 on RAIEd2020 dataset, which is very close to the best performance on RAIEd2020 dataset, and the best architecture searched on the EdNet dataset also holds a very promising AUC value on EdNet: 0.8053. Thus, the best-found architecture holds good transferability and generalization.
Selected Features. In addition, their selected input features for encoder and decoder parts are also similar. Fig. 4(a) compares the selected features of SAINT, SAINT+, and the best architecture on EdNet. We can observe that the selected features in SAINT and SAINT+ are a subset of the selected features in the best-found architecture, where the elapsed time embedding and the lag time embedding are considered, and extra features tag and bund are also considered.
Effectiveness of Each Component. To validate the effectiveness of each devised component, four variant architectures (A-D) are first given and SAINT+ is taken as the baseline. The comparison results on Ednet dataset have been summarized in Table 4. The variant architecture A refers to the vanilla Transformer with all candidate features concatenated and then mapped by a linear transformation as inputs, variant B is the same as A but only takes the selected features in the best architecture as inputs, variant C is the same as A but applies the hierarchical fusion method to the selected features, and variant D is the Transformer architecture in (11) whose local operations are fixed with Conv  $1 \times 3$  and adopts C's input module. The comparison among SAINT+, A, B, and C verifies the effectiveness of the proposed hierarchical fusion method and selected features in the architecture. The comparison between C and D indicates incorporating convolution operations with Transformer is helpful to improve performance. Moreover, the results of D and the best architecture validate the effectiveness of the proposed approach, including the effectiveness of search space as well as the search approach.
Similarly, other four variant architectures  $(E - H)$  are established in Table 4 to show the influence of each designed component on the best-found architecture's performance. As can be seen, the searched model architecture, the selected features, and the devised fusion module play important roles while the searched model architecture holds the biggest influence on the performance of Ours.
(a) Selected features of the best architecture and(b) Convergence profiles of AUC on SAINT(+) two datasets.
Figure 4: The comparison of selected features on EdNet (a), and the effectiveness of search space reduction strategy (b).
Table 4: The effectiveness validation of the proposed different components on the EdNet dataset, and the influence of the searched model architecture and the selected features on the best-found architecture's performance is also presented. The Wilcoxon rank sum test with  $\alpha = 0.05$  is performed.
Method	RMSE↓ ACC↑ AUC↑ +/-/~
SAINT+	0.4285 0.71880.7916 3/0/0
A: All Features + Concat	0.4276 0.72030.7937 3/0/0
B: Selected Features + Concat	0.4262 0.72170.7958 3/0/0
C: Selected Features + Hierarchical	0.4250 0.72360.7987 3/0/0
D: C's Input + Convolution	0.4235 0.72530.8012 3/0/0
E: Ours (without Hierarchical Fusion, with Concat)	0.4223 0.72690.8041 3/0/0
F: Ours (without the Selected Features, with All Features)	0.4221 0.72600.8030 3/0/0
G: Ours (without Selected Features & Hierarchical, with SAINT+'s input)	0.4238 0.72490.8008 3/0/0
H: Ours (without the Searched Architecture, with SAINT+'s model), i.e., C	0.4250 0.72360.7987 3/0/0
Searched by ENAS-KT(f) (under a small Supernet with fewer training: embedding size 64, epoch 30), retrain under size 128, taking 9.1 hours totally	0.4224 0.72710.8036 3/0/0
Ours	0.4209 0.72950.8062 -
To investigate the effectiveness of ENAS-KT with less search cost, as shown in Table 4, a variant ENAS-KT(f) is created. Although its searched architecture's performance is slightly worse than Ours, the model performance is better than other approaches and its search cost only takes 9.1 hours, which is much fewer than Ours and competitive to NAS-Cell and others (can be found in Appendix B).
Effectiveness of Search Space Reduction Strategy. To validate the effectiveness of the devised search space reduction strategy, Fig. 4(b) depicts the convergence profiles of best validation AUC value obtained by the proposed approaches with and without search space reduction strategy on both two datasets, which are denoted by Approach and Approach(-Reduction), respectively. As can be seen, the suggested search space reduction strategy can indeed accelerate the convergence of the proposed evolutionary search approach and leads to better final convergence. More experimental discussions about the proposed ENAS-KT can be found in Appendix B.
In this paper, we first introduced the local context modelling held by convolutions to the Transformer, then automated the balancing of the local/global context modelling and the input feature selection by proposing an evolutionary NAS approach. The designed search space not only replaces the original global path in Transformer with the sum of a global path and a local path that could contain different convolutions but also considers the selection of input features. Besides, we utilized an EA to search the best architecture and suggested a search space reduction strategy to accelerate its convergence. Experiments on two largest datasets show the effectiveness of the proposed approach.
This work was supported in part by the National Key R&D Program of China (No.2018AAA0100100), in part by the National Natural Science Foundation of China (No.62302010, No.62107001, No.62006053, No.61876162, No.61906001, No.62136008, No.62276001, No.U21A20512), in part by the Natural Science Foundation of Guangdong Province (Grant No. 2022A1515010443), and in part by the China Postdoctoral Science Foundation (No.2023M740015).
[1] Ghodai Abdelrahman and Qing Wang. Knowledge tracing with sequential key-value memory networks. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 175–184, 2019.
[2] Ahmed Baruwa, Mojeed Abisiga, Ibrahim Gbadegesin, and Afeez Fakunle. Leveraging end-to-end speech recognition with neural architecture search. arXiv preprint arXiv:1912.05946, 2019.
[3] Hao Cen, Kenneth Koedinger, and Brian Junker. Learning factors analysis—a general method for cognitive model evaluation and improvement. In Proceedings of the 2006 International Conference on Intelligent Tutoring Systems, pages 164–175. Springer, 2006.
[4] Haw-Shiuan Chang, Hwai-Jung Hsu, and Kuan-Ta Chen. Modeling exercise relationships in e-learning: A unified approach. In Proceedings of the 2015 International Educational Data Mining Society, pages 532-535, 2015.
[5] Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Glit: Neural architecture search for global and local image transformer. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, pages 12-21, 2021.
[6] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, pages 12270-12280, 2021.
[7] Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Baek, Byungsoo Kim, Yeongmin Cha, Dongmin Shin, Chan Bae, and Jaewe Heo. Towards an appropriate query, key, and value computation for knowledge tracing. In Proceedings of the Seventh ACM Conference on Learning Scale, pages 341-344, 2020.
[8] Youngduck Choi, Youngnam Lee, Dongmin Shin, Junghyun Cho, Seoyon Park, Seewoo Lee, Jineon Baek, Chan Bae, Byungsoo Kim, and Jaewe Heo. Ednet: A large-scale hierarchical dataset in education. In Proceedings of the 2020 International Conference on Artificial Intelligence in Education, pages 69-73. Springer, 2020.
[9] Albert T Corbett and John R Anderson. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-adapted Interaction, 4(4):253-278, 1994.
[10] Antoine Cully and Yiannis Demiris. Online knowledge level tracking with data-driven student models and collaborative filtering. IEEE Transactions on Knowledge and Data Engineering, 32(10):2000-2013, 2019.
[11] Xinyi Ding and Eric C Larson. Automatic rn cell design for knowledge tracing using reinforcement learning. In Proceedings of the Seventh ACM Conference on Learning@ Scale, pages 285-288, 2020.
[12] Cornelia E. Dowling and Cord Hockemeyer. Automata for the assessment of knowledge. IEEE Transactions on Knowledge and Data Engineering, 13(3):451-461, 2001.
[13] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997-2017, 2019.
[47] Frank Wilcoxon, SK Katti, Roberta A Wilcox, et al. Critical values and probability levels for the wilcoxon rank sum test and the wilcoxon signed rank test. Selected Tables in Mathematical Statistics, 1:171-259, 1970.
[48] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In Proceedings of the 2020 International Conference on Machine Learning, pages 10524-10533. PMLR, 2020.
[49] Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, and Tie-Yan Liu. Nas-bert: Task-agnostic and adaptive-size bert compression with neural architecture search. In Proceedings of the 2021 ACM Conference on Knowledge Discovery and Data Mining, 2021.
[50] Jin Xu, Xu Tan, Kaitao Song, Renqian Luo, Yichong Leng, Tao Qin, Tie-Yan Liu, and Jian Li. Analyzing and mitigating interference in neural architecture search. In Proceedings of the 2022 International Conference on Machine Learning, pages 24646-24662. PMLR, 2022.
[51] Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. In Proceedings of the 2020 European Conference on Computer Vision, pages 702-717. Springer, 2020.
[52] Zhixiong Yue, Baijiong Lin, Yu Zhang, and Christy Liang. Effective, efficient and robust neural architecture search. In Proceedings of the 2022 International Joint Conference on Neural Networks, pages 1-8. IEEE, 2022.
[53] Jiani Zhang, Xingjian Shi, Irwin King, and Dit-Yan Yeung. Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th International Conference on World Wide Web, pages 765-774, 2017.
[54] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, pages 8697-8710, 2018.
Appendix A Proposed Approach
Encoding Strategy
In the proposed search space, each candidate Transformer architecture can be represented by the following vector-based encoding:
$$
\begin{array}{l} \mathcal {A} = \left[ \mathbf {b} _ {\mathbf {E n}}, \mathbf {b} _ {\mathbf {D e}}, (l o \in \{0, \dots , 4 \}, g o _ {1}, g o _ {2} \in \{0, 1, 2 \}) ^ {1 \times 2 N} \right] = \\ \left[ \begin{array}{c} \mathbf {b} _ {\mathbf {E n}} = \left(b _ {1}, b _ {2}, \dots , b _ {i}, \dots , b _ {N u m}\right) \in \{0, 1 \} ^ {1 \times N u m}, 1 \leq i \leq N u m \\ \mathbf {b} _ {\mathbf {D e}} = \left(b _ {1}, b _ {2}, \dots , b _ {i}, \dots , b _ {N u m}\right) \in \{0, 1 \} ^ {1 \times N u m}, 1 \leq i \leq N u m \\ 2 N \left\{ \begin{array}{l} (l o, g o _ {1}, g o _ {2}) \\ (l o, g o _ {1}, g o _ {2}) \\ \dots \\ (l o, g o _ {1}, g o _ {2}) \end{array} \right. \end{array} \right], \tag {14} \\ \end{array}
$$
where  $Num$  denotes the number of available input embedding and  $N$  denotes the number of encoder/decoder blocks.  $\mathbf{b_{En}}$  and  $\mathbf{b_{De}}$  are used to specify which input embedding are selected for the inputs to encoder and decoder parts, and the triplet  $(lo, go_1, go_2)$  is used to specify three operations adopted for each encoder block or each decoder block, i.e.,  $lo, go_1$ , and  $go_2$  determine which operation is used for  $LO(\cdot)$ ,  $GO_1(\cdot)$ , and  $GO_2(\cdot)$ .
Figure 5: An illustrative example of the utilized encoding strategy.
For a better understanding of the utilized encoding strategy, Fig. 5 gives an illustrative example, where  $Num$  and  $N$  are set to 5 and 2, respectively. The given architecture takes the first and second embedding as the input to the encoder part, and takes the third and fifth embedding as the input to the decoder part, its two encoder blocks are encoded by  $[1,0,1]$  and  $[2,0,1]$ , and two decoder blocks are encoded by  $[0,2,1]$  and  $[4,2,2]$ . As a result, the plotted architecture's encoding is  $[1,1,0,0,0,0,0,1,0,1,1,0,1,2,0,1,0,2,1,4,2,2]$ .
Super-Transformer Training
Inspired by one-shot NAS approaches [20], we use a trained super-Transformer mode to reduce the computational cost, which can provide the weights to directly evaluate the validation AUC values of subsequently sampled sub-Transformer architectures. To make the supernet be trained well, we consider adopting the sandwich training rule [51], which has been the most effective way and widely adopted by one-shot NAS approaches [52, 50]. The original sandwich training rule commonly employs three sub-models for one batch of training, including the largest sub-model, the smallest sub-model, and a randomly sampled sub-model, where three forward passes and one backward pass are executed for one batch update.