SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 Main Steps of ENAS-kT
Input: Gen: Maximum number of generations; Pop: Population size; S: Search space  
Output:  $\mathbf{P_b}$ : The best individual;  
1:  $\mathbf{P} \gets$  Randomly initialize Pop individuals; % Population initialization  
2: Evaluate each individual in  $\mathbf{P}$  based on the trained super-Transformer and obtain its AUC value;  
3: for  $g = 1$  to Gen do  
4:  $\mathbf{P}' \gets$  Select Pop parent individuals from Population  $\mathbf{P}$ ; % Mating pool selection  
5:  $\mathbf{Q} \gets$  Apply the single-point crossover and bit-wise mutation to  $\mathbf{P}'$  based on current search space  $S$ ;  
6: Evaluate each individual in  $\mathbf{P}$  based on the trained super-Transformer and obtain its AUC value;  
7:  $S \gets$  Search Space Reduction  $(S, \mathbf{P}, \mathbf{Q})$ ; % Reduce search space  
8:  $\mathbf{P} \gets$  Individual Selection  $(\mathbf{P} \cup \mathbf{Q})$ ;  
9: end for  
10: Select the best individual  $\mathbf{P_b}$  from  $\mathbf{P}$  as the output;  
11: return  $\mathbf{P_b}$ ;
To make the training rule adapt the proposed super-Transformer, we utilize three sub-models, including a global sub-model  $sub_{g}$ , a local sub-model  $sub_{l}$ , and a randomly sampled sub-model  $sub_{r}$ , for one batch of training, whose encoding can be represented by
$$
\left\{ \begin{array}{l} \operatorname {s u b} _ {g} = \left[ \mathbf {b} _ {\mathbf {E n}} ^ {\mathbf {r}}, \mathbf {b} _ {\mathbf {D e}} ^ {\mathbf {r}}, (0, 2, 1), \dots , (0, 2, 1), \dots , (0, 2, 1) \right] \\ \operatorname {s u b} _ {l} = \left[ \mathbf {b} _ {\mathbf {E n}} ^ {\mathbf {r}}, \mathbf {b} _ {\mathbf {D e}} ^ {\mathbf {r}}, (1, 0, 0), \dots , (1, 0, 0), \dots , (1, 0, 0) \right] \\ \operatorname {s u b} _ {r} = \left[ \mathbf {b} _ {\mathbf {E n}} ^ {\mathbf {r}}, \mathbf {b} _ {\mathbf {D e}} ^ {\mathbf {r}}, (l o, g o _ {1}, g o _ {2}) ^ {r}, \dots , (l o, g o _ {1}, g o _ {2}) ^ {r}, \dots , (l o, g o _ {1}, g o _ {2}) ^ {r} \right] \end{array} . \right. \tag {15}
$$
Here  $\mathbf{b}_{\mathbf{En}}^{\mathbf{r}}$ ,  $\mathbf{b}_{\mathbf{De}}^{\mathbf{r}}$ , and  $(lo, go_1, go_2)^r$  denote these vectors are randomly generated. The global sub-model  $sub_g$  denotes the architectures with pure global operations but no local operations, where  $\mathbf{b}_{\mathbf{en}}$  and  $\mathbf{b}_{\mathbf{de}}$  are randomly generated,  $lo$  is set to 0,  $go_1$  and  $go_2$  are set to 2 and 1. The local sub-model  $sub_l$  refers to the architectures consisting of only convolution operations with the smallest kernel size, where  $\mathbf{b}_{\mathbf{en}}$  and  $\mathbf{b}_{\mathbf{de}}$  are also randomly generated,  $lo$  is set to 1, both  $go_1$  and  $go_2$  are set to 0.
As a result, there are three forward passes and one backward pass executed in one batch for updating the super-Transformer. In each batch of training, we employ the Adam [18] optimizer to update the mode parameters via minimizing the binary cross-entropy loss [43] between the predicted response  $\hat{r}_i$  and the true response  $r_i$ :
$$
\mathcal {L} = - \sum_ {i} \left(r _ {i} \log \hat {r} _ {i} + \left(1 - r _ {i}\right) \log \left(1 - \hat {r} _ {i}\right)\right). \tag {16}
$$
Evolutionary Search
The main steps of the proposed evolutionary search approach are summarized in Algorithm 1, which consists of the following six steps.
First, a population containing  $Pop$  individuals will be randomly initialized. During the initialization, it is worth noting that  $\mathbf{b}_{\mathrm{En}}$  and  $\mathbf{b}_{\mathrm{De}}$  in each individual have to satisfy the constraints, i.e.,  $1 \leq sum(\mathbf{b}_{\mathrm{En}})$  and  $1 \leq sum(\mathbf{b}_{\mathrm{De}})$ . By doing so, it can ensure there is at least one input embedding for the encoder part and decoder part. Second, the standard binary tournament selection [37] is used to select individual to form the mating pool  $\mathbf{P}'$ , which randomly selects two individuals from  $\mathbf{P}$  each time and keeps the better one in  $\mathbf{P}'$  until the size of  $\mathbf{P}'$  is equal to  $Pop$ . Next, select two individuals from  $\mathbf{P}'$ , and then apply the single-point crossover and the bit-wise mutation [40] to the two individuals to generate two offspring individuals, where the generation process needs to be consistent with current search space  $S$ . As a result, an offspring population  $\mathbf{Q}$  with the size of  $Pop$  is obtained. Fourth, the validation AUC values of individuals in offspring population are efficiently evaluated by extracting their needed weights from the trained super-Transformer. Fifth, delete some worse operations from the search space by a devised search space reduction strategy, which makes statistics from the population  $P$  and the offspring population  $\mathbf{Q}$  and sorts all operations according to the computed fitness. Sixth, all individuals with better validation AUC values in the population  $P$  and the offspring population  $\mathbf{Q}$  will be kept as new population.
The second to the sixth step will be repeated until the maximal number of generations  $Gen$  is exceeded. Finally, the best architecture will be output and then fine-tuned based on the inherited weights from the super-Transformer by minimizing the training loss function in (16).
Search Space Reduction
The original search space  $S$  has the same length as  $\mathcal{A}$ , where the item on each bit is an operation set. For example, the item on first bit is  $\{0,1\}$ , the items on the bits corresponding to  $lo$  are  $\{0,1,2,3,4\}$ . To accelerate the convergence of the EA, we further devise a search space reduction strategy to iteratively remove some worse operations from the search space.
Algorithm 2 has summarized the basic procedure. First, population  $\mathbf{P}$  and offspring population  $\mathbf{Q}$  are combined, two empty lists  $SF$  and  $SF_{std}$  are used to record each operation fitness and fitness standard deviation of each operation set. Then, get the operation set  $Bit_{OpSet}$  on the  $i$ -th bit of current search space  $S$ . Third, compute the fitness value of each operation in  $Bit_{OpSet}$  by averaging the fitness of the individuals that contain the corresponding operation. Next,  $fitness_{OpSet}$ , the computed fitness of the set of operations, will be stored as  $i$ -th item in  $SF$ , and the fitness standard deviation of  $fitness_{OpSet}$  will be also computed and stored in  $SF_{std}$ . The second to the fourth step will be repeated for all bits in  $S$ , where  $|S|$  denotes the number of bits, i.e., the length. Based on  $SF$  and  $SF_{std}$ , the worst two operations will be deleted, and the worst operation in the operation set with the largest standard deviation will be also removed, where there is at least one operation kept for each bit. By doing so, the ever-reducing search space makes the search process easier.
Algorithm 2 Search Space Reduction
Input:  $S$  : Current search space;  $\mathbf{P}$  : Population;  $\mathbf{Q}$  : Offspring population;   
Output: RS: Reduced search space;   
1:  $\mathbf{P}\gets \mathbf{P}\cup \mathbf{Q},SF\gets \emptyset ,SF_{std}\gets \emptyset$  .   
2: for  $i = 1$  to  $|S|$  do   
3: BitOpSet  $= S[i]$  ,fitnessOpSet  $\leftarrow \emptyset$  ; % OpSet : The operation set on i-th bit   
4: for op in BitOpSet do   
5: fit  $\leftarrow$  The fitness value averaged on the individuals containing op in  $\mathbf{P}$  .   
6: fitnessOpSet  $\leftarrow$  fitnessOpSet U fit;   
7: end for   
8:  $SF[i] =$  fitnessOpSet;  $\%$  Store the fitness of the set of operations on i-th bit   
9:  $SF_{std}[i] = Std(fitness_{opSet})$  ;  $\%$  Compute and store the fitness standard deviation for the operation set on i-th bit   
10: end for   
11:  $RS\gets$  Delete the worst two operations from  $S$  based on  $SF$  .   
12:  $RS\gets$  Delete the worst operation from the operation set with the largest standard deviation in  $RS$  based on  $SF_{std}$  .   
13: return RS;
Table 5: Performance comparison under splitting setting of training and testing of  $60\%$  and  $30\%$ . The results of proposed ENAS-KT and compared KT approaches on EdNet and RAIEd 2020 datasets are reported in terms of RMSE, ACC, and AUC, averaged over 5-fold cross-validation. The Wilcoxon rank sum test with  $\alpha = 0.05$  is also executed.
Dataset	Metric	DKT	HawkesKT	CT-NCM	SAKT	AKT	SAINT	SAINT+	NAS-Cell	Ours
EdNet	RMSE ↓	0.4661	0.4479	0.4372	0.4411	0.4396	0.4326	0.4288	0.4350	0.4215
ACC ↑	0.6507	0.6879	0.7048	0.6993	0.7019	0.7128	0.7182	0.7138	0.7289
AUC ↑	0.6908	0.7475	0.7733	0.7645	0.7683	0.7816	0.7906	0.7782	0.8055
RAIEd2020	RMSE ↓	0.4651	0.4466	0.4357	0.4385	0.4368	0.4313	0.4275	0.4311	0.4207
ACC ↑	0.6601	0.6905	0.7065	0.7019	0.7076	0.7140	0.7189	0.7164	0.7306
AUC ↑	0.7057	0.7491	0.7751	0.7665	0.7640	0.7855	0.7927	0.7828	0.8081
+/-/≈ (six results totally)	6/0/0	6/0/0	6/0/0	6/0/0	6/0/0	6/0/0	6/0/0	6/0/0	-
Datasets Descriptions
In our experiments, two largest and most challenging real-world education datasets, EdNet [8] and RAIEd2020 [34], were used. Different common education datasets, such as ASSISTments09 [15],
Table 6: Performance comparison under splitting setting of training and testing of  $50\%$  and  $40\%$ . The results of proposed ENAS-KT and compared KT approaches on EdNet and RAIEd 2020 datasets are reported in terms of RMSE, ACC, and AUC, averaged over 5-fold cross-validation. The Wilcoxon rank sum test with  $\alpha = 0.05$  is also executed.
Dataset	Metric	DKT	HawkesKT	CT-NCM	SAKT	AKT	SAINT	SAINT+	NAS-Cell	Ours
EdNet	RMSE ↓	0.4664	0.4480	0.4371	0.4423	0.4397	0.4325	0.4286	0.4352	0.4219
ACC ↑	0.6503	0.6876	0.7055	0.6986	0.7026	0.7121	0.7182	0.7133	0.7291
AUC ↑	0.6902	0.7473	0.7732	0.7640	0.7678	0.7811	0.7903	0.7779	0.8052
RAIEd2020	RMSE ↓	0.4659	0.4469	0.4354	0.4375	0.4398	0.4315	0.4276	0.4312	0.4200
ACC ↑	0.6588	0.6902	0.7068	0.7023	0.7010	0.7146	0.7187	0.7165	0.7309
AUC ↑	0.7043	0.7484	0.7759	0.7674	0.7668	0.7851	0.7928	0.7835	0.8085
+/-/≈ (six results totally)	6/0/0	6/0/0	6/0/0	6/0/0	6/0/0	6/0/0	6/0/0	6/0/0	-
JunYi [4], and so on [22, 29], EdNet collected by Santa is currently the largest open available education benchmark dataset, which contains over 90M interactions and nearly 800K students, while RAIEd2020 first appeared in the Kaggle is another publicly available large-scale real-world dataset, which consists of nearly 100M interactions and 400K students.
More Results under Other Two Splitting Settings
In previous experiments, all students in each dataset were randomly split into  $70\%$ ,  $10\%$ , and  $20\%$  for training, validation, and testing, respectively. Under this splitting setting, the architectures found by the proposed ENAS-KT exhibits better performance than comparison KT approaches.
For more convincing results, we adopted other two settings to split the dataset into training and test datasets for evaluating the model performance. The first one is to split  $60\%$  of the dataset as the training dataset and  $30\%$  of the dataset as the testing dataset, while the second one takes the splitting ratio of  $50\%$  and  $40\%$ . The overall comparison results under the two settings are summarized in Table 5 and Table 6, where the average RMSE, ACC, and AUC values over 5-fold cross-validation are reported, the best results are in bold, and the Wilcoxon rank sum test with  $\alpha = 0.05$  is also executed.
As can be seen from these two tables, the performance of nearly all approaches on both two dataset decrease when the size of the training dataset is reduced while the size of the testing dataset is increased, but some approaches (such as CT-NCM, SAKT, AKT, SAINT+, NAS-Cell, and our approach) hold slightly better performance on RAIEd2020 under the splitting setting of  $50\%$  and  $40\%$  than that of the splitting setting of  $60\%$  and  $30\%$ . Under whichever splitting setting, the architectures found by the proposed approach still outperform the state-of-the-art KT approaches, which can be seen from the statistical results of the Wilcoxon rank sum test.
Selected Features
Due to page limit, previous experiments do not show the selected features of the best-found architecture on the RAIEd2020 dataset. For a more comprehensive observation and comparison, Fig. 6 presents the selected input features of four models, i.e., SAINT, SAINT+, best-found architecture on EdNet, and best-found architecture on RAIEd2020. It can be found that the selected features in SAINT and SAINT+ are a subset of the selected features in both two best-found architectures, where the elapsed time embedding and the lag time embedding are considered in two est-found architectures, and extra features (such as tag, bund, and tagset) are also considered. Besides, we can also find that the selected input features of these two found architectures are very close, where there is only one input feature difference (i.e.,  $\text{cate\_lag\_d}$ ) between their encoder parts, and only two input features (bundle/expla and tagset) are different between their decoder parts. This implies that the selected input features are more important for our Transformer architectures to solve the KT task than those not selected by the two architectures.
Figure 6: The visualization of selected input features of SAINT, SAINT+, and two best-found architectures on EdNet and RAIEd2020.
Figure 7: The runtime comparison between ENAS-KT, ENAS-KT(f) and KT approaches on EdNet, which reports the cost of searching, supernet and single architecture training for NAS-based approaches.
Limitation Discussion
The search cost of neural architecture search (NAS)-based approaches are always the focus of researchers, especially when they are compared to non-NAS-based approaches. To investigate this, Fig. 7 compares the runtimes of the proposed ENAS-KT, its variant approach ENAS-KT(f) and comparison KT approaches. Here the runtimes of ENAS-KT, ENAS-KT(f) and NAS-Cell contain the cost of supernet training, architecture searching and single architecture retraining, while runtimes of other KT approaches are only used for training their models.
It can be observed that the training cost of the architecture found by the proposed ENAS-KT is 3.6h, which is better than CT-NCM and AKT, worse than NAS-Cell, DKT, and HawkesKT, and competitive to other KT approaches. Besides, the search cost of the proposed ENAS-KT is 34.6h, leading to a total cost of 38.6h for ENAS-KT, which is higher than that of NAS-Cell (6.3h). This is because Transformer architectures in the proposed approach are more complex than that of NAS-Cell and thus need more time to train them. However, considering the significant performance leading of the proposed ENAS-KT over comparison approaches, the high overall cost of the proposed approach is acceptable for the KT task. Moreover, note that ENAS-KT(f) is a variant of the proposed ENAS-KT,
which searches for architectures under a smaller supernet with fewer training epochs. Specifically, the embedding size is halved to 64 for the supernet, the number of training epochs is also halved to 30, but the best-found architecture holds the same settings as Ours for a fair comparison. As shown in Table 4, the architecture found by ENAS-KT(f) holds a 0.8036 AUC value. Although its performance is worse than ours, as shown in Fig 7, its overall cost only takes 9.1 hours (3.8hours for supernet training, 1.7 hours for evolutionary search, and 3.6 hours for final training), which is fewer than the proposed ENAS-KT and competitive to NAS-Cell and others.
In summary, the current settings adopted in the proposed approach provide significantly better performance with a much higher cost, but our approach can also achieve competitive results at a much lower cost, whose performance is still better than other KT approaches and whose cost is competitive. In future work, we would like to explore surrogate models in the proposed approach to further reduce the search cost.
In addition to the search cost, the validity of the proposed approach also suffers from one external threat, i.e., too many available input features in new datasets ( $Num$  is large). The hierarchical feature fusion manner in Equation 10 utilizes pairwise concatenation, which requires a large number of parameters. When the datasets hold too many input features, the number of feature pairs will increase exponentially, e.g., 10 features need 45 concatenation operations while 20 features need 190 concatenation operations, which further causes a high increase in the number of network parameters and deteriorates the models' efficiency. Therefore, the validity and efficiency of the proposed approach will be highly affected by the number of available input features in new datasets. But because nearly all known education datasets hold fewer available features, the proposed approach is effective and can counter most datasets in KT. In future work, we would like to design novel fusion modules to aggregate features without considering the influence of datasets to be handled.