Evolutionary Neural Architecture Search for Transformer in Knowledge Tracing
Authors: Shangshang Yang
Xiaoshan Yu
Ye Tian
Xueming Yan
Haiping Ma
Xingyi Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Knowledge tracing
intelligent education
neural architecture search
Transformer

Abstract: Knowledge tracing (KT) aims to trace students' knowledge states by predicting whether students answer correctly on exercises. Despite the excellent performance of existing Transformer-based KT approaches, they are criticized for the manually selected input features for fusion and the defect of single global context modelling to directly capture students' forgetting behavior in KT, when the related records are distant from the current record in terms of time. To address the issues, this paper first considers adding convolution operations to the Transformer to enhance its local context modelling ability used for students' forgetting behavior, then proposes an evolutionary neural architecture search approach to automate the input feature selection and automatically determine where to apply which operation for achieving the balancing of the local/global context modelling. In the search space, the original global path containing the attention module in Transformer is replaced with the sum of a global path and a local path that could contain different convolutions, and the selection of input features is also considered. To search the best architecture, we employ an effective evolutionary algorithm to explore the search space and also suggest a search space reduction strategy to accelerate the convergence of the algorithm. Experimental results on the two largest and most challenging education datasets demonstrate the effectiveness of the architecture found by the proposed approach.

Evolutionary Neural Architecture Search for Transformer in Knowledge Tracing
Shangshang Yang!.3 1.3
Xiaoshan Yu 1.3.3
Ye Tian 1.3
Xueming Yan²
Haiping Ma 13,44
Xingyi Zhang
Anhui University Guangdong University of Foreign Studies
KKey Laboratory of Intelligent Computing and Signal Processing of Ministry of Education 'Department Information Materials and Intelligent Sensing Laboratory of Anhui Province (yangshang0308, yxsleo, f field910921 (0g910921@mmail..... com xueming1 (etpnng26e12.... com hpmaCahu. edu. cn
Abstract
Transformer has achieved excellent performance in the knowledge tracing (KT) task, but they are criticized for the manually selected input features for fusion and the defect single global context modelling to directly capture students' forgetting behavior in KT, when the related records are distant from the current record in terms time. To address the issues, this paper first considers adding convolution operations to the Transformer to enhance its local context modelling ability used for students' forgetting behavior, then proposes an evolutionary neural architecture search approach to automate the input feature selection and automatically determine where to apply which operation for achieving the balancing of the local/global context modelling. In the search space design, the original global path containing the attention module Transformer is replaced with the sum of global path and local path that could contain different convolutions, and the selection of input features is also considered. To search the best architecture, we employ an effective evolutionary algorithm explore the search space and also suggest search space reduction strategy to accelerate the convergence of the algorithm. Experimental results the two largest and most challenging education datasets demonstrate the effectiveness of the architecture found by the proposed approach.
Introduction
With the rapid development of online education systems, such as Coursera and ASSISTment [10] there have been massive amount available data about student-system interactions recorded in
these systems. Based the students' historical learning activities over time, knowledge tracing (KT),
as fundamental task in intelligent education, aims to trace students' knowledge states on knowledge concepts, which plays important role in both personalized learning and adaptive learning [28 [2 and thus has received growing attention from the scientific community.
Since the concept of KT was first proposed by Corbett and Anderson in Bayesian knowledge tracing (BKT) [ considerable efforts have been devoted to developing various KT approaches, which can mainly be split into two categories. The first type of KT approaches are traditional approaches based on probabilistic models or logistic models, such as dynamic BKT [17] LFA 33) PFA (30], and KTM [2 The second type of approaches (21) have shown significantly better performance and generalization since they utilize various deep neural networks (DNNs) to solve the KT task, such as
"Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

recurrent neural networks (RNNs) as adopted in [53]. long short-term memory networks (LSTMs) as adopted in M graph neural networks as adopted in 26], and Transformer as adopted in 7 33 36]
Among these DNN-based approaches, Transformer-based KT approaches have exhibited significantly better performance due the better ability to tackle sequence tasks. However, demonstrated in [16, 36), when there is no extra specific information introduced, the attention mechanism in the vanilla Transformer [4]] cannot capture students' forgetting behavior [24], since its providing single global context modelling usually assigns high importance to the highly related records to predict the current record even with position information. However, those related records distant in time commonly pose little influence on the current prediction. Moreover, the input features in these approaches are manually selected based on domain knowledge and simply fused as the inputs the Transformer encoder and decoder, but the manually selected features and the simple feature fusion method may miss some valuable features and complex feature interaction information.
To address the issues for further improving the performance, we first equip the Transformer with convolutions to boost its local context modelling ability used for students' forgetting behavior, then propose an evolutionary neural architecture search (NAS) (54) approach for KT (termed ENAS-KT), which aims to automatically select input features and determine where and which operation is applied to strike the balancing of local/global context modelling. Our main contributions are as:
To the best of our knowledge, we are the first to apply NAS to search the Transformer architecture for the KT task. Here, the proposed ENAS-KT aims to find the best architecture that holds the optimal selected input features and the ideal balancing of the local/global context modelling, which can model students' forgetting behavior well and thus hold significantly good prediction performance KT.
In the proposed ENAS-KT, we design novel search space for the Transformer and devise an effective evolutionary algorithm (EA) as the search approach to explore the search space. The search space not only considers the input feature selection of the Transformer but also equips it with local context modelling by replacing the original global path containing the attention module with the sum of a global path and a local path that could contain different convolutions. Besides, an effective hierarchical fusion method is suggested to fuse selected features. The proposed ENAS-K? not only employs trained super-net for fast solution evaluation but also suggests a search space reduction strategy to accelerate the convergence.
Experimental results validated on two largest and most challenging datasets show that the architecture found by the proposed approach holds significantly better performance than state-of-the
the-hht-tttt art (SOTA) KT approaches.
2 Related Work
2.1 Transformer-based Knowledge Tracing
Existing Transformer-based KT approaches focus on taking different features as the inputs to the Transformer encoder and decoder based on human expertise without changing the model architecture. In 77, Choi et al. proposed to directly employ the vanilla Transformer for KT, namely SAINT, where the exercise embedding and knowledge concept embedding are taken as the input of the Transformer encoder part and the response embedding is taken as the input to the Transformer decoder part. As a result, the SAINT achieves significantly better performance than other KT approaches. To model the forgetting behavior in students' learning, Pu el al. (33 further fed the timestamp embedding of each response record to the Transformer model. Besides, to model the structure of interactions, the excreise to be answered at the next time is also fed to the model. As the successor of the SAINT, due to the problem in SAINT that students' forgetting behavior cannot be modelled well by the global context provided by the attention mechanism, the SAINT+ (36] aims to further improve the performance of KT by mitigating the problem. To this end, two temporal feature embedding, the elapsed time and the lag time, are combined with response embedding as input to the decoder part.
Different from above two approaches that only focus on how to manually select input features to alleviate the forgetting behavior problem, this paper focuses on not only realizing the input feature automatic selection of the Transformer but also the Transformer architecture automatic design by proposing an Ea-based NAS approach, where convolution operations are considered in the Transformer to enhance its local context ability used for students' forgetting behavior.

---

2.2 Neural Architecture Search for Transformer
NAS has been applied many domains B3 such as convolution neural networks in computer
vision [54], RNNs in natural language processing [46] or automatic speech recognition [2].
So al. [8 are the first to apply NAS to design the Transformer architecture, where EA is employed to explore the devised NASNet-like search space consisting of two stackable cells for translation tasks. To save computation resources, a progressive dynamic hurdles method is employed to early stop the training of poor performance architectures. Inspired by this work, there have been series of NAS work proposed search the Transformer architecture in different research fields. For NLP, the representative approaches include NAS- TARTBERT (49) and HAT (45), For ASR, the representative approaches include the LightSpeech [2]] and DARTS-CONFORMER 35 For CV, the representative approaches include the AutoFormer (6, GLiT [5), and VITAS (39. The Transformer architectures found by these approaches have demonstrated significantly better performance than the SOTA competitors in their own domains.
Despite the emergence of these NAS approaches for the Transformer, their designed search spaces for different domains cannot be directly used for solving the feature selection problem and the problem of balancing local context and global context in this paper. To the best of knowledge, the first to apply the NAS to search the Transformer architecture for the KT task. Different from NAS-Cell [I]] that directly applies existing NAS methods used for general LSTMs to KT, our work designs effective search space for the Transformer to strengthen its modeling capability in KT.
3 Preliminaries
3.1 Knowledge Tracing Problem
Given a student's response records on a set of exercises over time, i.e., a sequence of interactions denoted by / = {Ii. I2.... ], the KT task [2] aims to predict the probability that student answers correctly on the exercise in the next interaction In+1- I; = (e; 0g, dt,, r;) is the i-th interaction of student, e; denotes the the i-th exercise assigned to the student, and o; contains other related information about the interaction I, such as the timestamp, the type of exercise ez. knowledge concepts in e;, and so on. dl, represents the elapsed time the student took to answer, r; E {0, 1} is the student's 's responsc/answer exercise e;, where r,=1 means the answer is correct otherwise the answer is incorrect.
Based on the interactions before the ++1-th interaction 4+1, the KT focuses on predicting the probability of the student answering correctly on exercise et+1, formally:
12, ,r, ((+1),
(1)
where the response T and the elapsed time d cannot be accessed for the prediction of 4++1-
3.2 Transformer-hased KT
Transformer 41 consists of an encoder part and decoder part, where each part composed of stack of N identical blocks, and each block has a multi-head self-attention (MHSA) module and a position-wise feed-forward network (FFN) module. Then, each encoder block can be expressed
blocken (N(FFN(1) + A), h LN X),
(2)
where X is the input of the encoder block, and LN(-) is layer normalization (88 Each decoder block can be denoted by
block De
(3)
h LN(FFN((A) + h), h = IN(MHHSA(b, OKo, Oe.) +
where Masked (- the MHSA(-) with the mask operation prevent current position to attend the later positions, and OEn is the last encoder block's output.
For the Transformer-basee KT approaches, the following sequential inputs Xiput with length of L:
Xinput = {F. 0F DT, R) (F = {ez, ....... ,ec), 0 = (01,02; ,,,),, DT = (dlaters, dl,, ... ,d }, R = (Fstors, "I,"" TLL-1]]
(4)
3

---

Search Space
N*
Candicatt Operations
Cardidite Operation
dlbb
for Local Opendio
for Globd Ogavalion
N*
Blebe
Opendaas
[1-4] Stendent Concrlations
[0] Zero Opentioo
[0] Operation
[J] Part Fark------aaaa Saroadd
Natoork
[2] Milb-tees Selt Atlr-ot
Inputt
Noen
Narrr
Openton
X
Outpu
Caroe
Nation 39
Grecal & Lineer& Tush
Sevectvv Eleerrhee
Ire Modilo
Saledira Haenrchcc
Inzal Kotal
Sabucael [1.34]
All Avallable Hearames
Figure I: The proposed search space. There are five candidate operations for the local operation and three operations for the global operation.
will be first mapped into a set of embedding Xembed = {embed, E R 1 is Num). where D is the embedding dimension, Num is the number of available features in dataset, distart and T Tstart are the start tokens. O could contain multiple features and thus produce multiple embedding. Then, set of embedding XEn and D will be selected from Xembed and fed to the encoder part and decoder part, respectively. Finally, the decoder will output the predicted response f (f1, f2,.
} as
Decoder Decoder(FFusion) pe), OEn; OEn); OE Encoder (Fusion(X
(5)
where Fussio(.) is fusion method to get L D tensor. Encoder()) and Decoder(-) denote the forward passes the encoder part and decoder part, whose each block's forward pass as shown in (2) (3) Due to the nature of KT task, mask operation is necessary for each MHSA module to
prevent current position from accessing the information in later positions. Thus the MHSA module mentioned in subsequent sections refers to Masked (MASAe))MMAA(()..
Existing Transformer-base KT approaches empirically fuse the exereise-related embedding and response-related embedding as the inputs to encoder and decoder parts, respectively. However, the adopted feature fusion method is simple and the selected feature embedding sets may be not the best choice. Therefore, we further employ NAS to realize the input feature selections of both encoder and decoder parts and suggest an effective hierarchical feature fusion.
The Proposed ENAS-KT
To achieve the automatic input feature selection and automate the balancing of local/global modelling, we design a novel search space for the Transformer, and propose an evolutionary algorithm to search the optimal architecture. The following two subsections will present the details.
4.1 Search Space Design
We first replace Post-LN [48] with the pre-LN 19 to stabilize the training of the Transformer. Then, design novel search space for the Transformer, which has been presented in Fig. and there exist three important designs: (1) A selective hierarchical input module; (2) The sum global path and local path to replace the original global path; (3) The global operations containing MHSA module, FFN module, and zero operation.
4.1.1 Selective Hierarchical Input Module
Given all available input features Xinput DT, R} in student's response records, we can obtain each candidate embedding in Xemtee = fembed, RI < i Num) by categorical embedding or continuous embedding. For example, the categorical embedding of E R2IL can be computed
W embedcate one-hot(E) Weate cate. W Weate gaumgxt
(6)

---

where numg is the number of all exercises, one-hot(.) outputs a L x numg one-hot- one-hot-vector vector matrix, and W cate denotes the trainable parameters of the embedding layer. And the continuous embedding of DT E RIxL can be computed by
embedeons DT W.ont W cone R!XD
(7)
Based on all candidate embedding Xembed, selective hierarchical input module is proposed to first select two sets of suitable embedding from Xembed and then fuse the selected embedding as the inputs encoder and decoder, respectively. To this end, employ two binary vectors determine which candidate embedding is selected:
by (bx E {(. 1xNu" 1 sumn(ee)) bn (6. {0, 1 1 sumn(bb.),
Num
(8)
where 1 < S Num and 1 S sum ensures at least one embedding is selected. b; =1 means embedding embed,; in Xembed is selected and b; = otherwise. By doing so, two sets of selected embedding X and X used for the encoder and decoder parts can be obtained by
NE (embbed, E Xemsredl; 1, by ben), Xpe
{embbed, E Xemboo (b. 1,b, bDe), (9)
Instead of directly applying the simple sum or concatenation operation to X we propose a hierarchical fusion method to fully aggregate the selected embedding XEn En
Hier Hier(Xea (Xen): impute. = ctmoat(temp) Wout, temp = U
= Ex., (cconca(((e 2,1) * Wey), (10)
where concal(-) is concatenation operation, a()) is the tanh activation function, W iy E R2Dx
and Wont Rltemp)+DDD are trainable matrices, and [templ denotes the number of temporary features in temp. The first item in equation (TO) is to combine the selected embedding in pairs to fully fuse pair of embedding without the influence of other embedding, which is inspired by the feature fusion in multi-modal tasks (25). After obtaining temp, the second item in equation (1O) is to directly apply the concatenation operation to temp followed by a linear transformation to get the final input inputes En to the encoder part. The input input De to the decoder part can be obtained in the same way.
As shown in Fig. the size of W out is fixed as Numa CNinaNNuuee (Num-1).D D for easily training super- Transformer, but the temporary features related to non-selected embedding are set to zeros. For example, feature is not selected, thus the 1st, 4th, and 5th temporary features set to zeros.
4.1.2 The Global Path and The Local Path
Since students' forgetting behaviors in the KT cannot be directly modelled well by the single MHSA module, we aim to incorporate the Transformer with the local context modelling. Thus, we split the original global path containing the MHSA module in each encoder block into the sum of two paths: path is the original global path, and another path is the newly added local path containing convolution operations. Each decoder block is also set to be composed of these two paths. After applying the pre-LN to the input X of each block as Φ LX(X), each encoder and decoder block's forward pass in and (3) can be rewritten as
gp LN(21 + MHSA(E), 21.1))
gp TN + (H,
9p gp MHSA(gp, OEn, OEn) (11)
block
lp LN(=: (x))))
and block
Ip LN(æ1 + L.O(z)))
h FFN(4),
gp
= h + FFN(h), h Ip gp
where h refers to the output, gp is the latent representation obtained by the global path, lp is the latent representation obtained by the local path, and LO(-) denotes local operation used for the local path.
As shown Fig. there are five candidate operations for LO(-), including four one-dimension convolution operations with kernel sizes in (3, 5, 7, 11} and zero operation that returns a zero tensor. When the zero operation is used for LO(-), ) the blockgn and blockpe will degenerate to the encoder and decoder block in the vanilla Transformer.
4.1.3 The Global Operation
As demonstrated in the Sandwich Transformer 32 reordering the MHSA model and FFN module may lead to better performance. For this aim, global operation is used to replace the first MHSA
5

---

Search via Evolutionary Algorithm
① Train Super-Net
sample
Pepulation intialization
Maring pook selection
Offpprnn senenttio
Indirsidue ovthation
Space Reduction
Individual selection
excecded
Yes
Reture Jmaess
chimurer mutallen
decoding
Output best individial
Kepp
officing
BEEE
Extract weights from
Suparssokl ①
Figure 2: The overview of the proposed ENAS-KT.
module and the FFN module of each encoder and decoder block in 11 where MHSA(.) and F-N are replaced by GO1(-) and GO2(-). Here, GO1(-) and GO2(-) represent the adopted global operation. As shown in the light orange area on the right of Fig. 1 there are three candidate operations for (0), including the MHSA module, FFN module, and zero operation.
With the above modifications, the proposed Transformer architecture predicts the response {f1, f2.
=
f as
Decoder (Hier(XDe)) (Hierer((Hier(Xaaa Ogn, OEn), OEn Encoder(HHier(Xaa
(12)
where En and Xe are obtained based on ben. bDe and Xembed as shown and
4.1.4 Encoding Scheme
Under the proposed search space, there theoretically exist N Wmm (=22(2u22 (3+3++)) candidate Transformer architectures in total, where each candidate Transformer architecture can encoded by A [be,, bDe: (lo E {0..... .4,, g01, go2 E {0, 1, 1x2N]]
Here byn and bDe denote which embedding is selected for the inputs encoder and decoder parts, and 2.N triplets (lo, 100.g0019000 g01, determine which operations are adopted for each block, where lo, go1. and go2 determine which operation used for (-).). GO1 and GO2(-).
Next, we search the optimal Transformer architecture by solving the following optimization problem:
max ((A) fva!_auc (A, Deat),
(13)
where "val_auef (A, Deal) denotes the AUC (area under the curve) value of AA on dataset Deat-
4.2 Search Approach Design
Overview, An overview of the proposed approach is shown in Fig. where a super- Transformer is first trained and then EA with Pop individuals and Gen generations is used for searching.
4.2.1 Super-Transfornnn Training
Inspired by the sandwich rule (51], we propose to make the training rule adapt to our super- Transformer by utilizing three aub-models in one batch of training of the super- Transformer, including a global sub-model suby. local sub-model suby, and a randomly sampled sub-model sub,. sub, con- sists of pure global operations but no local operations, whose bgn and bp randomly generated; similarly, sub, consists of only convolution operations with the smallest kernel size.
4.2.2 Search Space Reduction-based Evolutionary Search
After training the supernet, a tailored EA is employed for searching by solving the problem in 13 The proposed EA takes the binary tournament selection [37] and pigle--ooint crossover together with the bit-wise mutation [40] to mate and generate individuals, where search space reduction strategy suggested to reduce the search space for accelerating the convergence at each generation.
Search Space Reduction Strategy. This strategy aims to iteratively remove some worse operations from the search space based on all individuals" fitness statistics speed up the convergence. The item each bit of A is an operation set, such the item first bit is {0, 1}. and the items

---

Table 1: Statistics of the two largest education datasets for knowledge tracing, EdNet and RAIEd2020.
# of interactions (ff of tags)
# of students
# of exercises (# of bundles)
# of skills
Datasets/Statisties
(#f of tag-sets,
(# of explanation)
95,293.926/99.277 2,1,999977,30000 1300
84,309/333.55
13, 13,139/33,523
7/7 (-)(22
EdNet/RATEd2020
(302)/(189)
(1,792)((15500)
(9,534V(-)
Table 2: The comparison between ENAS-KT and compared approaches in term of model performance and parameters. The results under splitting setting of 80% and 20% on EdNet and RATEd2020 are reported in terms of RMSE, ACC, and AUC, averaged over 5-fold cross-validation The Wilcoxon rank sum test with significance level a=0.05 is used for analyzing the significance, where '+; and "x' denote Ours significantly better than, worse than, and similar to the compared approach, respectively. "M" is short for "million" in "Param. (M)". counting the number of model parameters.
Dataset
Metric
DKT
HawkesKT CT-NCM SAKT AKT SAINT SAINT+ NAS-Cell Ours
EdNet
0.13495
0.019578
1.9974
2.0864 1,2330 2.7492
3.1862
1.8692
3.8232
(a)am.(M)
RAIEd2020
0.13531
0.019932
2.0431
2.1317 1.2335 2.7945
3.2315
1.9455
4.1262
RMSE &
0.4653
0,4475
0.4364
0.4405
0.4399 0.4322
0.4285
0.4345
0.4209
EdNet
ACC f
0.6537
0.6888
0.7063
0.6998
0,7016 0.7132
0.7188
0.7143
0.7295
AUC f
0.6952
0.7487
0.7743
0.7650 0.7686 0.7825
0.7916
0.7796
0.8062
RMSE +
0.4632
0.4453
0.4355
0.4381
0.4368 0.4310
0.4272
0.4309
0.4196
RAIEd2020
ACC f
0.6622
0.6928
0.7079
0.7035
0.7076 0.7143
0.7192
0.7167
0.7313
AUC f
0.7108
0.7525
0.7771
0.7693 0.7752 0.7862
0.7934
0.7839
0.8089
+/-/ (six results totally)|
6/0/0
6/0/0
6/0/0
6/0/0 6/0/0 6/0/0
6/0/0
6/0/0
the bits related to lo are 2, 3, 4). To start with, combine individuals in the population and offspring population; secondly, record each operation fitness in list SF, where each operation fitness is obtained by averaging the fitness of the individuals containing the corresponding operation; thirdly, compute fitness standard deviation of each each operation set as list S next, based on SF and SFaa. the worst two operations will be deleted, and the worst operation in the operation set with the largest standard deviation will be also removed, where there is at least one operation kept for each bit. By doing so, the ever-reducing search space makes the search process easier. More details (about the super-net net training, the EA, and the reduction strategy) can be found in Appendix A
Experiments
5.1 Experimental Settings
Datasets. For convincing validation, two largest and most challenging real-world education datasets EdNet [8] and RAIEd2020 (34] were used in experiments, whose statistics are summarized in Table
In addition to the above five feature embedding in Table exercise exer, skill sk, tag tag, tag-set tagset, and bundle/explanattinn bund/expla, there other seven feature embedding in candidate input embedding X Xembed (Num = 12), including answer embedding ans, continuous embedding for elapsed time cont_ell ela, categorical embedding for elapsed time in terms of seconds cate _atee_ee_______eeeeeeee continuous embedding for lag time cont cont_lag, categorical embedding for lag time in terms of seconds, minutes, and days (termed cate cate_lag_s, cate_lag_me and cale_lag_d)
Compared Approaches. The following eight SOTA KT approaches were used for comparison:
LSTM-based and Hawkes-based methods: DKT BI), HawkesKT 44. CT-NCM (24) and NAS- Cell M. HawkesKT and CT-NCM consider extra students' forgetting behaviors in modelling, and NAS- Cell is NAS-based method for searching LSTMs' cells.
Attention mechanism-basee neural network methods: SAKT 27 and AKT [6] Here AKT considers extra students' forgetting behavior in its attention weights.
Transformer-base methods: SAINT [7] and SAINT+ (36]. But SAINT+ considers extra features of elapsed time and lag time in the decoder input mitigate the forgetting behavior.
7

---

Parameter Settings. According to 36], all students were randomly split into 70%/10%/20% for trainnn/valiatioonnesing. The maximal length of imput sequences was set to 100 (L=100), we truncated student learning interactions longer than 100 to several sub-sequences, and 5-fold cross- validation was used. The number of blocks N=4, emhedding size D=128, the hidden dimension of FFN was set to 128. To train the supernet, epoch number, learning rate, dropout rate, and batch size was set to 60, le-3, 0.1, and 128, and the number of warm-up steps in Noam scheme 77] was set to 8,000. To train the best architecture, epoch number and the number of warm-up steps was set to 30 and 4,000, and others kept same as above. Pop and Gen was set to 20 and 30. For fair comparison, parameter settings of compared approaches are same those in their original papers. We adopted AUC, accuracy (ACC), and root mean squared error (RMSE) as evaluation metrics. All experiments were implemented with Py PyTorch and run under NVIDIA 3080 GPU. The source code of the proposed approach is publicly available at wthhtps:/// / //g thub com Dav il www.seiiiaann///na //ENAS-KT KT
5.2 Overall Results
To verify the effectiveness of the proposed approach, Table 2 summarizes the performance of the proposed ENAS- KT and all comparison KT approaches on predicting students' responses across EdNet and RAIEd2020 datasets, where the average RMSE, ACC, and AUC values over 5-fold cross-validation are reported and the best results are in bold. For more convincing, the Wilcoxon rank sum test [47 with a significance level a=0.05 is used for analyzing the significance. In addition, the number of model parameters for each approach is reported for profound observations.
From Table 2 we can obtain the following five observations: (1) The performance of Transformer- based KT approaches is significantly better than LSTM-based KT approaches and attention mechanism- KT approaches. (2) From the comparisons between DKT and CT-NCM, SAKT and AKT. SAINT and SAINT+, it can be found that considering the forgetting behavior in the KT can indeed improve the prediction performance to some extent under whichever type of neural archi- tecture. (3) The RMSE, ACC, and AUC values achieved by the architecture obtained by the proposed approach on both two datasets are significantly better than that achieved by comparison approaches, which indicates the architecture found by the proposed approach is more effective in tracing students' knowledge states. (4) Compared to NÁS- -Cell the architectures found by the proposed approach are more effective, which demonstrates the effectiveness of the proposed search space and search algorithm. (5) In terms of model parameters, HawkesKT is the best, DKT is the second-best, while the architectures found by the proposed ENAS-KT are worse than other KT models. Actually, high model parameters of Ours are attributed to its input part, which is used to get the input embeddings and aggregate these embeddings.
To analyze the influence of model parameters on the performance of the best-found architecture, four parameter constraints (including 2M, 2.5M, 3M, and 3.5M) are added the proposed ENAS-KT, respectively, where the parameters of all generated architectures cannot exceed the used parameter constraint. As result, four architectures are found on EdNet, denoted as ENAS-KT(2M), ENAS- KT(2.5M), KENAS-KT(3M)) these found architectures some KT models. As can be seen, the proposed ENAS-KT can still find good potential architecture under the parameter constraint, and the found architecture's 's performance increases with model parameters increasing, but the increasing model parameters is caused by the change of the found model architecture. not the embedding or hidden size. Therefore. we can conclude that the effectivvness of the proposed approach is mainly not attributed its high model parameters, to some extent, which can be mainly attributed to the suggested search space. Note that the comparison between SAINT/SAINT+ and SAINT (Sore)/AANNT (more) can indirectly validate the above conclusion that only increasing the model parameters to improve model performance does not work and is unworthy, which further demonstrates the effectiveness of the proposed search space. For more convincing, more results under other splitting settings can be found in Appendix B
and ENAS-KT(3.5M)) and Table gives a brief comparison between
5.3 Search Result Analysis
We can find that these two best-found architectures have very similar structures, their encoder parts prefer the convolution operations and FFN modules, especially for their first encoder blocks, while their decoder parts prefer the global operation: the MSHA module, especially for the decoder blocks close to the output, where the convolution operations with large kernel sizes are also preferred. To investigate the observation,
Architecture Visualization.
The best-found architectures are plotted in Fig. 3
8

---

Table 3: The comparison on EdNet dataset in terms of AUC and the number of model parameters:
some KT approaches and best architectures found by ENAS-KT with different parameter constraints. SAINT(more) and (AINT+(more) refer to two models with larger hidden and embedding sizes (192).
Model
Param.(M) AUC
Model Param.(M)
AUC
Model
Param.(M)
AUC
ENAS-KT(MM
1.938
0.8021 CT-NCM
1.997
0.7743
SAINT
2.749
0.7825
ENAS-KT((.5M)
2.389
0.8047 SAKT
2.086
0.7650
SAINT+
3.186
0.7916
ENAS-KT(M
2.918
0.8056
AKT
1.233
0.7686 SAINT(more)
4.910
0.7828
ENAS-KT((.5M)
3,489
0.8059NAS-Cell
1.869
0.7795SAINT+<morr)
5.627
0.7921
FIN
MISA
FFN
MHSA
Coro
Zero
FEN
Con.
Cora 11
Zero
MESSA
Cann 7
FEN
MFNSA
Cony
MISA
FFN
MHSA
MHSA
Cora
MESA
MHSA
Come 11
Cocy 7
Zero
MFSA
Coray 5
FFN
FFN
Zero
Carw 5
FIN
CoRw 7
PNN
Flero
Hirt)
Hirr
(a) Scarched on EdNet.
(b) Searched on RAIEd2020.
Figure 3:
The best architectures found on EdNet (a) and RAIEd2020 (b), here simplify the connections in each encoder/decoder block for better observation.
we further transfer the best architecture found on one dataset to another datasets. The best architecture searched on EdNet dataset achieves an AUC value of 0.8078 on RAIEd2020 dataset, which is very close to the best performance on RAIEd2020 dataset, and the best architecture searched on the EdNet dataset also holds very promising AUC value on EdNet: 0.8053. Thus, the best-found architecture holds good transferability and generalization.
Selected Features. In addition, their selected input features for encoder and decoder parts also similar. Fig. | compares the selected features of SAINT, SAINT+, and the best architecture on EdNet. We can observe that the selected features SAINT and SAINT+ are subset of the selected features in the best-found architecture, where the elapsed time embedding and the lag time embedding considered, and extra features tag and bund are also considered.
5.4 Ablation Study
Effectiveness of Each Component. To validate the effectiveness of each devised component, four variant architectures (A-D) are first given on Ednet dataset have been summarized in Table 4 The variant architecture refers to the vanilla Transformer with all candidate features concatenated and then mapped by linear transformation as inputs, variant is the same as A but only takes the selected features the best architecture as inputs, variant C is the same as A but applies the hierarchical fusion method to the selected features, and variant D is the Transformer architecture in (T]) whose local operations are fixed with Conv 1 x and adopts C's input module. The comparison among SAINT+, A, B, and C verifies the effectiveness of the proposed hicrarchical fusion method and selected features in the architecture. The comparison between C and D indicates incorporating convolution operations with Transformer is helpful to improve performance. Moreover, the results of D and the best architecture validate the effectiveness of the proposed approach, including the effectiveness of search space well as the search approach.
girstgiven and SAINT+ is taken as the baseline. The comparison results
Similarly, other four variant architectures (E-H) are established in Table 4 to show the influence of each designed component the best-found architecture's performance. As can be seen, the searched model architecture, the selected features, and the devised fusion module play important roles while the searched model architecture holds the biggest influence on the performance of Ours.

---

cont lag
cale éla
taquet
lag
lag.
6.790
exer
cont
ela
bund
lag_d
0.790
0600
SAINT
0.190
0.79
0150005
0.33
SAINT+
Appraac RAIF
030005
Bost Auchileuuure (TeNet)
0.12
Non-stlected
0.7
Narther of Gezentioos
(a) Selected features of the best architecture and(h) Convergence profiles of AUC on two datasets.
S*INT(+).
Figure 4: The comparison of selected features on EdNet (a), and the effectiveness of search space reduction strategy (b).
Table 4: The effectiveness validation of the proposed different components on the EdNet dataset, and the influence of the searched model architecture and the selected features on the best-found architecture s performance is also presented. The Wilcoxon rank sum test with a=0.05 is performed.
Method
RMSEJ ACC1 AUCT +/-/>/
SAINT+
0.4285 0.71880.791 3/0/0
A: All Features Concat
0.4276 0.72030.7937 7937 3/0/0
B: Selected Features Concat
0.4262 0.72170.
7.598 7958 3/0/0
C: Selected Features Hierarchical
0.4250 0.72360. 0.7877 3/0/0
C's Input Convolution
0.4235
0722530.801 3/0/0
Ours (without Hierarchical Fusion, with Concat)
0,4223
0.72690 8041 3/0/0
F: Ours (without the Selected Features, with All Features)
0.4221
0.72600 8030 8030
3/0/0
G: Ours (without Selected Features & Hierarchical. with SAINT+'s input)
0.4238
0.7249 O, 8008 3/0/0
H: Ours (without the Searched Architecture, with SAINT+'s model), i.e., C
0.4250
0.72360.7981
3/0/0
Searched by ENAS-KTH (under small Supernet with fewer training:
embedding size 64, epoch 30), retrain under size 128, taking 9.1 hours totally
0,4224
0.7271 0.8036 3/0/0
Ours
0.4209 0.7295 0.8062
To investigate effectiveness ENAS-KT with less search cost, as shown Table a variant ENAS-KT(() is created. Although its searched architecture's performance is slightly worse than Ours, the model performance better than other approaches and search cost only takes 9.1 hours, which much fewer than Ours and competitive to NAS-Cell and others (can be found in Appendix BB.
Effectiveness of Search Space Reduction Strategy. To validate the effectivenees of the devised search space reduction strategy, Fig. b)) depicts the convergence profiles of best validation AUC value obtained by the proposed approaches with and without search space reduction strategy on both two datasets, which are denoted by Approach and Approach( Aeproach(-Reduction respectively. As can be seen, the suggested search space reduction strategy can indeed accelerate the convergence of the proposed evolutionary search approach and leads to better final convergence. More experimental discussions about the proposed ENAS-KT can be found in Appendix
Conclusion
In this paper, first introduced the local context modelling held by convolutions to the Transformer, then automated the balancing of the local/global context modelling and the input feature selection by proposing evolutionary NAS approach. The designed search space not only replaces the original global path in Transformer with the sum of global path and a local path that could contain different convolutions but also considers the selection of input features. Besides, the best architecture and suggested search space reduction strategy to accelerate its convergence. Experiments on two largest datasets show the effectiveness of the proposed approach.
utilized EA to search
10

---

Acknowledgements
This work was supported part by the National Key R&D Program of Chima in part by the National Natural Science Foundation of China (No.62302010, 10.62107001, No. 63.62206053, N0.61876162, 60.61906001, No.621 16008,, No.62276001, No.U21A20512), in part by the Natural Science Foundation of Guangdong Province (Grant No. 2022A151501044)) and in part by the China Postdoctoral Science Foundation (No.2023M740015).
References
[1] Ghodai AbdeIrahman and Qing Wang. Knowledge tracing with sequential key-value memory networks. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 175-184, 2019.
[2] Ahmed Baruwa, Mojeed Abisiga, Ibrahim Gbadegesin, and Afeez Fakunle. Leveraging end- to-end speech recognition with neural architecture search. arXiv preprint arXiv: UrXiv:1912.05946 2019.
[3] Hao Cen, Kenneth Koedinger, and Brian Junker. Learning factors analysis-a general method for cognitive model evaluation and improvement. Proceedings of the 2006 International Conference on Intelligent Tutoring Systems, pages 164-175. Springer, 2006.
[4] Haw-Shiuan Chang, Hwai-Jung Hsu, and Kuan-Ta Chen. Modeling exercise relationships in e-learming: A unified approach. In Proceedings of the 2015 International Educational Data Mining Society, pages 532-5522 032-535,50 2015.
[5] Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjic Yan, and Wanli Ouyang. Glit: Neural architecture search for global and local image transformer. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, pages 12-21, 2021.
[6] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching trans- formers for visual recognition. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, pages 12270-12280, 2021.
[7] Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Baek, Byungsoo Kim, Yeongmin Cha, Dongmin Shin, Chan Bae, and Jaewe Heo. Towards appropriate query, key, and value computation for knowledge tracing. Proceedings of the Seventh ACM Conference on Learning
Scale, pages 341-344, 2020.
[8] Youngduck Choi, Youngnam Lee, Dongmin Shin, Junghyun Cho, Seoyon Park, Seewoo Lee, Ji- neon Baek, Chan Bae, Byungsoo Kim, and Jaewe Heo. Ednet: A large-scale hierarchical dataset education. Proceedings of the 2020 International Conference on Artificial Intelligence in Education, pages 69-73. Springer, 2020.
[9] Albert T Corbett and John Anderson. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-adapted Interaction, 4(4):253-27, 1994.
[10] Antoine Cully and Yiannis Demiris. Online knowledge level tracking with data-driven student models and collaborative filtering. IEFE Transactions on Knowledge and Data Engineering, 32(10))2000-2013 2019.
[11] Xinyi Ding and Eric Larson. Automatic rnn cell design for knowledge tracing using reinforce- ment learning. In Proceedings of the Seventh ACM Conference on Learning Scale, pages 285-288, 2020.
[12] Cornelia Dowling and Cord Hockemeyer. Automata for the assessment of knowledge. IEEE Transactions Knowledge and Data Engineering, 13(3):451-41,, 2001.
[13] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997-2017 2019.

---

[14] Engui Fan. Extended tanh oun-function method and its applications to nonlinear equations. Physies Letters A, 277(4-5):212-2115 2000.
[15] Mingyu Feng, Neil Heffernan, and Kenneth Koedinger. Addressing the assessment challenge with online system that tutors as it assesses. User Modeling and User-adapted Interaction, 19:243-266, 2009.
[16] Aritra Ghosh, Neil Heffernan, and Andrew S Lan. Context-aware attentive knowledge tracing. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 2330-2339, 2020.
[17] Tanja Käser, Severin Klingler, Alexander G Schwing, and Markus Gross. Dynamic bayesian networks for student modeling. IEEE Transactions Learning Technologies, 10(4))450 462, 2017.
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
[19] Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. survey of transformers. arXiv preprint 2tt.:21000045554., 2021.
[20] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 1r06vv:806600055, 2018
[21] Qi Liu, Shuanghong Shen, Zhenya Huang, Enhong Chen, and Yonghe Zheng. A survey of knowledge tracing. arXiv preprint arXiv:2105.15106 2021.
[22] Yu Lu, Yang Pian, Ziding Shen, Penghe Chen, and Xiaoqing Li. SLP: A multi-dimensional and consecutive dataset from K-12 education. In Proceedings of the 29th International Conference on Computers in Education Conference, volume 1, pages 261-266. Asia-Pacific Society for Computers in Education, 2021.
[23] Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Jinzhu Li, Sheng Zhao, Enhong Chen, and Tie- Yan Liu. Lightspeech: Lightweight and fast text to speech with neural architecture search. In Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5699-5703, IEEE, 2021.
[24] Haiping Ma, Jingyuan Wang, Hengshu Zhu. Xin Xia, Haifeng Zhang, Xingyi Zhang, and Lei Zhang, Reconciling cognitive modeling with knowledge forgetting: A continuous time-aware neural network approach. In Proceedings of the 31 International Joint Conference on Artificial Intelligence, pages 2174-2181, 2022.
[25] Navonil Majumder, Devamanyu Hazarika, Alexander Gelbukh, Erik Cambria, and Soujanya Po- ria. Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowledge- Based Systems, 161:124-133, 2018.
[26] Hiromi Nakagawa, Yusuke Iwasawa, and Yutaka Matsuo. Graph-based knowledge trac- ing: modeling student proficiency using graph neural network. In Proceedings of the 2019 IEEE/WIC/ACM International Conference Web Intelligence, pages 156-163. IEEE, 2019.
[27] Shalini Pandey and George Karypis. A self-attentive model for knowledge tracing. International Educational Data Mining Society, 2019.
[28] Zachary Pardos, Yoav Bergner, Daniel Seaton, and David Pritchard. Adapting bayesian knowl- edge tracing massive open online course in edx. In Proceedings of the 2013 Educational Data Mining, Citeseer, 2013.
[29] Thanaporn Patikorn, Neil T Heffernan, and Ryan S Baker. Assistments longitudinal data mining competition 2017: A preface. In Proceedings of the Workshop on Scientific Findings from the ASSISTments Longitudinal Data Competition, International Conference on Educational Data Mining, 2018
[30] Philip Pavlik Jr, Hao Cen, and Kenneth Koedinger. Performance factors analysis-a new alternative to knowledge tracing. Online Submission, 2009.
12

---

[31] Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. Deep knowledge tracing. In Proceedings of the 2015 Advances in Neural Information Processing Systems, volume 28, 2015.
[32] Ofir Press, Noah A Smith, and Omer Levy. Improving transformer models by reordering their sublayers. In Proceedings of the 58th Annuul Meeting of the Association for Computational Linguisties, pages 2996-3005, 2020.
[33] Shi Pu, Michael Yudelson, Lu Ou, and Yuchi Huang. Deep knowledge tracing with transformers. In Proceedings of the 2020 International Conference on Artificial Intelligence in Education, pages 252-256. Springer, 2020.
[34] Labs Riidd. Riidd AIEd Challenge 2020. riiid-test- answer prediction 2020.
https:
[35] Xian Shi, Pan Zhou, Wei Chen, and Lei Xie. Darts-conformer: Towards efficient gradient-baset neural architecture search for end-to-end ast. arXiv preprint 1ttp::2200.028665 2021.
[36] Dongmin Shin, Yugeun Shim, Hangyeol Yu, Seewoo Lee, Byungsoo Kim, and Youngduck Choi. Saint+: Integrating temporal features for ednet correctness prediction. In Proceedings of the 11th International Learning Analytics and Knowledge Conference, pages 490 390-496, 2021.
[37] Anupriya Shukla, Hari Mohan Pandey, and Deepti Mehrotra. Comparative review of selection techniques in genetic algorithm. In Proceedings of the 2015 International Conference on Futuristic Trends Computational Analysis and Knowledge Management, pages 515-519. IEEE, 2015.
[38] David So, Quoc Le, and Chen Liang. The evolved transformer. In Procedding of the 2019 International Conference Machine Learning, pages 5877-5886. PMLR, 2019.
[39] Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang. Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vitas: Vision transformer architecture search. Proceedings of the 2021 European Conference on Computer Vision, pages 139-157. Springer, 2022.
[40] Ye Tian, Xingyi Zhang, Chao Wang, and Yaochu Jin. An evolutionary algorithm for large-scale sparse multiobjective optimization problems. IEEE Transactions on Evolutionary Computation, (4(2)38003333 2020.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 2017 Advances in Neural Information Processing Systems, pages 5998-6008, 2017.
[42] Jill-Jênn Vie and Hisashi Kashima. Knowledge tracing machines: Factorization machines for knowledge tracing. In Proceedings of the 2019 AAAI Conference on Artificial Intelligence, volume 33, pages 750-757, 2019.
[43] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antonn Manzagol. Extracting and composing robust features with denoising autoencoders, In Proceedings of the 25th International Conference on Machine Learning, pages 1096-1103, 2008
[44] Chenyang Wang, Weizhi Ma, Min Zhang, Chuancheng Lv, Fengyuan Wan, Huijie Lin, Taoran Tang, Yiqun Liu, and Shaoping Ma. Temporal cross-effects in knowledge tracing. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 517-525. 2021.
[45] Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware- transformers for efficient natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7675-7688,
2020.
[46] Yujing Wang, Yaming Yang, Yiren Chen, Jing Bai, Ce Zhang, Guinan Su, Xiaoyu Kou, Yunhai Tong, Mao Yang, and Lidong Zhou. Textnas: A neural architecture search space tailored for text representation. In Proceedings of the 2020 AAAI Conference on Artificial Intelligence, volume 34, pages 9242-9249, 2020.
13

---

[47] Frank Wilcoxon, SK Katti, Roberta A Wilcox, et al. Critical values and probability levels for the wilcoxon rank sum test and the wilcoxon signed rank test. Selected Tables in Mathematical Statistics, 1:171-259. 1970.
[48] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang. and Tieyan Liu. On layer normalization the transformer architecture. In Proceedings of the 2020 International Conference on Machine Learning, pages 10524-10533.
PMLR, 2020.
[49] Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, and Tie-Yan Liu. Nas-bert: Task- agnostic and adaptive-size bert compression with neural architecture search. In Procedding of the 202/ ACM Conference on Knowledge Discovery and Data Mining, 2021.
[50] Jin Xu, Xu Tan, Kaitao Song, Renqian Luo, Yichong Leng, Tao Qin, Tie-Yan Liu, and Jian Li. Analyzing and mitigating interference neural architecture search. Proceedings of the 2022 International Conference on Machine Learning, pages 24662.2 PMLR, 2022.
[51] Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. In Proceedings of the 2020 European Conference on Computer Vision, pages 702-717. Springer, 2020.
[52] Zhixiong Yue, Baijiong Lin, Yu Zhang, and Christy Liang. Effective, efficient and robust neural architecture search. In Proceedings of the 2022 International Joint Conference Neural Networks, pages 1-8. IEEE, 2022.
[53] Jiani Zhang, Xingjian Shi, Irwin King, and Dit- Yan Yeung, Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th International Conference on World Wide Web, pages 765-774, 2017.
[54] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, pages 8697-8710, 2018
14

---

Appendix A
Proposed Approach
Encoding Strategy
In the proposed search space, each candidate Transformer architecture can be represented by the following vector-based encoding:
A [En: bDe (lo f0,-.. 94).
901, g02
{0, (0,1,2))xx27)
bgn = (b1, (6, .i.... bNum) [0, 1] 1x Num
,15 i v Num
bDe = (b1, b2, ... bi.... bNum)
E {0,1}
,1 i v Num
(lo.go go2)
(14)
(lo.go1.goz
2N
(lo.go1.go2)
where Num denotes the number of available input embedding and N denotes the number of en- coder/decoder blocks. bgn and bDde are used to specify which input embedding are selected for the inputs to encoder and decoder parts, and the triplet (lo, go1, go2) is used to specify three opera- tions adopted for each encoder block or each decoder block, i.e., lo, go1, and go2 determine which operation used for LO(.), (.) and GO2(.).
Encoding
FFN
MIHIA
[2,0,1]
[4,2,2]
Conv 5
Zero
MHSA
Conv 11
[1,0,1]
[0,2,]]
FFN
FFN
b [1,1,0,0,00 bDe 10,0,1,001]
Conv 3
Zero
MIHSA
Zero
Hier(
Hier
X embed
1,1,0,0,00 0,0,1,0,1 1,0,1, 2,0,1, 0,2,1 4,2,1
Figure 5: An illustrative example of the utilized encoding strategy.
For better understanding of the utilized encoding strategy, Fig- 5 gives an illustrative example, where Num and N are set to 5 and 2, respectively. The given architecture takes the first and second embedding as the input to the encoder part, and takes the third and fifth embedding as the input to the decoder part, its two encoder blocks are encoded by [1, O, 1] and [2, 0, 1], and two decoder blocks are encoded by 0,, 2, 1] and [4, 2, 2). As a result, the plotted architecture's 's encoding is 0, 1, 1, 2, 1, 0, 2, 2,
Super-Transformer Training
Inspired by one-shot NAS approaches (20), we use trained super-Transformer mode to reduce the computational cost, which can provide the weights to directly evaluate the validation AUC values of subsequently sampled sub- Transfrrferrmrr architectures. To make the supernet be trained well, we consider adopting the sandwich training rule
[51], which has been the most effective way and widely
adopted by one-shot NAS approaches [52 50 The original sandwich training rule commonly
employs three sub-models for one batch of training, including the largest sub-model, the smallest sub-model, and randomly sampled sub-model, where three forward passes and one backward pass executed for one batch update.
15

---

Algorithm Main Steps of ENAS-KT
Input: Gen: Maximum number of generations; Pop: Population size; S: Search space
Output: Pu: The best individual;
1: + Randomly initialize Pop individuals; Population initialization
2: Evaluate each individual in P based on the trained super- Transformer and obtain its AUC value;
3: for 1 to Gen do
4: P' +- Select Pop parent individuals from Population P; % Mating pool selection
5: Q Apply the single-point crossover and bit-wise mutation P based on current scarch space S;
6:
Evaluate each individual in P based on trained super- Transfomerr and obtain its AUC value;
7:
S < Search Space Reduction(S, P, Q); % Reduce search space
8:
P <- Individual Selection Q);
9: end for
10: Select the best individual Pb from as output;
11:
return Pb:
To make the training rule adapt the proposed super-rransformer, we utilize three sub-models, includ- ing a global sub-model suby. local sub-model subi, and a randomly sampled sub-model suby, for one batch of training, whose encoding can be represented by
sub, = b biee (0, 1). ... 20, ... (0,
subt = b bi.. (1, 0, 0), ... ,,,,,,,,,,,,,,,. (1,0,0), ,,,),,,,,,,,,0000
(15)
sub, [be.. bb.. (lo, go1, go2)", ...- (lo, go1.g00""''' .... (lo, go1, go2)"]
Here En be and (lo, go,, go2)" denote these vectors are randomly generated. The global sub- model suby denotes the architectures with pure global operations but no local operations, where ben and bae are randomly generated, to is set to O, goi and go2 are set to 2 and 1. The local sub-model suby refers to the architectures consisting only convolution operations with the smallest kernel size, where ben and ba are also randomly generated, lo is set to 1, both go, and go2 are set to O.
As result, there are three forward passes and one backward pass executed one batch for updating the super- Transforerr. In each batch of training, we employ the Adam [18] optimizer to update the mode parameters via minimizing the binary cross-entropy loss 43 between the predicted response A and the true response
C L=
E (rilogfi + (1 - (1-r:)logs) rog( - -f:)).
(16)
Evolutionary Search
The main steps of the proposed evolutionary search approach are summarized in Algorithm T which consists of the following six steps.
First, population containing Pop individuals will be randomly initialized. During the initialization. it is worth noting that ben and bp in each individual have to satisfy the constraints, i.c., 1 S sum(ben) and < sum(bDe). By doing so, it can ensure there is least one input embedding for the encoder part and decoder part. Second, the standard binary tournament selection [37] is used select individual to form the mating pool P', which randomly selects two individuals from P each time and keeps the better one in P' until the size of P' is equal to Pop. Next, select two individuals from P', and then apply the single-point crossover and the bit-wise mutation 40 to the two individuals to generate two offspring individuals, where the generation process needs to be consistent with current search space S. As a result, an offspring population Q with the size of Pop is obtained. Fourth, the validation AUC values of individuals in offspring population are efficiently evaluated by extracting their needed weights from the trained super- Transformer. Fifth, delete some worse operations from the search space by devised search space reduction strategy, which makes statistics from the population P and the offspring population Q and sorts all operations according to the computed fitness. Sixth, all individuals with better validation AUC values in the population P and the offspring population Q will be kept as new population.
The second to the sixth step will be repeated until the maximal number of generations Gen is exceeded. Finally, the best architecture will be output and then fine-tuned based on the inherited weights from the super- Transformer by minimizing the training loss function in (16).
16

---

Search Space Reduction
The original search space S has the same length as A, where the item on each bit is an operation set. For example, the item on first bit is {0, 1), the items on the bits corresponding to lo are {0, 1, 2, 3, 4). To accelerate the convergence of the EA, we further devise search space reduction strategy to iteratively remove some worse operations from the search space.
Algorithm has summarized the basic procedure. First, population P and offspring population Q are combined, two empty lists SF and SF atd are used to record each operation fitness and fitness standard deviation of each operation set. Then, get the operation set BitopSet on the i-th bit of current search space S. Third, compute the fitness value of each operation in BitopSet by averaging the fitness of the individuals that contain the corresponding operation. Next, filnessopSer, the computed fitness of the set of operations, will be stored as i-th item in SF, and the fitness standard deviation of fitnessOpSet will be also computed and stored in SF Ftd.d. The second to the fourth step will be repeated for all bits in S, where |S| denotes the number of bits, i.c., the length. Based on SF and Fttd the worst two operations will deleted, and the worst operation in the operation set with the largest standard deviation will be also removed, where there is at least one operation kept for each bit. By doing so, the ever-reducing search space makes the search process easier.
Algorithm 2 Search Space Reduction
Input: S: Current search space; P: Population; Q: Offispring population;
Output: RS: Reduced search space;
I: + PUP, Q.FFFF +- 0, Fstd +- 0;
SF
2: for i :ori=1 t [S] do
3: BilopSet = =S(i). filnessppee <- 0: % OpSel: The operation set on i-th bit
4: for op in Bitonset do
5: fit + The fitness value averaged on the individuals containing op P: fitnessOpset +- fitnessopsa fit; 7: end for
6:
8: SFR] = filmessopseti % Store the fitness of the set of operations i-th bit
SFeeald = Stdesssnesssppee) % Compute and store the fitness standard deviation for the operation set on i-th bit
10: end for
11: RS t Delete the worst two operations from S based on SF;;
12: RS Delete the worst operation from the operation with the largest standard deviation in RS based on
SFstai
13: return RS;
Appendix B Experiments
Table Performance comparison under splitting setting of training and testing of 60% and 30%.
The results of proposed ENAS-KT and compared KT approaches on EdNet and RAIEd 2020 datasets are reported terms RMSE, ACC, and AUC, averaged over 5-fold vross-validation. The Wilcoxon rank sum test with -=05 is also executed.
Dataset
Metric
DKT HawkesKT CT-NCM SAKT AKT SAINT SAINT+ NAS-Cell Ours
RMSE +
0.4661
0.4479
0.4372
0.4411 0.4396 0.4326
0.4288
0.4350
0.4215
EdNet
ACC
0.6507
0.6879
0.7048
0.6993 0.7019 0.7128
0.7182
0.7138
0.7289
AUC t
0.6908
0.7475
0.7733
0.7645 0.7683 0.7816
0.7906
0.7782
0.8055
RMSE
00.651
0.4466
0.4357
0.4385
0.4368 0.4313
0.4275
0.4311
0.4207
RAIEd2020
ACC f
0.6601
0.6905
0.7065
0.7019
0.7076 0.7140
0.7189
0.7164
0.7306
AUC t
0.7057
0.7491
0.7751
0.7665
0.7640 0.7855
0.7927
0.7828
0.8081
+/-/R: (six results totally)]
6/0/0
6/0/0
6/0/0
6/0/0
6/0/0
6/0/0
6/0/0
6/0/0
Datasets Descriptions
In our experiments, two largest and most challenging real-world education datasets, EdNet g and RAIEd2020 [34]. were used. Different common education datasets, such ASSISTments09 [1].
17

---

Table 6: Performance comparison under splitting setting of training and testing of 50% and 40%. The results of proposed EÑAS-KT KT and compared KT approaches on EdNet and RAIEd 2020 datasets are reported in terms of RMSE, ACC. and AUC, averaged over 5-fold cross-validation. The Wilcoxon rank sum test with a=0.05 is also executed.
Dataset
Metric
DKT HawkesKT CT-NCM SAKT AKT SAINT SAINT+ NAS-Cell Ours
RMSE_ +
0.4664
0.4480
0.4371
0.4423 0.4397 0.4325
0.4286
0.4352
0.4219
EdNet
ACC1 f
0.6503
0.6876
0.7055
0.6986
0.7026 0.7121
0.7182
0.7133
0.7291
AUC 1
0.6902
0.7473
0.7732
0.7640
0.7678 0.7811
0.7903
0.7779
0.8052
RMSE L
0.46599
0.4469
0.4354
0.4375
0.4398 0,4315
0.4276
0.4312
0.4200
RATEd2020
ACC t
0.6588
0.6902
0.7068
0.7023
0.7010 0.7146
0.7187
0.7165
0.7309
AUC t
0.7043
0.7484
0.7759
0.7674
0.7668 0.7851
0.7928
0.7835
0.8085
+/-/Rs (six results totally)]
6/0/0
6/0/0
6/0/0
6/0/0
6/0/0 6/0/0
6/0/0
6/0/0
JunYi 4 and so on 22 29,
EdNet collected by Santa is currently the largest open available education benchmark dataset, which contains over 90M interactions and nearly 800K students, while RAIEd2020 first appeared in the Kaggle is another publicly available large-scale real-world dataset, which consists of nearly 100M interactions and 400K students.
More Results under Other Two Splitting Settings
In previous experiments, all students in each dataset were randomly split into 70%, 10%, and 20% for training, validation, and testing, respectively. Under this splitting setting, the architectures found by the proposed ENAS-KT KT exhibits better performance than comparison KT approaches.
For more convineing results, we adopted other two settings to split the dataset into training and test datasets for evaluating the model performance. The first one is to split 60% of the dataset as the training dataset and 30% of the dataset as the testing dataset, while the second one takes the splitting ratio of 50% and 40%. The overall comparison results under the two settings are summarized in Table and Table where the average RMSE, ACC, and AUC values over 5-fold cross- vols-vvlldationn are reported, the best results are in bold, and the Wilcoxon rank sum test with a=0.05 is also executed.
As can be seen from these two tables, the performance of nearly all approaches on both two dataset decrease when the size of the training dataset is reduced while the size of the testing dataset is increased, but some approaches (such as CT-NCM, SAKT, AKT, SAINT+, NAS-Cell, and our approach) hold slightly better performance on RAIEd2020 under the splitting setting of 50% and 40% than that of the splitting setting of 60% and 30% Under whichever splitting setting, the architectures found by the proposed approach still outperform the tate-of-the-aa KT approaches, which can be seen from the statistical results of the Wilcoxon rank sum test.
Selected Features
Due to page limit, previous experiments do not show the selected features of the best-found archi- tecture on the RAIEd2020 dataset. For a more comprehensive observation and comparison, Fig, 6 presents the selected input features of four models, i.c., SAINT, SAINT+, best-found architecture on EdNet, and best-found architecture on RAIEd2020. It can be found that the selected features in SAINT and SAINT+ are a subset of the selected features in both two best-found architectures, where the elapsed time embedding and the lag time embedding are considered in two est-found architectures, and extra features (such as tag. bund, and tagset) are also considered. Besides, we can also find that the selected input features of these two found architectures are very close, where there is only one input feature difference (i.e., cate_ vale_lag_di between their encoder parts, and only two input features (bund/e:rpla and lagsel) are different between their decoder parts. This implies that the selected input features are more important for our Transformer architectures to solve the KT task than those not selected by the two architectures.
18

---

cont lag
cale ela
tagset
cont lag
calle ela
tagset
sk
tag
cate lag_
sk
tag
cate
lag. m
exer
ans
cont ela
bund
cate log.
cate lag_d
exer
ans
cont ela
ex exp
cale
lag,
a0o0cer
enooder dc00de3
05000e0
SAINT
SAINT
enooder
enoode decoder
decoder
Archtectee (EdNet)
Best A Acchteecture (RATEd2020)
Non-selected
Selected
Figure The visualization of selected input features of SAINT, SAINT+, and two best-found
architectures on EdNet and RAIE42020.
40
38.2
3.6
35
SuperNet Training Cost
34.6
Cost of Searching by EA/RL
30
9.7
Single Architecture Training Cost
Runtime (h)
25
24.9
20
3.6
1.7
10
9.1
1.2
6.3
1.6
7
5
5.5
5.1
2.8
3
3.5
1.7
0.9
0
ENAS-KT
ENAS-KT(f)
NAS-Cell
DKT
HawkesKT
CT-NCM
SAKT
AKT
SAINT
SAINT
Figure 7: The runtime comparison between ENAS-KT, ENAS-KT(D) and KT approaches on Ed- Net, which reports the cost of searching, supernet and single architecture training for NAS-based approaches.
Limitation Discussion
The search cost of neural architecture search (NAS)-based approaches are always the focus of researchers, especially when they are compared to non-NAS-bast approaches. To investigate this, variant approach ENAS-KTTO and
Fig. 7 compares the runtimes of proposed ENAS-KT, comparison KT approaches. Here the runtimes of ENAS-KTT, ENAS-KT(f) and NAS-Cell contain the cost of supernet training, architecture searching and single architecture retraining, while runtimes of other KT approaches are only used for training their models.
be observed that the training cost of the architecture found by the proposed ENAS-KT 3.6h, which is better than CT-NCM and AKT, worse than NAS-Cell -ell, DKT, and HawkesKT, and competitive other KT approaches. Besides, the search cost of the proposed ENAS-KT is 34. 24.6h, leading to total cost of 38.0 68.6h for ENAS-K' KT, which is higher than that of NAS-Cell (6.3h). This is because Transformer architectures thus need more time to train them. However, considering the significant performance leading of the proposed ENAS-KT over comparison approaches, the high overall cost of the proposed approach acceptable for the KT task. Morcover, note that SNAS-KT(O is variant of the proposed ENAS-KT,
the proposed approach are more complex than that of NAS-Cell and
19

---

which searches for architectures under a smaller supernet with fewer training epochs. Specifically, the embedding size halved to 64 for the supernet, the number of training epochs is also halved 30, but the best-found architecture holds the same settings as Ours for fair comparison. As shown in holds a 0.8036 AUC value. Although its performance
T the architecture found by ENAS-K is worse than ours, shown in Fig 7 its overall cost only takes 9.1 hours (3.8hours for supernet training, 1.7 hours for evolutionary search, and 3.6 hours for final training), which is fewer than the proposed ENAS-KT and competitive to NAS-Cell and others.
In summary, the current settings adopted in the proposed approach provide significantly better performance with a much higher cost, but our approach can also achieve competitive results at a much lower cost, whose performance still better than other KT approaches and whose cost competitive. In future work, we would like to explore surrogate models in the proposed approach to further reduce the search cost.
In addition to the search cost, the validity of the proposed approach also suffers from one external threat, i.e., too many available input features in new datasets (Num is large). The hierarchical feature fusion manner in Equation 10 utilizes pairwise concatenation, which requires a large number of parameters. When the datasets hold too many input features, the number of feature pairs will increase exponentially, e.g., 10 features need 45 concatenation operations while 20 features need 190 concatenation operations, which further causes a high increase the number of network parameters and deteriorates the models' efficiency. Therefore, the validity and efficiency of the proposed approach will be highly affected by the number of available input features in new datasets. But because nearly all known education datasets hold fewer available features, the proposed approach is effective and can counter most datasets in KT. future work, we would like design novel fusion modules to aggregate features without considering the influence of datasets to be handled.
20


Reviews:
[{"review_id": "xoTexfXykq", "reviewer": "Reviewer_rdQD", "summary": "The architecture of a transformer model for a knowledge tracing task is searched for using evolutionary computation. The found solutions perform well.\n\n", "questions": "Questions:\n- What is \"Fusion\" method?\n- l135 Mashed=Masked?\n- Is a supernet with EA a fancy dropout?\n- What is the magnitude of the search space? Is it N^2 * C^{2N} ?\n- What post-hoc adjustment did you use?\n- Were the ablation results significant?\n", "limitations": " none.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- ANN design is difficult \n- KT is important\n- Simple NAS method", "weaknesses": "- Interpretability of the solution, i.e. ANN\n- Limited discussion regarding computational cost. E.g. how much improvement for the additional search\n- Readability of only acronyms introduced, e.g. LFA, PFA, KTM\n- Figure 1 is non-intuitive\n- Limited discussion regarding transfer of the solutions to other KT datasets", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Question 1\n*The 'fusion' method* refers to that **a hierarchical fusion module** is employed in our approach to **aggregate** the input features fully. \n\nIn **the hierarchical fusion module**,  the pairwise concatenation between input features is first performed to obtain some intermediate temporary features, then aggregate all temporary features as the final input feature by concatenation. \n\nThe above **aggregation** process for input features is called *the 'fusion' method*.\n\n## Response to Question 2\nWe apologize for ***the typo in line 135***.  It should be $Masked\\\\_MHSA(\\cdot)$: a masked multi-head self-attention module in Eq.(3) of the paper. The mask operation prevents the current position from accessing the later positions. Many thanks and we will revise this error.\n\n## Response to Question 3\nIn my opinion, the answer is yes. In a network with dropout, some weights will be set to active while others are set to inactive (zeros) to get the output of the network, the output of the network is only determined by the activated weights. Actually, in the proposed supernet, there exist many paths from its input to its output, and each path represents a sub-model. During the search with EA, when one path is active, other paths are set to inactive in the supernet, and the output of the supernet is only determined by the activated sub-model, i.e. the sub-model's output is the output of the supernet. Here the EA is used to find the best path in the supernet. \n\nAs can be seen, the search process of the supernet with EA is similar to dropout, where some paths are set to inactive to get the model output. Thus we agree with the reviewer that the supernet with EA is dropout-like.\n\n## Response to Question 4\nThe magnitude of the search space is  $Num^2*2^{2(Num-1)}\\*(3\\*3\\*5)^{2N}$, where $Num$ denotes the number of available input features and $N$ represents the number of encoder/decoder blocks.\n\nTo be specifically, the architecture in the search space can be represented by $ [\\mathbf{b_{En}},\\mathbf{b_{De}}, 2N \\times (lo\\in\\\\{0,1,2,3,4\\\\}, go_1\\in\\\\{0,1,2\\\\}, go_2\\in\\\\{0,1,2\\\\})]$. $\\mathbf{b_{En}}$ ($\\mathbf{b_{De}}$)  is a binary vector with the length of $N$, whose at least one bit is 1. \n\nFor $\\mathbf{b_{En}}$ in the feature selection part, it is first to determine which bit is 1, leading to $Num$ choices; \nand the remaining $Num-1$ bits will provides $2^{Num-1}$ choices; as a results, there are $Num\\*2^{Num-1}$ choices for  $\\mathbf{b_{En}}$. \n\nIt is same for $\\mathbf{b_{De}}$ to know there are also $Num\\*2^{Num-1}$ choices.\n\nFor each encoder/decoder block, $lo$ has 5 candidate operations, $go_1$ has 3 candidate operations, and  $go_2$ also has 3 candidate operations, which leads to $5\\*3\\*3$ choices in each block; thus, $N$ encoder blocks and $N$ decoder blocks provides $(5\\*3\\*3)^{2N}$ choices totally.\n\nIn summary, $\\mathbf{b_{En}}$, $\\mathbf{b_{De}}$ and $2N$ blocks totally provide $Num\\*2^{Num-1}\\times Num\\*2^{Num-1}\\times(5\\*3\\*3)^{2N}$ choices, i.e., $Num^2*2^{2(Num-1)}\\*(3\\*3\\*5)^{2N}$.\n\n## Response to Question 5\nWe did not use any post-hoc adjustment method for the Wilcoxon rank sum test, because we compared each KT approach with our approach to assess whether there is a significant difference.\n\nThe Wilcoxon rank sum test, also known as the Mann-Whitney U test, is typically used to compare two independent groups. In this context, post-hoc tests are not necessary because we're only comparing two groups.  Post-hoc tests are typically used in the context of analysis of variance (ANOVA) to compare multiple groups after finding a significant difference.\n\nIn this paper, we employed the Wilcoxon rank sum test to analyze the significance because this test method has been used for various scenarios, such as the performance comparison in evolutionary computation [1~3]. \nIn these studies, the Wilcoxon rank sum test is directly used to assess whether there is a significant difference between one compared approach and their approach without using additional post-hoc tests.\n\n[1] Song Z, et al. Balancing Objective Optimization and Constraint Satisfaction in Expensive Constrained Evolutionary Multi-Objective Optimization[J]. IEEE Transactions on Evolutionary Computation, 2023.\n\n[2] Cheng R, et al. A reference vector guided evolutionary algorithm for many-objective optimization[J]. IEEE Transactions on Evolutionary Computation, 2016.\n\n[3] Kropp I, et al. Improved Evolutionary Operators for Sparse Large-Scale Multiobjective Optimization Problems[J]. IEEE Transactions on Evolutionary Computation, 2023.\n\n## Response to Question 6\nTo answer the question, we have executed the significance analysis based on the previous ablation results, \nand the overall results have been summarized in  **Table II** of ***global_response***.\n\nAs can be seen, the Wilcoxon rank sum test with a significance level α=0.05 is performed to compare each variant architecture\nwith **ours**. Finally, we can find that **ours** is significantly better than each variant architecture, demonstrating that the ablation results are significant.\n\n## Response to Weakness \n1.The found solution indeed does not have good interpretability, which will be one of our future research directions.\n\n2.**Fig.1(a)  and Table II** show its cost and improvement: the supernet after training holds a 0.7847 AUC value, and the search process takes 9.7 hours to find an architecture $\\mathbf{E}$ (**ours**  without retraining) having a 0.7969 AUC. The high improvement demonstrates that the additional search is deserved.\n\n3&4.We will give all full names for all acronyms, and replot Figure 1 for better understanding and intuitive observation.\n\n5.We ever considered a similar question, but the solutions cannot be directly used for other datasets because the available input features of other datasets are not the same as the two datasets in our approach. \n\n### We appreciate your valuable feedback and will revise the paper based on the above."}, {"review_id": "MFRm5UVg7w", "reviewer": "Reviewer_YzcD", "summary": "This paper introduces neural architecture search (NAS) for Transformer in the field of Knowledge Tracing (KT) for the first time. The authors propose a Transformer architecture that combines local and global paths, as well as a search space, to address the issue of students’ forgetting behavior in the knowledge tracing field. In this process, they propose a hierarchical fusion method that selects and fuses embeddings from various features. The proposed model demonstrates outstanding performance in experiments conducted on two large knowledge tracing datasets.\n", "questions": "- The reported ablation study (Table 3) does not demonstrate the dependencies between each component. It would be beneficial to describe the contribution of each component to the final model's performance. In particular, it would be valuable to know whether the performance improvement is primarily due to the model architecture (Figure 3) or the feature selection (Figure 4a).\n\n- Figure 2 appears to require clarification. In particular, the meaning of the figure regarding “search space reduction” is unclear.\n\n- The proposed model requires a computational cost that is at least 5 times higher compared to existing models, with most of the cost attributed to the challenging super-Transformer training cost, which is difficult for users to optimize (Appendix, Figure 3). Is this cost an inherent limitation of proposed evolutionary NAS? Can sufficiently competitive results be achieved even with lower costs (e.g., smaller architecture, fewer training steps)?\n", "limitations": "- The authors well summarized the limitations of the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The attempt to apply Transformer NAS in the field of knowledge tracing is innovative. Moreover, the search space they introduced to apply Transformer NAS in the KT field is convincingly related to students' forgetting behavior.\n\n- To reduce the high computational cost of evolutionary algorithms, the method used by the authors efficiently decreased the computational cost to a competitive level.\n", "weaknesses": "- Hierarchical fusion is excessively large architectures. The authors suffered from high computational costs and had to limit the number of feature candidates (Appendix 2, Limitation Discussion). While the proposed fusion method is intriguing, it would be beneficial to conduct more experiments on datasets with a large number of features, datasets with limited training data, or specifically focusing on the hierarchical fusion method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Question 1\nAccording to your suggestion, we have added more variants to the original Table 3 (forming **Table II** of   ***global_response***) to show the effectiveness of searching and the contribution of each component to the final model's performance.\n\nFirstly, two variant architectures $\\mathbf{H}$ and $\\mathbf{I}$ are created to answer the reviewer's main concerns:\n-  $\\mathbf{H}$ is our final model without the input feature selection part, which directly takes  all available features as input features and aggregates the features as the  model's input  by **the simple sum operation**;\n- $\\mathbf{I}$ is our final model without the model architecture found in Figure 3, which employs the vanilla Transformer architecture (same as SAINT+) as the model architecture. Here $\\mathbf{I}$ is equal to $\\mathbf{C}$ in the submitted paper.\n\nAs can be observed from the results of $\\mathbf{H}$ and $\\mathbf{I}$ in  **Table II** of   ***global_response***, both the model architecture part and the hierarchical feature selection part play important roles in performance improvement, but the model architecture part is more important, whose performance improvement on AUC  is more than that of the hierarchical feature selection part by 0.002. \n\nSecondly, we  added  two more  variant architectures  $\\mathbf{F}$ and $\\mathbf{G}$ to analyze the contribution  of the selected features and the hierarchical fusion module  to the final model's performance:\n- $\\mathbf{F}$  denotes our final model without the hierarchical fusion module, which directly utilizes the concatenation operation to aggregate the embeddings of selected features;\n- $\\mathbf{G}$  denotes our final model without using the selected features, which directly takes all available features as input features and aggregates them by the hierarchical fusion module.\n\nAs can be seen from the results of $\\mathbf{F}$ and $\\mathbf{G}$, both the selected features and the hierarchical fusion module are effective in improving the final model's performance,  and the selected features are more important than the hierarchical fusion module.\n**PS**: the importance of the two parts changes when applied to vanilla Transformer from the results of $\\mathbf{A}$, $\\mathbf{B}$, and $\\mathbf{C}$.\n\nBesides, we also added two architectures' results to **Table II** of  ***global_response*** to show the effectiveness of evolutionary search:\n-  \"Super-Transformer (after training)\" refers to the performance of the super-Transformer after training. The forward pass process for each prediction is randomly sampled, thus its performance can be seen as the average performance of all  sampled architectures；\n- $\\mathbf{E}$ denotes our final model without retraining, inheriting weights directly from the super-Transformer.\n\nThus, we can conclude that the utilized evolutionary search effectively finds high-performance architectures to some extent.\n\n \n## Response to Question 2\n\nIn Figure 2, there are two main processes: **(1)** training a super-net and **(2)** searching via an evolutionary algorithm.  Here the first process will not be introduced because it has been presented in **Section 1-2 of Appendix**.\n\nThe second process contains six steps, \n- First, i.e., **Population initialization** in Figure 2,  randomly generate $Pop$ individuals and form a population $P$;\n- Second, i.e., **Mating pool selection**,  randomly select two individuals from $P$ each time and keep the better one in $P'$ until the size of $P'$ equal to $P$;\n- Third, i.e., **Offspring generation**, randomly select two individuals from $P'$ each time,  apply crossover and mutation to the two individuals to generate two offspring individuals, forming offspring $Q$;\n- Fourth, i.e., **Individual evaluation**,  decode each individual in $Q$ to get its neural architecture, and directly evaluate each architecture's performance based on the inherited weights from the supernet;\n- Fifth, i.e., **Search space reduction**, (1) **Determine which operations are available**: As you can see, each column of figures in Figure 2 presents the available operations' encoding; the first column $[1,2,3,4]$ represents the available operations' encoding for the first bit is $[1,2,3,4]$, where $0$ is unavailable because it has been deleted. (2) **Compute the score of each available operation**.  (3) **Delete the operations whose scores are unpromising**: In Figure 2, X upon two figures represents these two operations are deleted in current generation.\n- Sixth, **Individual selection**, keep better individuals in $P$ and $Q$ to form the next population.\n\n## Response to Question 3\nFor the first concern, we would like to clarify that the increased computational cost is a trade-off we made to achieve improved performance and explore the potential of the super-Transformer. While it does come with a higher cost, it allows us to capture more complex student-exercise interactions, leading to better results.\n\nTo answer the second concern,  a variant **ENAS-KT(f)** is created: the embedding size $D$ is halved to 64 for the supernet, the number of training epochs is also halved to 30, but the best-found architecture holds the same settings as **ours** for a fair comparison.  \n\nAs shown in **Table II** of   ***global_response***,  **Searched** holds a 0.8036 AUC value. Although its performance is worse than **ours**, as shown in **Fig.1(a)**, its overall cost only takes **9.1 hours** (**3.8hours** for supernet training, **1.7 hours** for evolutionary search, and 3.6 hours for final training), which is fewer than **ours** and competitive to *NAS-Cell* and others.\n\nIn summary, current settings adopted in the proposed approach provide significantly better performance with a much higher cost, but our approach can also achieve competitive results at a much lower cost, whose performance is still better than other KT approaches and whose cost is competitive.\n\n### We appreciate your valuable feedback and will revise the paper based on the above."}, {"review_id": "lMqZT19hrq", "reviewer": "Reviewer_YAb1", "summary": "This paper presents neural architecture search (NAS) for transformer in the context of educational application, specifically knowledge tracing. The authors employed two level NAS:\none at a local level to encorporate forgetting behavior and architecture serach at global level for exercise and response related embedding. The authors evaluated proposed approach\non two large datasets and compared with eight SOTA KT approaches.", "questions": "How does the proposed NAS generalize to other transformer-based KTs, such as SAKT and AKT?", "limitations": "Yes, the trade-off between the time complexity of NAS is discussed in the Supplementary.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Local path for forgetting behavior encoding in KT\n2. NAS for KT.", "weaknesses": "Comparison with other neural architecture search methods is not discussed. Please see Reference [1] for NAS experiment settings.\nThe technical contribution of the paper seems rather limited. \nHow does the proposed NAS generalize to other transformer-based KTs, such as SAKT and AKT? \n\nReferences.\n1. Ding M, Lian X, Yang L, Wang P, Jin X, Lu Z, Luo P. Hr-nas: Searching efficient high-resolution neural architectures with lightweight transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition 2021 (pp. 2982-2992).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Weakness 1\nActually, we have discussed the comparison of the proposed approach ENAS-KT and existing NAS approaches for other tasks in terms of approach design in the *Related work* of the submitted paper, where the approach design comparison between ENAS-KT and NAS-Cell is also presented.\n\nBut we have to admit that there lacks a discussion about the comparison between the proposed ENAS-KT and some NAS approaches for KT tasks, especially in terms of model performance, model complexity, and overall search cost. \n\nWe would like to give a comprehensive and detailed comparison like the reference [1] you suggested by comparing our approach with many NAS approaches.\nHowever, due to the rarity of NAS-based KT approaches (only one approach NAS-Cell), the following will discuss the comparison between the proposed ENAS-KT and NAS-Cell from three perspectives:\n1. - **In terms of approach design**.   Up to now,  NAS-Cell is the first and only method to apply NAS to KT.  But NAS-Cell is simple and lacks the design for KT tasks. It directly uses ENAS[R1]  to search the optimal LSTM cell for  KT.  As you know, ENAS is a one-shot NAS based on reinforcement learning, which trains a supernet for all subsequently sampled sub-models to obtain their performance directly.\n- It can be seen that there is no KT-specific design in NAS-Cell, which still searches for the general LSTMs.  \nInstead, our approach not only devises the input feature selection part to select suitable input features but also modifies the Transformer's architecture to achieve the balance between the global and local context modeling abilities.\nIn short, our approach is well-designed for KT tasks, which can improve the model performance highly. \n\n2. - **In terms of model performance and complexity**.  To compare this, we have summarized all approaches' performance and model parameter information in **Table I**  of ***global_response***. （Since the number of parameters for KT models is related to inputs,  **Table I** presents model parameter information for each KT approach on EdNet and RAIEd2020.）\n Besides, for a deep insight into the model parameter, we further plotted  **Fig.1 (b)** in ***global_response*** to depict the model parameter comparison between ENAS-KT and KT approaches in terms of the **model main body** and **input part**. (Here *the input part* refers to the parts in  KT models to get the embeddings from inputs and aggregate these embeddings, while *the model main body*  refers to the other parts in KT models excluding *the input part*.)\n\n- As can be seen, the architecture found by the proposed *ENAS-KT* holds the most model parameters, which is worse than other KT models, especially worse than NAS-Cell.  Besides, as can be seen from **Fig.1 (b)**, although the high model parameters of **Ours**  are attributed to its input part,  the model parameters in its *the model main body*  are still worse than NAS-Cell, only competitive to SAINT, SAINT+, and AKT. Therefore,  the architecture found by our approach indeed holds significantly better performance while holding more model parameters compared to NAS-Cell.\n\n3. - **In terms of the search cost**.  To this end, **Fig.1** in ***global_response*** reports the cost for each KT approach, where the supernet training time, the search time, and the final architecture training time for *ENAS-KT*, *ENAS-KT(f)*, and  NAS-Cell are also elaborated.  Here *ENAS-KT(f)* is a smaller variant of *ENAS-KT* to show that our approach is able to final a sufficiently good architecture at a much lower cost. (*ENAS-KT(f)* uses a smaller supernet with less training:  size $D$ is halved to 64, and training epochs are also halved to 30, but its best-found architecture holds the same settings as *ENAS-KT* for a fair comparison).\n\n- As can be seen, *ENAS-KT* takes 3.6 hours, 9.7 hours, and 24.9 hours for final training,  evolutionary search, and supernet training, respectively. Its overall search cost is higher than all KT approaches including *NAS-Cell*. This is because the supernet in *ENAS-KT* is more complex and needs more epochs for training.  However,  as shown in **Table II** and **Fig.1(a)**, when *ENAS-KT* is with a smaller supernet (i.e., *ENAS-KT(f)*),  its performance is still sufficiently good and better than *NAS-Cell* but its search cost is competitive to *NAS-Cell*. \n\n In summary, the proposed approach is better than *NAS-Cell* regarding approach design and model performance, competitive to    *NAS-Cell* regarding search cost, but worse than  *NAS-Cell* regarding mode complexity.\n\n[R1] Pham H, et al. Efficient neural architecture search via parameters sharing[C]//International conference on machine learning. PMLR, 2018. \n\n## Response to Weakness 2 (Question 1)\nThe proposed approach cannot be directly applied to SAKT and  AKT,  because our search space (including the input feature selection part and the model main architecture part) is inapplicable to them, especially for the model main architecture part, which is not original Transformer. \n\nHowever, from the comparison between SAINT+ and $\\mathbf{C}$ in **Table II**, we think that the input feature selection part could be used for SAKT and  AKT to select their suitable input features to improve performance. \n\nTo this end, we executed the same supernet training and evolutionary search procedure for SAKT and AKT, and obtained their own best architectures, denoted by SAKT(NAS)  and AKT(NAS). The following is the comparison results:\n|AUC |SAKT |SAKT(NAS) |AKT |AKT(NAS)|\n|----|----|----|----|----|\n|EdNet| 0.7650|0.7898| 0.7686|0.7791|\n|RAIED2020| 0.7693|0.7925 |0.7752|0.7873|\n\nAs can be seen, the performance of both SAKT and AKT is improved but the improvement of AKT is lower than that of SAKT. This may be because the input form of SAKT is close to ours, but the query computation manner for AKT from its input is unique and different from ours.\n\n### We appreciate your valuable feedback and will revise the paper based on the above."}, {"review_id": "p6AUluPh3z", "reviewer": "Reviewer_XrVw", "summary": "This paper modifies the Transformer architecture and employs Neural Architecture Search (NAS) to tackle knowledge tracing tasks", "questions": "See above", "limitations": "I don't see the discussion of limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Writing is good and the method looks sophisticated.", "weaknesses": "As I am not specialized in the field of knowledge tracing, I cannot accurately assess the novelty of this work or the adequacy of the experiments. However, as a general ML/NLP researcher, I can offer some intuition:\n\n1. If the architecture in the Transformer is modified, then the pre-trained model is unavailable. Would it be necessary to pre-train the model from scratch.\n\n2. NAS has been extensively researched. Although this paper appears to be the first to apply NAS specifically to the knowledge tracing problem, it is not clear why NAS is considered the most suitable technique for this task. For instance, have the authors considered alternative approaches such as mask-based methods [1]? Providing a comparison or justification would strengthen the argument for utilizing NAS in this context.\n\n[1]: Continual Training of Language Models for Few-Shot Learning. EMNLP 2022\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Weakness\n\nWe would like to clarify the novelty of our work.  While it is true that NAS has been extensively researched, its application in the field of KT is indeed novel. The main challenge in KT is to model the students' knowledge states accurately, which involves capturing both the global and local context of students' learning behavior. Traditional Transformer-based KT approaches have limitations in capturing the local context, especially the forgetting behavior of students when the related records are distant in time. Our work addresses this issue by introducing convolution operations to the Transformer, enhancing its local context modeling ability. Furthermore, we propose an evolutionary NAS approach to automate the input feature selection and determine where to apply which operation for balancing the local/global context modeling. This is a good improvement in the field of KT.\n\n## Response to Weakness 1\n\nMy answer is yes. Due to the difference between model architecture, the model in our approach can not make use of existing pre-trained language models to initialize the model's weights for a warm start. \n\n- You are correct, it is the best choice to train the supernet model from scratch when the proposed approach is used for new KT datasets. \n The reason behind this is that the difference in KT datasets' input features causes the difference in the supernet's input part, and thus a supernet (whose weights) for a specific KT dataset cannot be directly used as the supernet for other KT datasets. \n\n- However, in addition to training the supernet model from scratch,  once there has been a supernet trained for one KT dataset,  it could train the supernet model from the existing trained supernet for another KT dataset.  The supernet for another KT dataset could inherit the weights of the model part of the trained supernet (excluding the input part's weights) for a good initialization, and then fine-tune the supernet to get a well-trained model.\n\n- Besides, for  all sub-models in the same KT dataset, these sampled  sub-models do not need training from scratch, because they could inherit their needed weights from the trained supernet for continued training, \nwhich is equal to the fine-tuning process.\n\n- Furthermore, the main purpose of our work is to enhance the Transformer's ability to model local context, which is crucial for capturing students' forgetting behavior in knowledge tracing. This improvement is aimed at addressing challenges specific to knowledge tracing problems, which may not have been adequately considered in pre-trained models. \nTherefore, despite the need to start training from scratch sometimes, we believe it is worth it.\n\n\n## Response to Weakness 2\n\nRegarding your question about the suitability of NAS for this task, we believe NAS is a powerful tool for our problem for several reasons.\n- As validated in previous KT research, e.g., SAINT, SAINT+,  the input features of the KT model play important roles in enhancing KT performance and improving the accuracy of tracing student's proficiency.  However, manually selecting suitable features as the KT model's inputs is labor-intensive,  and general feature selection strategies（such as embedded-based feature selection strategies）are also inapplicable, needing substantial computational costs. Thus it is necessary to propose an effective yet efficient method to find suitable features for KT models. NAS is designed to automate the process of architecture selection, which aligns with our goal of automating input feature selection in KT.  Therefore, taking the feature selection task as a NAS task and solving it with a super-based NAS approach is feasible and effective.\n\n- Traditional Transformer-based KT approaches have limitations in capturing the local context. We aim to introduce convolution operations to the Transformer, enhancing its local context modeling ability. However, finding an architecture that can balance the global and local context modeling abilities in the KT model well is difficult. In such a context,  NAS allows us to explore a vast search space of possible architectures, which is crucial for finding an optimal solution that can balance local and global context modeling in KT.\n\nWe apologize that we are unfamiliar with mask-based methods. After we read the reference (CPT) you suggested, we think such a method is similar to but different from our approach. The following is our understanding (maybe not quite right): \n- The CPT uses a \"mask-based method\". Its basic idea is to introduce a special mask in the input of the model, which can be used to indicate which parts of the input of the model are related to the current task and which parts are related to other tasks.  In this way, the model can only pay attention to the input part related to the new task while ignoring the input part related to the old task when processing the new task. In this way, the model can effectively learn new tasks without forgetting old tasks\n\n- The above process of masking some inputs of the model for a new task can be seen as sampling a sub-model from the supernet, where only the inputs and architectures related to the sampled sub-model are active while others are masked. The fine-tuning of CPT for a new task is basically equal to the fine-tuning of the sub-model in our approach. \n\n- However,  in CPT, it is known which parts for a new task need to be masked, but in our case, it is unknown which input features and which architectures are useful, which needs extra searching to find the best combination of input features and model architectures.\n\n\nNevertheless, we agree that a comparison or justification for the choice of NAS would strengthen our argument. In the revised version of our paper, we will provide a more detailed discussion on why we choose NAS over other methods, including mask-based methods.\n\n### We appreciate your valuable feedback and will make the corresponding revision based on the above."}, {"review_id": "kRHopBRpoa", "reviewer": "Reviewer_EGZn", "summary": "This paper proposes to search for the optimal Transformer-based architecture for the knowledge tracing task. The authors have effectively constructed a search space that captures both global and local context modeling to accurately capture students' forgetting behavior. To further improve the efficiency of the model, a selective hierarchical input module has been designed to automate feature selection, eliminating the need for manual selection. By training the supernet and adopting the EA search, the authors have achieved superb results in two education datasets, EdNet and RAIEd2020.", "questions": "None", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a clear motivation for the study, emphasizing the novel approach of utilizing NAS techniques to solve problems in KT, which is worth exploring and bridges the gap between KT and NAS\n- The paper is well-structured and easy to follow, making it accessible to readers.\n- The design of the search space for KT is reasonable and effective. And the experimental results seem convincing.", "weaknesses": "- The search cost should be reported, including the supernet training time and the evolutionary search time.\n- The parameters and FLOPs of all architectures in Table 2 should be reported to provide a comprehensive understanding of the effectiveness.\n- As an expert in NAS, I appreciate the authors' efforts in customizing an effective search space for KT and conducting the search for better architectures, which is a reasonable approach. However, from my perspective, it's difficult to judge the improvement shown in Table 2 without considering the change in parameters and FLOPs, which are necessary for a comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Weakness 1\nAlthough we have reported the search cost of our approach and compared approaches in  **Figure 3** of ***the previous Appendix***, \nFigure 3 only reports each approach's overall cost and the training cost of the final architecture, \nwhere the cost of the supernet training and the evolutionary search are not presented for *ENAS-KT*.\n\n**According to the reviewer's suggestion**, we have replotted Figure 3 as **Fig.1(a)** in  ***global_response***, which reports the supernet training time,  the evolutionary search time,  and the final architecture training time for the proposed approach *ENAS-KT*.\n\n- As can be seen from  **Fig.1(a)**, for the proposed *ENAS-KT*, its final architecture training takes 3.6 hours, the evolutionary search takes 9.7 hours, while the supernet training takes 24.9 hours, which accounts for a large proportion of its overall cost.  The computational cost of the proposed approach is higher than that of the traditional KT approaches (exclude *NAS-Cell*) since they are not search-based approaches, and their cost is only used to train one architecture in these approaches.\n\n- Even compared to *NAS-Cell*, the cost of the proposed approach is also non-competitive since the supernet in our approach is more complex and needs more epochs for training. To validate the effectiveness of  *ENAS-KT* under a smaller supernet trained for fewer epochs,  a variant **ENAS-KT(f)** is created. In **ENAS-KT(f)**, the embedding size $D$ is halved to 64 for the supernet, the number of training epochs is also halved to 30, but its best-found architecture holds the same settings as **ours** for a fair comparison.  \n\n- The comparison results have been summarized in **Table II**  of   ***global_response***, the best architecture found by **ENAS-KT(f)** is denoted as **Searched** and holds a 0.8036 AUC value.\nAlthough its performance is worse than **ours**, as shown in **Fig.1(a)** of   ***global_response***, its overall search cost only takes **9.1 hours** (**3.8hours for supernet training**, **1.7 hours for evolutionary search**, and 3.6 hours for final architecture training), \nwhose **search cost (i.e., 3.8+1.7 = 5.5 hours)** is significantly smaller than **ours** (34.6 hours) and competitive to *NAS-Cell* (5.1 hours).\n\nIn summary, the proposed approach under the original search settings indeed holds an extremely high computational cost due to its high cost for supernet training and evolutionary search, but the proposed approach can achieve a competitive computational cost with a bit expense of the final architecture performance. \n\n\n\n## Response to Weakness 2\nWe agree with the reviewer that comparing the parameters/FLOPs of networks is a very common metric for experiments in NAS, \nwhich is meaningful but commonly ignored in KT approaches' experiments.\n\nAs suggested by the reviewer, we would like to compare the parameters and FLOPs of all architectures in the original Table 2, \nbut we cannot find a suitable Python package to compute the  FLOPs for KT models because KT models' inputs are not as simple as CNNs' inputs that have a fixed form, which causes existing packages for FLOPs (such as fvcore, thop, and torchstat)  inapplicable.\nBesides, it is too complex for us to calculate FLOPs for each model manually.\n\n- Therefore, we only compared the number of model parameters and added the comparison to the original Table 2, forming **Table I** in ***global_response***. Since the number of parameters for KT models is related to inputs,  **Table I** presents the model parameter information for each KT approach on EdNet and RAIEd2020.\n\n- As can be seen, *HawkesKT* holds the fewest model parameters, *DKT* is the second-best, and the architecture found by the proposed *ENAS-KT* holds the most model parameters, which is worse than other KT models. Actually,  the high model parameters of **Ours**  are attributed to its input part, which is used to get the input embeddings and aggregate these embeddings. \n\n- For an intuitive observation, we further plotted  **Fig.1 (b)** in ***global_response*** to depict the number of parameters comparison between ENAS-KT and some KT approaches in terms of the **model main body** and **input part**. Here *the input part* refers to the parts in  KT models to get the embeddings from inputs and aggregate these embeddings, while *the model main body*  refers to other parts in KT models excluding *the input part*.\n \n- As can be observed, *the input part* of ENAS-KT  is the most complex, holding about 2.5 M parameters, which is worse than other KT models and only competitive to that of SAINT+. \nIt is reasonable because the architecture of  ENAS-KT has more input features to handle and contains a complex **hierarchical fusion module**. However,  in terms of *the model main body*, ENAS-KT   is competitive to SAINT, SAINT+, and AKT.\nEspecially for AKT, the number of its overall model parameters is about 1.23M, much better than ENAS-KT, \nbut the parameters in its  *the model main body* are similar to  ENAS-KT.\n\nIn summary,  the architecture found by our approach indeed holds more parameters, which is mainly caused by the input part.\n\n## Response to Weakness 3\nThanks for your recognition of our work.  According to your suggestions, we have added the model parameter comparison to the original Table 2 (forming **Table I** in ***global_response***) for comprehensive comparison, and we also presented a figure (**Fig.1 (b)** in ***global_response***) for a deep insight into the model parameters.\n\nTo sum up,  the proposed approach is non-competitive to other KT approaches in terms of the model parameters, but our *the model main body* is competitive to that of Transformer-based KT approaches because high parameters in our approach are attributed to *the input part*. As a result, it is acceptable for us to increase the model performance with a bit increasing of model parameters.\n\n### We appreciate your valuable feedback and will revise the paper based on the above."}]

Author rebuttal:
We sincerely appreciate the valuable feedback provided by all the reviewers. We have carefully addressed their questions and concerns in our response, aiming to provide satisfactory answers.

Here we uploaded a file named "global_response" to show some necessary results and comparisons, which contains two tables and one figure. 

 In all subsequent responses to reviewers,  this file is termed ***global_response*** for easy presentation and convenient discussion.

Meta-review: The paper studies the problem of neural architecture search for knowledge tracing in educational applications. The goal is to find an optimized transformer architecture that can better select the input features and balance different modeling aspects, including the student's forgetting behavior. The proposed method, ENAS-KT, is based on designing a novel search space and an evolutionary search algorithm suitable for knowledge tracing. Extensive experimental evaluation on two large-scale benchmarks showcases the effectiveness of the proposed method. The reviewers acknowledged that the paper considers an important problem setting and that the proposed neural architecture search method is of practical interest. However, the reviewers also raised several concerns and questions in their initial reviews. We thank the authors for their detailed responses and for actively engaging with the reviewers during the discussion phase. The reviewers appreciated the responses, which helped in answering their key questions. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback in their reviews, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.
Decision: Accept (poster)