A Closed-Form Solution for Fast and Reliable Adaptive Testing
Authors: Yan Zhuang
Chenye Ke
Zirui Liu
Qi Liu
Yuting Ning
Zhenya Huang
Weizhe Huang
Qingyang Mao
Shijin Wang
Venue: NeurIPS 2025 poster ()
Keywords: Computerized Adaptive Testing
Human Assessment
Psychometrics
Ability Estimation
AI Education
Primary area: machine_learning_for_sciences
TL;DR: A closed-form method for adaptive testing that selects questions efficiently and accurately without complex iterative optimization.

Abstract: Human ability estimation is essential for educational assessment, career advancement, and professional certification. Adaptive Testing systems can improve estimation efficiency by selecting fewer, targeted questions, and are widely used in exams, e.g., GRE, GMAT, and Duolingo English Test. However, selecting an optimal subset of questions remains a challenging nested optimization problem. Existing methods rely on costly approximations or data-intensive training, making them unsuitable for today's large-scale and complex testing environments. Thus, we propose a Closed-Form solution for question subset selection in Adaptive Testing. It directly minimizes ability estimation error by reducing ability parameter's gradient bias while maintaining Hessian stability, which enables a simple greedy algorithm for question selection. Moreover, it can quantify the impact of human behavioral perturbations on ability estimation. Extensive experiments on large-scale educational datasets demonstrate that it reduces the number of required questions by 10% compared to SOTA methods, while maintaining the same estimation accuracy.

A Closed-Form Solution for Fast and Reliable Adaptive Testing
Yan Zhuang', Chenye Ke², Zirui Liu', Qi Li Yuting Ning', Zhenya Huang!
Weizhe Huang', Qingyang Mao', Shijin Wang
State Key Laboratory Cognitive Intelligence, University Science and Technology of China 3: Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
2: Anhui University
4: Ohio State University 5: iFLYTEK Co., Ltd
{zykb, liuzirui, www.rrui,,u2771922220 192829988777 macqy0503>$mal
wstc edu. en,
kecyōstu. ahu edu. en, ning. 15100su edu, hhuniugl,hanngzhy/ huanghhy edu. en, sjwang3@iflytek.com
Abstract
Human ability estimation is essential for educational assessment, career advance- ment, and professional certification. Adaptive Testing systems can improve estima- tion efficiency by selecting fewer, targeted questions, and are widely used in exams, e.g., GRE, GMAT, and Duolingo English Test. However, selecting an optimal subset of questions remains challenging nested optimization problem. Existing methods rely on costly approximations or data-intensive training, making them unsuitable for today's ss large-scale and complex testing environments. Thus, we pro- pose Closed-Fores solution for question subset selection in Adaptive Testing. It directly minimizes ability estimation eπror by reducing ability parameter's gradient bias while maintaining Hessian stability, which enables a simple greedy algorithm for question selection. Moreover, it can quantify the impact of human behavioral perturbations on ability estimation. Extensive experiments on large-scale educa- tional datasets demonstrate that it reduces the number of required questions by 10% compared to SOTA methods, while maintaining the same estimation accuracy.
1 Introduction
Accurate assessment of human abilities plays crucial role in education, career advancement, and professional certification, directly influencing future opportunities. As a result, the demand for effective and efficient assessment methodologies has grown significantly 2 3. Traditional paper- and-pencil tests require examinees to answer a large number of questions, leading to cognitive load and inefficiency. In contrast, Adaptive Testing has emerged highly efficient ability estimation approach and has been widely adopted in education systems, and has been successfully integrated into various standardized testing systems
The effectiveness of adaptive testing lies in a key insight: not all questions are equally valuable for estimating ability. To achieve efficiency while maintaining accuracy, an adaptive testing system relies on two key components: 1) Question selection algorithm - Identifying and selecting the most informative subset of questions from the full question pool; 2) Item Response Theory (IRT) A psychometric framework (6) that models the relationship between an examinee's s latent ability 6 and their observed responses (correct/incorreet) IRT serves the "user model" for estimating ability based on response data to the selected questions.
"Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

From a machine learning perspective, the overall adaptive testing process can be formulated as subset selection problem that seeks to minimize the error of ability estimation 77 [] Given large question pool V. selecting a question subset S S V for an examinee to answer such that the ability estimate fg (inferred from responses to S) is as close as possible to the true (or optimal) ability 6*:
1|0g - 0 I|, SCV
min
Os = arg min Σ [4()), s.t.
(I)
BEB iES
where (0) denotes the cross-entropy loss associated with the response to question i, and 0 represents the ability parameter modeled by IRT. Obviously, adaptive testing is a complex nested optimization w.r.t. the subset variable S, requiring iterative updates: the outer loop selects the optimal subset S (often represented parameter via supervised learning.
sparse selection vector (8), while the inner loop estimates the ability
Given its complexity, recent works often rely on data-driven meta-learning [9. or reinforcement learning [TO, 2 to derive the question selection policy. However, these approaches introduce significant computational overhead and may amplify biases present in the data [9]. Even latest heuristic algorithms 7 [1] still require approximating and matching gradients across the entire ability parameter space θ, leading prohibitively high complexity. These limitations are especially critical in real-world online assessments, e.g., the Duolingo English Test, GRE Online, and remote certifications, which involve massive item pools, diverse examinees, and complex user behavior. These settings demand adaptive testing systems that are interpretable, robust, and efficient enough for real-time operations 22 [3].
To address these, this paper proposes a fundamental shift the optimization paradigm of adaptive testing. For the first time, we derive closed-form solution for the unknown subset variable S, referred to as CFAT (Closed-Form expression for Adaptive Testing). It allows us to directly solve for the optimal subset without iterative sampling or complex nested optimization. Specifically, we successfully quantify the ability estimation error and demonstrate that it can be interpreted as minimizing the gradient bias while maintaining a stable Hessian structure. Furthermore, we prove that the objective function exhibits approximate submodularity, enabling the use of a simple greedy algorithm to efficiently select the subset.
Beyond improving question selection, such closed-form formulation allows us to quantify the impact of human behavioral perturbations (e.g.. guessing and slipping) on ability estimation. CFAT ultimately enables a bias correction mechanism for more reliable assessments. By fundamentally shifting the optimization paradigm of adaptive testing, CFAT uses statistical learning principles for efficient, direct computation. Experiments on three educational datasets demonstrate that our method reduces the number of required test questions by 10% compared to the best baseline, under the same estimation accuracy. Moreover, CFAT achieves at least a 12x improvement in selection efficiency (computation time) over latest methods. It can also exhibit higher robustness in high-noise scenarios, accurately recovering ability estimates and improving prediction reliability.
Background and Related Works
Adaptive testing has been widely adopted in human ability assessment especially in education, and has gradually been incorporated into high-stakes examinations. To achieve both accuracy and efficiency, adaptive testing typically consists of two key components: IRT and question selection algorithms:
(1) Item Response Theory (IRT). IRT serves as user model that captures the relationship between an examinee's ability and their responses 4 Widely used in various large-scale assessments such as OECD/PISA, a common example is the two-parameter logistic (2PL) model, which defines the probability of correct response to question i as: p(correct) (a, (0 B,)). where a, and 3, represent the discrimination and difficulty parameters, respectively. These question parameters are pre-calibrated [14, while the examinee's 's ability 0 is estimated during testing. IRT models are interpretable: higher ability implies higher probability of success on items of fixed difficulty. Extensions include multidimensional IRT [15] and neural cognitive diagnosis models [16] 18). which capture more complex interactions. All these methods rely on maximum likelihood estimation (minimizing cross-entropy loss) estimate ability parameters from observed response data.
(2) Selection Algorithm. This is the core of achieving efficient assessment, as determines a valuable subset for estimating examinee ability in IRT. Traditional algorithms rely on statistical heuristics

---

based on information measures, such as Fisher information [14). KL information [99] and various improved information metrics 220 21 22 to guide selection. Alternatively. active learning methods
select informative questions based on question diversity and uncertainty [23] Recently, to directly
solve the nested optimizations, researchers have increasingly adopted data-driven approaches, e.g.. reinforcement learning and meta-learning, to optimize subset selection [T0 9 2 8]. These methods
iteratively train a policy (often represented as a neural network) from large-scale response data.
In this work, we aim to bypass the nested optimization by deriving a closed-form expression for the estimation error w.r.t. the selected subset. allows us determine the optimal subset directly. approaches, this statistical method eliminates the
Compared to data-drivve/neural network- need for extensive training. Compared the latest gradient- pradient-based heuristic algorithms [7 [T] incorporates second gradient (Hessian matrix) information, meanwhile, mitigating the impact of guessing and mistakes on ability estimation. Furthermore, CFAT is up to 12x more efficient than these SOTA heuristic methods, making it highly practical for real-time human assessments.
3 Method
Adaptive testing estimates ability efficiently by selecting small, informative subset S C V from larger question pool V. It can reduce test length while maintaining accuracy.
Problem Statement. Formally, an examinee responds to the selected subset S, producing {(g1 y1).... (9 B]s])}, where S { S V is the question set selected by the adaptive selection algorithm, and yı E {0, 13 denotes the response label, with I representing correct response and 0 otherwise. The examinee's 's ability is then estimated by minimizing cross-entropy loss l over S.
0.- arg min
Σ Ee((0= = arg min B
Σ
log (o 9k),
(2)
where pe pg(q..30 34) represents the probability of observing response (q,, y4) from an examinee with ability 0. The precise form of py depends the IRT model. Assuming an examinee's 's true latent ability is denoted as 0", one can theoretically approximate by minimizing the expected cross-entropy loss over the entire question pool: 0* arg ming Eiev (0) 77. The objective of adaptive testing is to ensure that the estimated ability 0g from the subset is as close as possible to 8 (Figure D:
Definition 1 (Definition of Adaptive Testing). Given a fixed test length T. the task is to select an optimal subset S S V such that the ability estimate 0s approximates the estimate 6", The adaptive lesting task can be formulated to nested optimization problem follows::
min 10ss - 0*1,. S.I. Ag = arg min Σ (0). S-=T
ES
In the outer loop, the subset S can be generated using selection policy 10 2 it can be treated as trainable indicator or sparse selection vector that determines question selection [8]. In the inner loop, base optimization algorithm estimates Og using the responses on the selected S, following standard supervised learning principles.
While reinforcement/meeta leaaniinn methods have shown promise in adaptive testing M. they often computationally intensive due to multi-step gradient descent and repeated backpropagation. This raises: Can directly formulate |[&s 8rl and optimize S without resorting to iterative meta-optimization? If the effect of question selection on ability estimation can be explicitly modeled, more efficient selection strategies may be possible
3.1 Avoid the Nested Optimization Trap
The key challenge in reformulating is to establish direct link between As - 0- and S, without relying on an inner-loop optimization (arg min). To achieve this, we can simplify the problem by framing as an issue of parameter estimation under data reduction: Consider the pool V as the full dataset for estimation, while S is its selected subset. The problem then becomes: analyzing how removing a subset Z (where S V \ Z) affects the estimated ability.
3

---

airg min Σ (1())
(Informative)
(Representative)
IES Estimated Ability
Subset
10g 001l
(5,0"
Σ Ve,(o")
0" arg min E4(0) <EV True Ability
Estimation Error
Closed-Form Expression Estimation Error
Question Pool V
Figure 1: Illustration of subset selection in adaptive testing. The full question pool V is divided into a selected subset and a remainder Z. The estimation error is approximated via first-order (gradient) and second-order (Hessian) terms, capturing S's representativeness and informativeness, respectively.
Measuring the Impact of Question Reduction on the Ability Estimator. Obviously, the most direct approach would be to recompute/retrain the parameter estimate from scratch for each choice of SS, done in the inner loop's ss minimization in Eq.2 But that is computationally prohibitive. Thus, instead of outright removing questions from V. we down-weight their influence in the ability estimation process. This leads to the definition of perturbed estimator:
0g = arg min
Σ 400) 7 Σ L())), RVV
(4)
V
E
where Z is the set of dlonn-weighte (or "removed") questions, and y E [0, 1/[1I] This formulation to the total loss, thereby approximating the effect of
reduces the contribution of response excluding them from estimation. For a first-order approximation, we expand the gradient of the loss function evaluated at 03 using a Taylor expansion of Eq. 4) around 0*, Since 03 a minimizer, its gradient is approximately zero:
VL,00) - Σ V4(0)) +
1 EEV
(e( - Σ IV(00)) (03 (03- iEZ
- 0*). (5)
REV
IE
In particular, when we set Z = VV S and choose y = 1/ , the perturbed estimator exactly recovers
the ability estimate based on the subset S, i.e. 03 0g. Since 6* satisfies the optimality condition Eiev (0") = O, we obtain:
03 8" = 0g 0" = H (S Σ Ve.(0'.
(6)
RES
where H(S,0*) Eies (0() denotes the Hessian of the loss function for ability estimation. and H-1 denotes its inverse. Here, H(S, (*) is invertible, which holds under standard regularity conditions in IRT [8, [.]. For complex neural network-based models, where computing the exact Hessian is often intractable, we adopt quasi-Newton approximation (details are provided Ap- pendix A). This result can be viewed as extension of influence function theory [24 25 26), which originated statisties the 1970s. characterizes how perturbations in the data affect an estimator. Here, we approximate the effect of selecting subset S without resorting to explicit re-optimization.
Closed-Fore Expression Estimation Error. The key takeaway is that the error ability estimation based on the selected subset S admits closed-form expression (Figure []:
Lemma 1. Let the true ability parameter be 0*, When using IRT for ability estimation, the estimation error based on any subset S S V can be directly computed as in O This allows for directly optimizing the selection of S to minimize the estimation error without the need to recompute 0g:
lirst-order
min ||0g 80 min
H (S,05
Σ VE(
second-order
This reformulated objective directly quantifies the influence of the selected subset S on the estimation error, bypassing the need for re-optimization of 0s in previous nested optimizations. The selection of balances two critical factors: simultaneously managing both bias minimization (first-order stability) and conditioning of the Hessian (second-order stability):

---

Factor 1: First-Order Gradient Alignment. The term Eigs V4,(d)) captures the aggregate first- order (gradient) contribution of the selected questions. If this sum deviates significantly from zero, it introduces directional bias into the estimated parameter Os. Intuitively, the goal is to find subset whose gradients "agree" with those of the full question pool. This ensures that the subset representative of the entire pool in terms of gradient information, and does not skew the estimation.
Factor Second-Order Information Control. The Hessian inverse, H H (S,0"). controls how the subset's curvature information influences estimation stability. The optimal subset must ensure that the Hessian remains well-conditioned while retaining crucial second-order information stabilize parameter updates. Consider the case of IRT: the expected Hessian can be approximated by the Fisher information I, i.e., E E[H(SS, 8)] -
Eies (0) -ies of po 0) po(g, 1). This suggests that it tries to find informative questions with high discrimination (a) and maximum response
uncertainty, e.g., p(g, 1) R 0.5
Thus, the best subset is both diverse and cerrmmavvvemmmmmmiii gradient bias while maintaining a stable Hessian stading to efficient and reliable estimation.
3.2 Approximate Optimization for Subset Selection
Based the above reformulated objective, we aim to select subset S that minimizes the set function: min f(S) = min (S, 8*) Eies (e,( This problem is combinatorial and generally NP- hard. Exhaustively searching for the optimal subset is computationally infeasible for large pool due to the exponential number of possible combinations.
Fortunately, we observe that this objective function exhibits a diminishing marginal gain property, which aligns with the concept of submodularity (29]. Submodularity a useful concept combinato- rial optimization problems that plays a crucial role in designing efficient approximation [30 31 32)
e.g., greedy algorithm. More precisely, the objective function f(S) is approximately submodular, a property referred to e-submodularity. This property implies that the incremental benefit of adding an element x decreases as the set grows:
Theorem 1 (e-Submodularity of the Set Function). Estimating the ability 0 using IRT, the loss function {(O) is p-stroongy convex. Assume that the gradient norm and Hessian's spec- tral norm are bounded: [V)], I G and [V < H. The objective f(S) = HH (S, Eies (l](0*)] is e-submodular, and 2G(++H) 2BG i.e., for any subsets 1lAA T
BCBCV:
[2G(u= + H)
2HG
((A f(A) ((BU f(B)
12|1A1
The proofs can be found in Appendix B The e 2G(++H1) 2HG
[²[A| 4 AA bound decreases as |A]] increases.
This means the function becomes more submodular as the subset grows, which is intuitive the marginal benefit of adding new question becomes more stable as more questions are already selected. This bound provides theoretical justification for using greedy methods: if e is small (e.g., due to large [A)), greedy selection will be near-optimal even though the function is not strictly submodular.
Greedy Question Selection. Given that the objective fSS is e-submodular, we can use a greedy al- gorithm iteratively construct an optimal subset S. The approximate submodularity property ensures that greedy selection achieves near-optimal solution with theoretically bounded suboptimality.
For size-constrained minimization of f(S), simple reverse greedy algorithm be adopted. It sequentially selects elements that yield the smallest marginal increase in f(S). Specifically, it starts with an empty subset So 0. each step the question that minimizes the marginal gain is selected, formally given by q. arg mingev (V, (f(S,-1 U v))) - f(S(-1)). After selecting qt. the subset is updated as: S, = S U {( 3r)
In practice, the parameter 0* is unknown and we use the estimate 0' obtained from S.. The objective function can be approximated: f(S | 8*) = - (S,0) Eies V4,(')) Meanwhile, the true
Frrm the perspective of Active Learning, samples near the decision boundary- where the model is most uncertain- are typically the most informative [27 28
5

---

labels are also unobserved, we take the expectation over y, selecting the next question:
q: = arg min By L S
0 11-1)]
(9)
The sequential selection process continues until the selected questions reach predefined maximum size T. corresponding to the termination condition of the test. Based on the asymptotic unbiasedness of MLE, we find an upper bound on the approximation error when substituting 0' for 0*
Lemma 2 (Approximation Error). When using IRT for ability estimation, the function f(S) is Lipschitz continuous w.r.t. 6. With probability least
the approximation error incurred by
using& l satisfies the upper bound: [f( 0*) - f(S) -
Pr + pi
MG C(8) where μ11, p2, M, H, v/s.l'
G, and C' are model-dependent constants characterizing the properties of the objective function..
The proofs can be found in Appendix C The substitution of 0" with 0e is justified due to the consistency and asymptotic normality of estimators. According this bounded approximation error in Lemma 2] the error introduced by estimating 0* diminishes rate of ((S,,11122, ensuring the robustness of the adaptive selection process.
3.3 Bias Correction in Ability Estimation: Guessing and Slipping
The idealized ability estimation above assumes that an examinee's 's responses accurately reflect their true ability. However, in practical testing, the observed response y may not correspond perfectly to true ability due to guessing and slipping [. 1) Guessing: An examinee correctly answers question they should not have been able to solve purely by chance. For example, if multiple-choice question has three options, random guessing yields 33.3% success probability; 2) Slipping: An examinee fails to answer a question correctly despite having the ability to do so. It arises due to carelessness, misreading, other lapses.
Both factors induce label flipping in the observed responses y E {0, 1}. leading to biased ability estimates 0g that contain unpredictable noise. This distortion can be explicitly quantified within this CFAT framework: Consider a response (gm, Vm) affected by label flipping, resulting in the
incorrect label (gm, 1 - ). The corresponding loss becomes: (0) = -(1 ym
Uon log po (Zm= (om; 0). After incorporating the flipped response, the new ability estimate on is: (5mm) =
arg ming [ Eies (0) - (0) + (0). Note that this also applics a weighted adjustment rather than physically replacing the affected data. When 7 = | the correction becomes equivalent full replacement of the original response.
Applying a Taylor expansion around 0g, similar to the derivations in Section 3.1 we approximate:
pe p 9m 1)
0 (s(m) = 0s + [H((S) \ 9m. 0s) + Hl(qm. 0s)] 0s)]-1(1 2ym) V log
(10)
po po(gm, (m., 0)*
where Hlgm. 0g)= = V The term A0s(m) As(m) 0y provides quantitative measure of how a flipped response skews the estimate. Even if we cannot pinpoint the specific flipped samples, analyzing the expected effect enables us understand the direction and magnitude of the systematic bias caused by response errors. See Appendix D for a detailed derivation of Eq.(TQ)
Thus, instead of relying on the potentially biased estimator 0s. introduce bias-corrected ability estimate by subtracting the expected distortion: fg |[ = 0g Emes [Ty ym)
TaHm A0s(m), where Ty is the guessing probability, capturing the likelihood of obtaining a correct response by chance, and म. is the slipping probability, representing the likelihood of incorrect responses despite having the requisite ability. The complete CFAT framework is shown in Algorithm
4 Experiments
Evaluation Tasks. To assess the efficiency of question selection algorithms in adaptive testing, we evaluate the accuracy of ability estimation under fixed test length. Specifically, we compare the final estimated ability 0s. where S represents the selected question subset chosen by different selection algorithms. The evaluation is conducted across two primary tasks 9 7 1) Student Performance

---

[34] originates Algorithm 1: The proposed framework CFAT
Require: V Question pool, Po Parameterized probability model (IR' neural network), "g
Guessing probability, 7s Slipping probability
Initialize: Initialize the ability estimate 6 and responses data So < 0.
for t == 1 to ' do
Select the next question ge by minimizing the function:
q: = arg mingev's V\,,, E L {q:
(--1).
Obtain the examinee's response label yr: S; S u [(....).
Update examinee's is ability estimate: 6r < arg mingge Lies. (()) Apply bias correction adjust for response errors:
g 6³ Emes, [ ym) + nsynm (0s
Output: The examinee's ability estimate 03 = HT using the responses on the selected S.
Using the estimated 0s. predicting students' responses (correct/ncorrect) on a held-out test set and measure predictive performance using Accuracy and AUC: 2) Ability Estimation Error: In simulation setting, the ground-truth ability 6* constructed and simulate students' response behavior during testing. We then compute the estimation error using the Mean Squared Error (MSE) E||0s- 0-|
Prediction:
Experimental Implementation Details. We set the maximum test length to |S| = = 20, consistent with typical adaptive tests. All methods implemented in Py PyTorch and trained Tesla V100 GPU. Hyperparameters are tuned via grid search, with batch size 64, learning rate 0.001, and behavioral noise parameters kg- - 0.002, nT 0.001. Optimization is performed using Adam.
we split examinees into 70% training, 20% validation, and 10% testing. The training set is used estimate question parameters and train some data-driven models. During validation and testing, we simulate adaptive testing: Specifically, for the student performance prediction task, each examinee's 's responses are divided into candidate set Vi (for question selection and ability estimation) and a meta set M; (for evaluation via Accuracy/AUC). At each step, a question is selected from V,, ability is updated, and performance is evaluated on M,. For ability estimation error, ground-truth abilities 8* are estimated from full responses, allowing simulated examinees to answer any question in V for direct error computation.
Following
Datasets. We conduct experiments on three widely used educational testing benchmark datasets: ASSIST, NIPS-EDU, and EXAM: ASSIST 333 derived from the online educational platform ASSISTments and contains examinees' practice logs on mathematics; NeurlPS-EDD from the NeurlPS 2020 Education Challenge, comprising a large- scale dataset collected from exami- nees' responses to questions on Fedi, an educational platform. EXAM is dataset from iFLYTEK Co., Ltd. that records junior high school students' performances on mathematical exams. The implementation code is available on: https: /tte:://gttttttu cww 542////4yCCFFTTT
Compared Approaches. For ability estimation, we consider both classical IRT model and neural network-based approaches: NeuralCDM (35). flexible framework that generalizes various IRT and cognitive diagnosis models, e.g., MIRT 36 and Matrix Factorization 37 38). The objective of our
experiments compare the proposed selection algorithm against existing selection methods terms of their impact on ability estimation. Thus, we evaluate the following SOTA algorithms as baselines: Random Selection serves as benchmark by selecting questions uniformly at random, providing reference for assessing the improvements achieved by other algorithms; Fisher Information [14] and KL Information [19 are classical methods that prioritize questions based on their informativeness: MAAT (33 uses active learning to balance uncertainty and diversity. BOBCAT 9 and UATS [ apply meta-learning to solve the nested selection problem via cross-entropy minimization. NCAT 10 and GMOCAT 2 frame selection as reinforcement learning, leveraging transformers and GNNs train a data-driven selection policy. BECAT [ uses a greedy heuristic based on first-order gradient approximation, between the selected subset and the entire question pool.
7

---

Table 1: The performances on Student Performance Prediction. It reports ACC and AUC at 5th, 10th, and 20th step (subset size). Panel I presents results based on the IRT model for ability estimation, while Panel 2 uses neural network- based methods (e.g., Fisher) are not applicable to deep models. Bold values indicate statistically significant improvements (p-value 0.01) over the best baseline.
model (NeuralCDM), Note that information/umeertaintyy
ASSIST (ACC/AUC)
CUPSEDDUAACC/AII
EXAM (ACC/AUC)
Method 65
@10
000
@55
@10
8:20
e5
@10
@20
Random
70.89/20.78
71,99/7184
73.02/72.45
66,57/69.02
68. (8.11/71 42
70.00/73.90
77:58/70.34
77.22/71.8
80.33/74.09
Fisher
71.87/71.22
72.63/2.330
73.11/73.56
67.70/70.62
70.59/73.51
71.23/76.33
77.35/70.51
79.75/72.25
83.03/75.90
KI.
71.95/71.31
72.68/2..50
73.13/73.57
67.09/69.71
19.29/73.30
70.41/75.73
77.37/70.58
19.22/72.11
83.01/75.73
MAAT
72.11/71.24
72.03/2.338
73.20/73.05
66.44/69.31
69.10/71.12
69.27/73.40
75.27/70.32
77.99/72.12
80.12/73.67
BOBCAT
72.33/71.72
72.56/72.18
73.78/73.31
69.55/74.41
70.99/75.66
71.71/76.44
80.61/88.2 29
83.81/72.02
81,44/72.82
NCAT
72.22/71.66
72.52/72.38
73.83/73.51
67.30/72.11
70.68/75.80
71.1176.66
80,92/70.72
83.96/72.67
83,88/74.19
LATS
72.29/72.82
72,04/2..74
74.14/74.8
67,58/73.33
70.50/74.82
7184476.57
79.17/70.22
82.33/73.29 29
84.91/75,24
RECAT
71,92/1.34
73.01/72.73
73.9673363
66.9773.15
71.61/75.85
72.00/76.22
80.93/70.74
83.80/72.88
84,20/75.03
CFAT
7286/73.48
73.37/73.26
74.29/5.22
69.62774.55
72.25/7622
7388778.03
81.11/7103
84.13/73.80
86.05/77.83
ASSIST (ACC/UU)
NIPS-EDU (ACC/AU))
EXAM(ACC/AUC)
Method
695
@:10
69:0
68.5
@0
6:20
6R5
@110
@220
Random
71,21/91.02
72.53/72.08
72.51/72.83
67.13/89.39
68.42/7151
70.39/74.93
79,80/72.48
78.33/74.52
79.31/78.22 22
MAAT
72.09/70.74
72.31/72.03
71.75/72.29
67.83/70.00
70.42/72.58
70.63/75.85
82.87/70.22
82.55/74.29
83.72/79.88
ROBCAT
72.64/1144 46
12.71/72.78
73.80/2282
71.02/16.1
72.46/77 82
73.42/79.06
78.13/78.28
78.13/8144
78.04/79.53
NCAT
72.29/71.64
72.62/72.34
73.92/73.56
70.43/7..12
72.84/77.92
73.44/79.99
82.33/78.54
83.13/81.6
81.44/99,35
UATS
73.02/22.32
72.92/73.05
73.16/72.73
71.87/75.13
73.13/78.11 12
74.14/79.70
81 81.26/77.12
82.46/80.92
83/79/8082
BECAT
72.30/71.61
73.11/72.87
74.13/73.70
71.33/7..31
73.07/78.24
73.58/79.26
82.87/78
83,22/81.49
84.77/79.70
CEAT
74.13/2292
73,45773.88
74.53/74.38
71,20/76.19
74.43/ 78.38
74.72/81.77
83,33/80.88
84.12/8287
85.12/81.66
4.1
Experimental Results
We evaluate the proposed CFAT framework two core tasks (i.e., student performance prediction and ability estimation error) across three benchmark datasets.
Task 1: Student Performance Prediction:
This task assesses the efficiency of ability estima- tion in adaptive testing. Specifically, we compare the prediction accuracy of response labels (cor- rect//ncorrect) under different question selection strategies, where each algorithm selects the same number of questions. As shown in Table [ we report the AUC and ACC scores for each method at the 5th, 10th, and 20th steps. The proposed CFAT consistently achieves the highest prediction accuracy under limited question settings. Notably, simple greedy algorithm CFAT outperforms neural network methods, e.g., reinforcement learning (NCAT) and meta-learning approaches (BOBCAT, UATS), by an average margin of 2% in AUC.
These results support our central claim: formulating the subset selection problem with closed-fom objective yields better performance than complex nested paradigm. Furthermore, CFAT outperforms the gradient-based BECAT. It highlights the advantage of incorporating second-order information (i.e., the Hessian matrix) over relying solely on first-order gradients. This superiority is observed both theoretically and empirically at scale. Although our theoretical derivation is grounded in the classical IRT model, the CFAT framework also demonstrates strong performance when applied to neural network ability estimation models (NeuralCDM), This suggests that our subset selection formulation and its approximation are generalizable and extensible across different modeling paradigms.
Task 2: Ability Estimation Error:
To evaluate the accuracy of ability estimation, we adopt widely used simulation protocol in adaptive testing. Specifically, we treat the ability estimate derived from an examinee's 's full response data, denoted as 0", as the ground truth. During the testing process, this ground-truth ability allows us to simulate response labels for any question, while the tested algorithms only have access to observed response data and not the true ability. Figure 2 illlsstraess the estimation error ||0* - 8*1 || over the testing process, where 8e denotes the estimated ability at step f. The results show that our proposed CFAT achieves comparable estimation accuracy using only 30%-45% of the questions required by random selection. Compared to recent SOTA methods (c.g.. UATS), CFAT reduces the number of required questions by at least 15%.
Although CFAT exhibits a relatively slower start in the early stages of testing, its estimation error decreases rapidly as more questions are selected. This initial lag is somewhat less favorable compared to other data-driven methods (e.g., meta learning) that are good at mitigating the cold-start problem 39 These empirical observations align well with the analysis in Theorem [] which demonstrates that the submodular nature guarantees the near-optimality of the greedy question selection algorithm as the selected subset grows.
8

---

ASSIST
NIPS EDU
EXAM
Average per Selection
1.4
Randoon
1/4
1.6
1.4
NCAT
19.58
1.2-
1.0
LITTS
1.3
MAAT
148.23 (cu))
NCAT
0.8
0.4
DEAT
0.8
0.8
0.4
0.8
0.6
Fisher
0.23
4.36
0.2
CFAT
0.35
0.0
5
10
15
10
15
20
10
15
20
Test Step
Test Step
Test Step
Time per item)
Figure 2: (a) Simulation results for ability estimation: MSE of ability estimation, E| 0ol 2
under different subset sizes (steps) for five representative question selection algorithms. Results are averaged over 10 repetitions, with error bars indicating the standard deviation. (b) Average time required to select a single question for each method.
NGAT
BECAT
MAAT
Decriminttoo
-3
-5
-0
-3
Difieuty (P)
Diffo.uly (F)
Difficuly
Difficity
Questioons
Selected Questions
Examinee Abilly (0)
Figure 3: Characteristics of selected questions across different methods. We randomly sample 10 examinees and compare the question subsets selected by CFAT with several SOTA baselines. The distributions of question difficulty and discrimination parameters are visualized.
Analysis of Computational Efficiency and Subset Characteristics. We compare the computa- tional efficiency of different question selection algorithms to assess their practical applicability in large-scale testing (note: no acceleration techniques or engineering optimizations are applied). Specifically, in Figure b)), we report the average time required to select single question. CFAT demonstrates higher effciency compared SOTA methods e.g. MAAT and BECAT achhevin approximately 12x speedup over BECAT. Notably, CFAT matches the speed of the classical Fisher information method, while simultaneously delivering least 20% improvement in estimation accuracy, evidenced in Figure 2 a). Meanwhile, Figure iillsttrates the characteristics of question subsets selected by different methods, along with the true ability estimates 6* of 10 randomly sampled examinees. As shown, the questions selected by CFAT tend to have higher discrimination and are well-aligned with the examinees' ability levels (i.e., question difficulty closely matches ability). In contrast, other methods often prioritize diverse questions, many of which are "outliers" "eitterr"eeither too easy or too difficult for the examinees. Such lwwddiscrinination or mismatched questions tend to be less informative and may hinder accurate ability estimation [40]
Reliability under Guessing and Slipping Noise. In real-world scenarios, examinee's responses may be affected by guessing (label flipping 0 - 1) slip- ping (label flipping + [1 To model this, label noise is introduced in above simulation by flipping response labels with a certain probability. Table 2 illustrates the estimation error across different algo- rithms as the flipping probability increases. Previous approaches exhibit significant performance degrada- tion under noise. In contrast, CFAT consistently main- tains lower estimation error, outperforming its ablated version (CFAT w/o correction), which lacks the bias correction term. Notably, under high noise levels (e.g., 20% label flipping), CFAT still achieves stable and accurate ability estimates. These results empir- ically validate our theoretical analysis in Section 3.3 highlighting the effectiveness of incorporating cor- rection term when estimating the ability 0s.
Table MSE for different selection algo- rithms in ASSIST under varying levels of la- bel perturbation (Step=20). Perturbation applied to the examinee's 's response label. "No Pert.' denotes MSE without any label noise.
Method
10%
20%
0.3827
0.4936
0.6681
Random
0.3765
(+0.0062)
(+0.1171)
(+0,2316)
0.3599
(40.0089)
0.3638
(40.1145)
0.4744
(4022270)
0.5869
BECAT
0.3697
(s0.0044)
0.3741
(40.1117)
D-4814
(40.2308)
GMOCAT
0.2322
(40.0053)
0.2375
(40.0634)
0,2956
(40.1156)
0.3478
0.2024
(wlo conrection)
0.1962
(+0.0062)
(+0.1159)
(+0.2362)
CFAT
0.1738
(+0.0040)
0.1778
(+0.0344)
0,2082
(+0.1032)
0.2770

---

Conclusion
This paper addresses the subset selection problem in ability estimation: how to select a small question subset such that the estimated ability closely approximates the true ability. Instead of relying on the traditional nested optimization paradigm, we derive a closed-form objective that allows for direct optimization. It shows that simple greedy algorithm effectively solve this problem, and partially correct the bias of the ability estimator. Extensive experiments demonstrate that it is computationally efficient, yields more accurate ability estimates, better adapts to individuals, and remains robust under high-noise conditions.
Acknowledgements
This research was supported by grants from the National Key Research and Development Program of China (Grant No. 2024YFC3308200, 62477044), the National Natural Science Foundation of China (62525606). the Key Technologies R & D Program of Anhui Province (No. 202423k09020039) the National Education Science Planning Project (Grant No. ZSA240466), and the Fundamental Research Funds for the Central Universities.
References
[1] Qi Liu, Yan Zhuang, Haoyang Bi, Zhenya Huang, Weizhe Huang, Jiatong Li, Junhao Yu, Zirui Liu, Zirui Hu, Yuting Hong, et al. Survey of computerized adaptive testing: A machine learning perspective. arXiv preprint arXiv:2404.00712 2024.
[2] Hangyu Wang. Ting Long, Liang Yin, Weinan Zhang, Wei Xia, Qichen Hong, Dingyin Xia, Ruiming Tang, and Yong Yu. Gmocat: A graph-enhanced multi-objective method for comput- erized adaptive testing. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 2279-2289, 2023.
[3] Chris Piech, Jonathan Bassen, Jonathan Huang. Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha -ock-Dickst Deep knowledge tracing. Advances in neural information processing systems, 28, 2015.
[4] Wim J Van der Linden and Peter J Pashley. Item selection and ability estimation in adaptive testing. In Computerized adaptive testing: Theory and practice, pages 1-25. Springer, 2000.
[5] Xiaoshan Yu, Chuan Qin, Dazhong Shen, Haiping Ma, Le Zhang, Xingyi Zhang, Hengshu Zhu, and Hui Xiong. Rdgt: enhancing group cognitive diagnosis with colattion-gudded dual-side graph transformer. IEEE Transactions on Knowledge and Data Engineering, 2024.
[6] Hua-Hua Chang. Psychometrics behind computerized adaptive testing. Psychometrika, 80(1):1- 20, 2015.
[7] Yan Zhuang, Qi Liu, GuanHao Zhao, Zhenya Huang, Weizhe Huang, Zachary Pardos, Enhong Chen, Jinze Wu, and Xin Li. A bounded ability estimation for computerized adaptive testing. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[8] Junhao Yu, Yan Zhuang, Zhenya Huang, Qi Liu, Xin Li, LI Rui, and Enhong Chen. A unified adaptive testing system enabled by hierarchical structure search. In Forty-first International Conference Machine Learning, 2024.
[9] Aritra Ghosh and Andrew Lan. Bobcat: Bilevel optinization-based computerized adaptive testing. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelli- gence, IJCAI-21, pages 2410-2417, International Joint Conferences on Artificial Intelligence Organization, 2021.
[10] Yan Zhuang, Qi Liu, Zhenya Huang, Zhi Li, Shuanghong Shen, and Haiping Ma. Fully adaptive framework: Neural computerized adaptive testing for online education. Proceedings of the AAAI Conference on Artificial Intelligence, 36(4):4734-477 Jun. 2022.
10

---

[11] Zirui Liu, Yan Zhuang, Qi Liu, Jiatong Li, Yuren Zhang, Zhenya Huang, Jinze Wu, and Shijin Wang. Computerized adaptive testing via collaborative ranking. Advances in Neural Information Processing Systems, 57:95488-9555 2024.
[12] Micheline Dhalloub-Deville and Craig Deville. Computer adaptive testing in second language contexts. Annual Review of Applied Linguistics, 19:273-290 299, 1999.
[13] Yan Zhuang, Qi Liu, Zhenya Huang, Zhi Li, Binbin Jin, Haoyang Bi, Enhong Chen, and Shijin Wang. A robust computerized adaptive testing approach in educational question retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 416-426, 2022.
[14] Frederic M Lord. Applications of item response theory to practical testing problems. Routledge, 2012.
[15] Giles Hooker, Matthew Finkelman, and Armin Schwartzman. Paradoxical results in multidi- mensional item response theory. Psychometrika, 74(3):419-444 2009.
[16] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. Neural cognitive diagnosis for intelligent education systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6153-6161, 2020.
[17] Weibo Gao, Qi Liu, Linan Yue, Fangzhou Yao. Hao Wang, Yin Gu, et al. Collaborative cognitive diagnosis with disentangled representation learning for learner modeling. Advances in Neural Information Processing Systems, 37:562-588, 2024.
[18] Wei Song, Qi Liu, Qingyang Mao, Yiyan Wang. Weibo Gao, Zhenya Huang, Shijin Wang, Enhong Chen, et al. Towards accurate and fair cognitive diagnosis via monotonic data augmen- tation. Advances in Neural Information Processing Systems, 37:47767-47789) 2024.
[19] Hua-Hua Chang and Zhiliang Ying. A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20(3):213-22 229, 1996.
[20] Lawrence M Rudner. An examination of decision-theory adaptive testing procedures. In annual meeting of the American Educational Research Association, 2002.
[21] Wim van der Linden. Bayesian item selection criteria for adaptive testing. Psychometrika, 63(2):201-216, 1998.
[22] Wim JJ Veerkamp and Martijn PF Berger. Some new item selection criteria for adaptive testing. Journal of Educational and Behavioral Statistics, 22( 1997.
[23] Haoyang Bi, Haiping Ma, Zhenya Huang, Yu Yin, Qi Liu, Enhong Chen, Yu Su, and Shijin Wang. Quality meets diversity: A model-agnostic framework for computerized adaptive testing. In 2020 IEEE International Conference on Data Mining (ICDM), pages 42-51. IEEE, 2020.
[24] Frank R. Hampel. The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69(346):383-39) 1974.
[25] L. Jaeckel. The infinitesimal jackknife. Unpublished memorandum, Bell Telephone Labora- tories, Murray Hill, NJ, 1972.
[26] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1885-1894. PMLR, 06-11 Aug 2017.
[27] Zhilei Wang, Pranjal Awasthi, Christoph Dann, Ayush Sekhari, and Claudio Gentile. Neural active learning with performance guarantees. Advances in Neural Information Processing Systems,
2021.
[28] Alex Tamkin, Dat Nguyen, Salil Deshpande, Jesse Mu, and Noah Goodman. Active learning helps pretrained models learn the intended task. Advances Neural Information Processing Systems, 35:28140-22 40-21813,, 2022.
11

---

[29] Abhimanyu Das and David Kempe. Approximate submodularity and its applications: Subset selection, sparse approximation and dictionary selection. Journal of Machine Learning Research, 19(3):1-34, 2018
[30] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficctn training of machine learning models. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 1960- PMLR, 13-18 Jul 2020.
[31] Artin Tajdini, Lalit Jain, and Kevin Jamieson. Nearly minimax optimal submodular maxi- mization with bandit feedback. Advances in Neural Information Processing Systems, 37:96254- 96281, 2024.
[32] Abir De and Soumen Chakrabarti. Neural estimation of submodular functions with applica- tions to differentiable subset selection. Advances in Neural Information Processing Systems, 35:19537-19552, 2022.
[33] Zachary A Pardos, Ryan SJD Baker, Maria OCZ San Pedro, Sujith M Gowda, and Supreeth M Gowda. Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In Proceedings of the third international conference on learning analytics and knowledge, pages 117-124, 2013.
[34] Zichao Wang. Angus Lamb, Evgeny Saveliev, Pashmina Cameron, Yordan Zaykov, José Miguel Hernández-LLbato, Richard E Turner, Richard G Baraniuk, Craig Barton, Simon Peyton Jones, Simon Woodhead, and Cheng Zhang. Diagnostic questions: The neurips 2020 education challenge. arXiv preprint arXiv:2007. 2020.
[35] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yu Yin, Shijin Wang, and Yu Su. Neuraled: general framework for cognitive diagnosis. IEFE Transactions Knowledge and Data Engineering, 35(8):8312-8327, 2022.
[36] Mark D. Reckase. Multidimensional Item Response Theory Models, pages 79-112. Springer New York, New York, NY, 2009.
[37] Andreas Tscher. Collaborative filtering applied to educational data mining. Journal of Machine Learning Research, 2010.
[38] Michel Desmarais. Mapping question items to skills with nonnegative matrix factorization. Sigkdd Explorations, 13:30-36, 05 2012.
[39] Manasi Vartak, Arvind Thiagarajan, Conrado Miranda, Jeshua Bratman, and Hugo Larochelle. A meta-learning perspective on cold-start recommendations for items. Advances in neural information processing systems, 30, 2017.
[40] Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P Lalor, Robin Jia, and Jordan Boyd-Graber. Evaluation examples are not equally informative: How should that change nlp leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1486-4503, 2021.
[41] Sujith Gowda, Jonathan P Rowe, Ryan Shaun Joazeiro de Baker, Min Chi, and Kenneth R Koedinger. Improving models of slipping. guessing, and someettby-momentt learning with estimates of skill difficulty. EDM, 201 1:99--00, 2011.
[42] John Dennis, Jr and Jorge Moré. Quasi-newton methods, motivation and theory. SIAM review, 19(1):46 89, 1977.
[43] J Shermen and WJ Morrison. Adjustment of an inverse matrix corresponding to changes in the elements of given column or given row of the original matrix. Annual Mathmatical Statistics, 20:621-625, 1949.
[44] Ronald Aylmer Fisher. Theory of statistical estimation. In Mathematical proceedings of the Cambridge philosophical society, volume 22, pages 700-725. Cambridge University Press, 1925.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions of the paper, including the formulation of a closed-foe form objective for subset selection in ability estimation and the development of an efficient greedy algorithm. These claims are supported by both theoretical analysis and extensive empirical results, as detailed in Sections 3 and 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: As discussed in Section 3.1 the proposed theoretical results are derived under the assumptions of IRT models, and may not directly hold for complex neural networks. However, we address this limitation by employing quasi-Newton approximations (Appendix A). Additionally, as noted in Section 4.1, while the computational cost of our method is not best, it achieves the best overall accuracy among the compared approaches.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged create separate "Limmittions"" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
13

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
See Appendix and C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
See Section It provides all the details.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results.
14

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have uploaded the code to the anonymous link https: com/ 54zy/CFAT (See Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
public//guiess/ooeeattmm www.aimmiiiioooccccciii.. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips cttp:/scccpbbbccccuicceeee guides/ ToodeSSumissiiooooll
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
See Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
The main results (Figure a))) in Section 4.1 report the deviation over 10 repetitions.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Figure bb) reports the time cost of each method.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurri: wwtpp//cwc..cciccctiiiiiiiiooeee
Answer:
[Yes]
Justification:
This work adheres to the NeurlPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The item selection process in adaptive testing is inherently personalized, and societal impacts such as faireess constitute separate line of research within the field. Due the page limitation, we put into Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use the NIPS-EDU and ASSIST, which is publicly available under the CC BY 4.0, MIT License. We have properly cited the original source in the paper and included the version and URL where applicable.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thooe cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce a new EXAM as part of this work. At submission time, all links and files have been anonymized to preserve double-blind review.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips htct:::://eerrpss.cc//ooeeeeeeee /c/////neeeeeecc///22////1. for what should or should not be described.
19

---

A Complete Algorithmic Procedure of CFAT
This section presents the complete optimization process of CFAT in practical adaptive testing. Algo- rithm provides a detailed illustration of the gradient-based ability estimation procedure. However, during the actual question selection phase of CFAT, computing the inverse of the Hessian matrix is required. While this is tractable for traditional IRT models, becomes computationally infeasible for neural network-based models due the high dimensionality and complexity of their parameter spaces. To address this, Algorithm introduces an efficient approximation of the Hessian inverse using quasi-Newton method 422 This enables the practical deployment of CFAT in neural network settings, and forms the basis of our complete CFAT algorithm tailored for deep learning adaptive testing systems.
Algorithm 2: Full Procedure of CFAT
Require: V Question pool, Pe Parameterized probability model (IRT or neural network), "y Guessing probability. Vs Slipping probability.
Initialize: Initialize the ability estimate 8 and responses data So <- Ø.
for t 1 to T do
Select the next question ae by minimizing the function:
q: arg m ( U {q.
0'-1).
Obtain the examinee' response label yı: St S U {)).
Initialize examinee's ability estimate og t 0F-
Update examinee's ability estimate:
for i = to K do
L Update 0f:- of + 0,11 (V
Apply bias correction to adjust for response errors: 5
8% + 0 - Emes, [T ym) + T.Hm A0s, (m)
Output: The examinee's 's ability estimate 0g 0% using the responses the selected S.
Algorithm 3: Full Procedure of CFAT (Approximate)
Require: V Question pool. Po Parameterized probability model (IRT or neural network), Ty Guessing probability. n. Slipping probability, a: learning rate.
Initialize: Initialize the ability estimate do and responses data Sg + 0, the approximation of the
inverse of Hessian matrix = (0) I and the examinee's ability estimate 0k-
for t = to do
Let 7- + H (*- and select the next question by minimizing the set function: 4r arg mingev.ss. (S, U {g, w 0²-1). Obtain the examinee's response label y:: S, + St-1 U ((t.9))
Initialize examinee's s ability estimate of +- 0%x1
Update examinee's ability estimate:
for & = 1 to K do
Calculate the search direction: da + - H (t)
Eies vei(0%-1).
Update 0% 0% + 8 + adІ.
Let uy + 0% - 0 and U) + Eies VL((O)) Eies ve,(
10
Update the approximation of the inverse of Hessian matrix H:
HT (1)
UkUn TH (1)
H- + - (8)
+
u Uk
2 N 0k
(0)
Apply bias correction adjust for response errors: 0% 0% Emes, [T, ym.) T. A0s, (m)
Output: The examinee's 's ability estimate 05 = 0% using the responses on the selected S.
20

---

B Proofs of Theorem
Theorem (e-Submodularity of the Set Function). Estimating the ability parameter 0 using IRT, the loss function E(O) is assumed to be j-stroongy convex fZ]. Assume that the gradient norm and Hessian's 's spectral norm are bounded by V[ (V)]( S G and (0)] S H. The subset selection
objective f(S) = H (S, 0*) Eses Ve;(0") is e-submodular, and E = MATA for any subsets A _ B V
20(x+H) + A i.e.
2HG
f(AU (z)) f(A) 2 ( (BB - f(B) - (2G(H + H)
(
2HG
12l [ + 12
(11)
Proof. Based on Lemma
T the objective function can be formulated as:
f(5) H (5,
Σ (E))
where H( = Eies Eies
(0) is Hessian matrix.
We assume the following boundedness conditions on the loss function (0) over the set V: 1) The gradient norm is upper-bounded: [|Vol:(0)| I G; 2) The spectral norm of the Hessian is also bounded: (()]l S H.
For IRT-based ability estimation, the loss function {(O) is known to be u-strongly convex 77. As a result, the Hessian matrix satisfies: H(x 0) = pLn. This implies that all eigenvalues of H((x, 0) satisfy Amin (z(z 0)) 2 p. Considering the inverse Hessian matrix H(z 0)- we have (1H(xx,) ()) x(H((,,)))' Thus, the largest eigenvalue of H(r, 0)- satisfies:
Amax (1(200)=²) Amiu (H(z, 0))
S
(12)
Since the spectral norm (2-norm) of a symmetric matrix is equal to its largest eigenvalue, we conclude:
H7(,, 01 1
(13)
Define: Hs H(S, 0*) the Hessian of the current subset S. 9s = cumulative gradient for the current subset. When we add a new element x to S. the function gain is given by:
- Eies (l, the
f (=)) f(S) = ||0ss + Vil,( -gs + Voll ||555 9s|1 (14)
To prove that the function is e-submodular, must show that for any subsets A C B V, the following inequality holds:
A A) 2 A(z, B) c, where >
(15)
For simplicity, we define AH- approximation the inverse matrix
V L (0") and Ag = (o') Now, applying the first-order
(Hs + AH)
H.s H AHHS 0(1AAH)))
(16)
Substituting this into Eq. 14 for A S):
A(x, || gs + H Ag H.s AHH. gs H. AHH Agll H5 9s
(17)
Using the triangle inequality, the gain associated with subset A satisfies:
A (x. A)
2 - H, g4 -, ". | A
|| gall
94 + 25 AHH 1| 9all
2 -
dl11 - 7 AHH A4 |
9A||
= || 9a
|| AHH Ag
(18)
21

---

Using norm bounds, we can further estimate:
G
HG
HG
(xx A)
(19)
HAA1
1A
1²|
Similarly, for subset B, obtain:
(xx B3 ~
H B 9B + H Ag - H AHH 9B
H
AHH Agوه - || 9B
1 M gB
+ || اووه + || AHH gD
+ 7 AHHz Agl
M 9B
G
HG
HG
(20)
B
p
11
Since A C B, the difference:
2G( + H) 2HG
A A(z B)
(21)
p3|Aa 1²
Thus, the parameter 2G(x+H)
This completes the proof. 1A
2HG
C Proofs of Lemma 2
Lemma 2. When using IRT for ability estimation, the function fS) is Lipschitz continuous with respect O. Furthermore, with probability at least
5, the approximation error incurred by using
of satisfies the upper bound: If(S 6¹) f(S) <
( + P1P2
MG
C(8) where μ1, M, H, G,
visel
and C are model-dege dedendde constants characterizing the properties of the objective function. function..
Proof. We first should prove that f(S 0) (Eiss V24(0))
Vel,(0) is Lipschitz
continuous with respect to 0, we analyze its sensitivity to small changes in 0.
Since the gradient (o); is continuously differentiable in 0, the Mean Value Theorem guarantees the existence of some & between 01 and 0g such that
(01; - vol;(02) = ())(()) - -02).
(22)
Assuming that [22((0)] < and taking the norm and summing over e S gives
(o1( Σ (02)
I
(o((())) ||0 02 = H| 02l1-
(23)
iES
iES
iES
Similarly, assuming
(o),(0011 < M, there exists some 77 between 01 and 0g such that
Σ Vzl,(01) Σ (32)(0)))
Σ
[ge. 110 02l s M|S1|00 - 0211-
(24)
1E8
iES
iES
Define: H(0) = Eies (0) and g(0) Eies (ol( analyze
(f((01)) f(S,02)) = [11)(01) - 9(01)] - 11H((02) (T((0))_'((+2)II
S |1((0,)- (1a))-'gg)) -9((0)) H((2) '9(02)ll
|[(((0))''((1) H(O)) -1 (0((2) + H((01)-1((0) -((02- (02)))
< [[H(()))'(((0) g(02)) + ([(((0) -1 H((2) ')9(0)))
(25)
For the first term, using matrix norm properties:
[+((0)))'(((}) g(02)) < |I((0)) [l((0)) (02))l
(26)
Assuming ||H(0,))~1| | in a well-conditioned region (similar to Theorem 1), we obtain:
00 - 02
(27)
μ4
22

---

For the second term, using:
-(((2) H((01) H((8)) - -((1)))(0))
and assuming [[?((02) [ we obtain:
()))))) | |[H(0())'[I((((( (1) H( =
(28)
Using previous bound [}((02) H(01)l - M|S||10 021l and assuming [|.(0)] - |S|G in bounded region, we get:
MG
[I(H((01) (((8))-))((0
|00 -21l
(29)
M1f2
Thus
H
MG
IS(S 01) S(S 02)) s
100
02/1-
(30)
141
P1
6f is obtained via Maximum Likelihood Estimation (MLE), we have: VIS.((0'
0*) 4
NO (-0,)-'(0))) where I(0") is the Fisher information matrix. This follows the asymptotic nor- mality of MLE [44). This implies that, with high probability, the estimation error satisfies
110' 1 C(8) VIS.
with probability at least
(31)
for some constant C()) depending on the trace and spectral norm of (0*())). Since S(S(0) is Lipschitz continuous in 0, with probability at least - 6,
H
MG
H MG ( f1
C(ő)
[f(S 0*) f(S)
110¹ -011
(32)
P1/2
M1p2
V
This guarantees that for sufficiently large [S,), the error introduced by using 0 in place of 0 is small with high probability.
D Full Derivation of Bias- Corrected Estimate
We begin by defining the perturbed objective:
0 Sm)
= min 8
(0)
(33)
JS|
iES
Since (m)) minimizes the objective, it satisfies the first-order optimality condition:
1 v(((0)(()))
Vm ( (f)(()) + +7VEn (9% (())))
(34)
S
We now apply a first-order Taylor expansion of the gradient around Ag:
0 Rd
v4,(0s) > (0s) + (O,,
- (As) +VVLmm((s) (0%(m) -8s). (35)
S
Choosing 7 we simplify:
[ and noting that 0g satisfies the original optimality condition Eies (l,(0s)
0 2
Vl Vem (0s) + |S|
(8s)
+
Σ ve((0s) + [s]
22 (0s) (0s(m) - 0s).
(36)
im
23

---

Let H(S 9m.0s) As) = Si (2c(( and Hm.m. 0s) = (2 (Os), we obtain:
(s(m) 0g = (H(S)) Qms 0s) H(qm 0s) (vem(es) Vim (0s)
(37)
Now, recall the definitions of the original and flipped losses:
mm (0) = -ym log Po (4m= 1) (1 Um log Pa (Am- 0),
(38)
fm (0) (1 Vm po 1) V/ log (Am 0)-
(39)
Taking the gradient difference:
(mm( Vl (0s) = V ym log Po (9m: 1) (1 3mm) log po 0)]
V 1-(1 - ymm log pa (Ams 1) - 3m log pe 0)]
po (9ms 1) po (ym, 0)
= (1 2ym ) v log
(40)
Substituting back, we obtain the final expression:
(s(m) +s + H(S 9m 0s) + H(qm 0s) 0s)] (1 2ym) Vlog Po 1) py (qmz 0)
(41)
E Limitations and Broader Impact
Despite the promising results CFAT, several limitations remain that open avenues for future research. For example, while CFAT incorporates analytical corrections for guessing and slipping, assumes these behavioral perturbations follow simple, predefined patterns. In practice, examinee behavior can be more complex and context-dependent. Future work could integrate richer cognitive models or leverage response time, clickstream, or eye-tracking data to better capture behavioral variability.
CFAT offers scalable, interpretable, and computationally efficient solution for adaptive testing, with potential for broad societal benefits:
Democratization of High-Qualics Assessment: By reducing the number required questions and computational overhead, CFAT can enable real-time, low-cost testing in low-resource settings, such as developing countries, where infrastructure is limited.
Fairer and More Inclusive Testing: The bias-correction mechanism in CFAT helps mitigate the influence of irregular behaviors (e.g., guessing), potentially leading to fairer assessments across diverse populations. This is particularly important high-stakes testing scenarios, where small inaccuracies can have significant consequences.
Privacy: CFAT does not rely on large-scale user data or extensive training, reducing the need for data collection and storage. This not only preserves user privacy but also reduces the environmental footprint of deploying large-scale Al-driven assessment systems.
24


Reviews:
[{"review_id": "r7csCHilR5", "reviewer": "Reviewer_Hxwf", "summary": "The authors proposed a closed form solution to tackle the question selection problem in Adaptive Testing systems. The detailed theoretical analysis is provided and experiments demonstrate the effectiveness of the proposed method.", "questions": "See above weaknesses.", "limitations": "See above weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This work is well motivated. \n2. Related work are well studied and discussed.\n3. The idea is neat, with extensive theoretical support.\n4. The experimental results demonstrate the improvement of the proposed method.\n\nWeaknesses:\n1. The reviewer's first question lies in the invertible assumption of Hessian matrix. Since the problem setup is to find a subset of questions that can reflect students' ability across all questions, it indicates a low rank assumption. In such cases, how could we assume the Hessian matrix is invertible, and thus leads to the closed form solution?\n2. The datasets used in the experiment section are not well introduced, e.g., statistics, distribution.\n3. It's not clear how the method can be used in large scale applications.", "was_revised": true, "final_justification": "Based on the discussions, I would vote for a borderline accept for this work. The proposed method is neat, with solid theoretical analysis, however, the effectiveness of this method in real world application is still under investigation.", "rebuttal": "Thank you for your positive feedback on the motivation, core idea, and experimental results of our paper. Here are the responses to each of your questions:\n> **Q1**: ...How could we assume the Hessian matrix is invertible? \n\n**A1**: We appreciate the opportunity to clarify. In the context of adaptive testing, **this assumption is actually standard and reasonable**. As mentioned in Section 3.1, line138 of our paper:  *\"$\\mathcal{H}(S, \\theta)$ is invertible, which holds under standard regularity conditions in IRT. For complex neural network-based models, where computing the exact Hessian is often intractable, we adopt a quasi-Newton approximation (details are provided in Appendix A).\"*\n\nLet us explain this in more detail:\n\n- The invertibility of the Hessian is a widely accepted assumption in adaptive testing, often referred to in psychometric literature as \"identifiability and estimability under local independence and monotonicity assumptions\" [1]. In fact, under standard regularity conditions of IRT models, the loss function $\\ell(\\theta)$ is strongly convex [2][3], which **ensures that the Hessian $\\mathcal{H}$ is positive definite and thus invertible**. So in classical IRT settings, this assumption is well-supported both theoretically and empirically. More importantly, all large-scale standardized tests today (e.g., GRE) are fundamentally based on IRT, which further motivates the paper’s focus on IRT-based frameworks.\n\n- For more complex models like neural network approaches (e.g., NeuralCDM), computing the exact Hessian is often intractable due to the high dimensionality and nonlinearity of the parameter space. Therefore, in these cases, we adopt **a quasi-Newton approximation of $\\mathcal{H}^{-1}$, as detailed in Appendix A**. This allows us to iteratively approximate the inverse Hessian without computing it directly, which makes the approach computationally feasible while still preserving the theoretical foundation of using $\\mathcal{H}^{-1}$ for subsequent analysis.\n\nTo avoid confusion, we'll make this assumption and its justification more explicit in the revised version. We hope this could address your concern.\n\n\n\n> **Q2**: The datasets used in the experiment section are not well introduced, e.g., statistics, distribution.\n\n**A2**: Thank you for pointing this out. Below is a summary of the key statistics for the datasets used in our experiments:\n\n| Dataset       | ASSIST     | NIPS-EDU   | EXAM       |\n|---------------|------------|------------|------------|\n| #Students     | 20,704     | 123,889    | 96,329      |\n| #Questions    | 15,071     | 28,561     | 1,714          |\n| #Responses    | 1,768,253  | 17,250,577  | 892,676    |\n| Responses/Student | 85.41  | 139.24      | 9.27      |\n| Responses/Question | 117.33 | 603.99    | 520.81      |\n\n\nWe will also provide additional visual analyses of the datasets (e.g., the distribution of the number of interactions per student), which are difficult to present here. We apologize for the oversight and all detailed dataset information and visualizations will be included in the appendix in the revised version.\n\n> **Q3**: It's not clear how the method can be used in large scale applications.\n\n**A3**: That’s a great question. Our method is actually well-suited for large-scale applications for two main reasons:\n\n1. No training required: Our approach is based on a simple greedy algorithm and does not involve any model training. This makes it significantly more efficient than training-heavy methods like reinforcement learning or active learning. As shown in Figure 2(b), our method achieves at least 12× speedup in runtime.\n   \n2. Validated on large-scale datasets: We've tested our method on very large datasets  (Table 1), such as NIPS-EDU, which includes over 220,000 students and 19 million response logs. Moreover, our method remains robust even under high-noise conditions in realistic testing environments (Table 2).\n\nWe appreciate your suggestion and we will highlight these advantages more clearly in the conclusion section in the next version, especially in the context of large-scale deployment. If anything's unclear, we'd be happy to discuss them.\n\n\nReference:\n\n[1] Lord, Frederic M. Applications of item response theory to practical testing problems. Routledge, 2012.\n\n[2] Zhuang, Yan, et al. \"A bounded ability estimation for computerized adaptive testing.\" Advances in Neural Information Processing Systems 36 (2023): 2381-2402.\n\n[3] Yu, Junhao, et al. \"A unified adaptive testing system enabled by hierarchical structure search.\" Forty-first International Conference on Machine Learning. 2024."}, {"review_id": "K8tAJuksZ5", "reviewer": "Reviewer_RhGM", "summary": "This paper addresses the approximation limitations of existing approaches to adaptive testing, which often require iterative computations and hinder their applicability in online settings. To overcome this, the authors propose a closed-form solution for question selection that supports efficient ability estimation with bias correction. The method is evaluated across four dimensions: (1) student performance prediction, (2) ability estimation accuracy, (3) computational efficiency, and (4) reliability. Experimental results on three datasets show that the proposed approach outperforms all baseline methods in most cases.", "questions": "* Table 1: To improve clarity, it would be helpful to explicitly label the variants of the proposed method in the table (e.g., \"CFAT + IRT\" and \"CFAT + NeuralCDM\") so that readers can easily distinguish the configurations being evaluated.\n\n* In the evaluation section, Figure 2 presents results for the ability estimation and computational efficiency tasks but includes only a subset of the baseline methods. Could the authors clarify the rationale for selecting these specific baselines, and why the others were omitted?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Motivation: The paper clearly identifies a relevant research gap and provides a strong motivation for addressing the computational limitations of adaptive testing in online settings.\n\nLiterature Review: The related work section effectively introduces the necessary background and the context of existing methods.\n\nMethodology: The proposed approach is well-designed and thoughtfully constructed. The theoretical analysis and proofs included in the paper provide solid support for the method's validity.\n\nEvaluation: The proposed method is thoroughly evaluated on three datasets against eight baseline models, including random selection, classical approaches, and a variety of learning-based (e.g., active learning, meta-learning, reinforcement learning, and graph neural networks) and heuristic methods. The results show that the proposed method consistently outperforms these baselines in both student performance prediction and ability estimation tasks, whether integrated with classical IRT models or neural network-based approaches. Furthermore, the method demonstrates better computational efficiency, relatively effective alignment between selected questions and examinee ability distributions, and robust reliability under response noise caused by guessing and slipping behaviors.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback on the motivation, methodology, and evaluation of our work. This is truly encouraging our commitment to further exploring this direction. We have carefully considered all your suggestions and have made following responses:\n\n> **Q1**: Table 1: To improve clarity, it would be helpful to explicitly label the variants of the proposed method in the table (e.g., \"CFAT + IRT\" and \"CFAT + NeuralCDM\") so that readers can easily distinguish the configurations being evaluated.\n\n**A1**: Thank you for pointing this out. To avoid confusion and improve clarity, we will revise Table 1 in the next version by merging the two sub-tables and explicitly labeling all method combinations, such as \"XXX + IRT\" and \"XXX + NeuralCDM\". This labeling will make it easier for readers to distinguish the configurations and better understand the comparative performance of each method.\n\n> **Q2**: In the evaluation section, Figure 2 presents results for the ability estimation and computational efficiency tasks but includes only a subset of the baseline methods. Could the authors clarify the rationale for selecting these specific baselines, and why the others were omitted?\n\n**A2**: In the evaluation section, we chose to include only a representative subset of baseline methods in Figure 2 due to space limitations. Specifically, we selected one widely recognized method from each category, following the taxonomy in [1]: Fisher for traditional information-based methods, MAAT for active learning, NCAT for data-driven approaches, BECAT for subset selection, and CFAT as our proposed method. Within each category, the runtime differences among methods are generally minor. For completeness, we've collected the full runtime statistics across all baselines as follows:\n\n| Method   | Time (s) |\n|----------|--------------------|\n| Fisher   | 0.23177            |\n| Random   | 0.25821            |\n| KL       | 0.79015            |\n| BECAT    | 4.38283            |\n| MAAT     | 148.23482          |\n| NCAT     | 19.58382            |\n| BOBCAT   | 0.69179            |\n| UATS     | 33.74935           |\n| **CFAT**     | 0.35180            |\n\n\nAs shown, CFAT achieves a strong balance between efficiency and accuracy. It runs only slightly slower than Fisher, yet delivers SOTA performance in ability estimation. Based on your suggestion, we will include the complete runtime statistics in the appendix of the next version.\n\n\n\n\nWe truly appreciate your suggestion. Please feel free to share any further insights or suggestions you might have.\n\nReference:\n\n[1] Liu, Qi, et al. \"Survey of computerized adaptive testing: A machine learning perspective.\" arXiv preprint arXiv:2404.00712 (2024)."}, {"review_id": "RU3eM8VEGd", "reviewer": "Reviewer_pzeL", "summary": "CFAT efficiently selects question subsets using a closed-form objective, outperforming meta-learning and reinforcement learning methods by an average of 2% in prediction accuracy while reducing the number of required questions by at least 15%. It achieves up to 12× faster selection than BECAT and consistently chooses high-discrimination questions aligned with examinees’ abilities, offering strong interpretability and generalizability.", "questions": "1. Same as in the Weaknesses Section. Is there a better baseline or a dataset that we can experiment with? For example, we can think of JUNYI as a dataset.\n\n2. The experimental results presented in the BECAT paper and the experimental results presented in this study are almost the same. However, in UAT and CCAT published at a similar time, there is a considerable difference in the baseline performance even though the same dataset and metrics are used. Why is this so? If there can be differences in each experiment, it seems that sufficient experimental data is needed (such as mean or std).", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1.The purpose of the research is clear and definite. The quality of the paper is also high.\n\n2.Not only are the experimental results excellent, but the theoretical support is also solid.\n\nWeaknesses\n1. Even considering the existing trends of the task, I have some doubts about whether it is a novel method that shows overwhelmingly good performance. I don't feel that it is a significant improvement over the existing baseline.\n\n2. It is questionable whether any comparisons with recent studies have been made. For example, we could easily find a comparable baseline, 'Liu, Zirui, et al. \"Computerized adaptive testing via collaborative ranking.\" Advances in Neural Information Processing Systems 37 (2024): 95488-95514.'", "was_revised": true, "final_justification": "Quality - I find this paper well written and am satisfied with the overall quality of the paper.\n\nClarity - The problem that the paper is trying to solve, its motivation, and its approach are all clearly understood.\n\nSignificance - I've looked into some related studies and provided additional baselines and datasets, but overall, I feel like the task is stagnant and saturated.\n\nOriginality - The authors improve efficiency compared to existing methods by correcting for bias caused by guessing and slipping and representing estimation errors as closed-form expressions. The proposed method is intuitive and novel. However, despite the performance improvement, it does not present a significant new direction, as it adheres to the standard evaluation methods of Adaptive Testing.\n\nOverall, this was extremely difficult to confirm, and this study has both strengths and weaknesses. The advantage is that it's a well-written, high-quality paper, but the disadvantage is that it maintains a baseline, dataset, and evaluation metrics that are largely similar to those of related studies from several years ago, making it somewhat lacking in freshness. There are reasons to accept and reasons to reject this research.\n\nConfidence - At first, this was an unfamiliar task for me, but as I continued to research related research and communicate with authors, I became quite confident.\n\nWhile the authors could have done a little more to advance the relevant tasks, the overall quality of the paper was satisfactory. Thank you.", "rebuttal": "Thank you very much for your high evaluation of the quality of our paper. Below are our detailed responses to your concerns regarding the experiments:\n\n>**Q1**: Even considering the existing trends of the task, I have some doubts about whether it is a novel method that shows overwhelmingly good performance. I don't feel that it is a significant improvement over the existing baseline.\n\n**A1**: This is a great point. In fact, almost **no** existing selection algorithm achieves overwhelmingly good performance in terms of ACC/AUC in adaptive testing. This is largely due to the nature of the task: these metrics are evaluated under small sample sizes (e.g., 5 or 10 items), but the theoretical upper bound (i.e., using all items) is limited by measurement model itself (e.g., IRT). For example, due to noise in student responses and the fitting capacity of IRT models, the AUC upper bound on the NIPS-EDU dataset is only around 0.79. Our method already achieves 0.78 with just 20 items, while the best of the other methods reaches only 0.76-0.77. So, further improvement is inherently constrained.\n\nAs you rightly pointed out, this suggests that ACC/AUC may be **approaching saturation** for this task. For example, UATS[1] only improved upon the previous best method by 0.5%–1%, and CCAT[2] did not outperform previous SOTA methods in terms of AUC/ACC at all. Because of this, the community often turns to another metric: ability estimation error (MSE) [3], which more directly aligns with the core goal of adaptive testing: minimizing estimation error. As shown in Figure 2 of our paper, our method **achieves a 15% reduction** in the number of items needed to reach the same estimation error, which we believe is a substantial and meaningful improvement.\n\n> **Q2**: It is questionable whether any comparisons with recent studies have been made. For example, we could easily find a comparable baseline, 'Liu, Zirui, et al. \"Computerized adaptive testing via collaborative ranking.\" Advances in Neural Information Processing Systems 37 (2024): 95488-95514.'\n\n**A2**: Yes. This is a recent and relevant work. Although we cited it in our paper, we didn't include it in the experimental comparison. The main reason is that its goal is quite different: it focuses on ranking students, while our work targets accurate ability estimation. Since their method is not optimized for estimation accuracy, we initially felt the comparison might not be entirely fair. That said, we agree it's important to be thorough. So, we've now included a comparison based on MSE across three datasets (ASSIST, NIPS-EDU, and EXAM), shown below:\n\n\n\n[Table A – Dataset ASSIST]\n\n\n| Method   | Step 5       | Step 10      | Step 15      | Step 20      |\n|----------|--------------|--------------|--------------|--------------|\n| Random   | 1.182±0.206  | 0.914±0.140  | 0.866±0.117  | 0.377±0.060  |\n| Fisher   | 0.931±0.133  | 0.691±0.082  | 0.498±0.078  | 0.360±0.038  |\n| UATS     | 0.918±0.073  | 0.780±0.097  | 0.328±0.065  | 0.208±0.033  |\n| NCAT     | 1.126±0.115  | 0.753±0.095  | 0.550±0.133  | 0.196±0.030  |\n| CCAT     | 1.027±0.187  | 0.702±0.052   | 0.492±0.082   | 0.192±0.021   |\n| **CFAT** | **0.894±0.044** | **0.473±0.057** | **0.216±0.022** | **0.074±0.037** |\n\n\n[Table B – Dataset NIPS-EDU]\n\n| Method   | Step 5       | Step 10      | Step 15      | Step 20      |\n|----------|--------------|--------------|--------------|--------------|\n| Random   | 1.283±0.200  | 0.994±0.156  | 0.583±0.094  | 0.402±0.162  |\n| Fisher   | 1.142±0.103  | 0.932±0.068  | 0.495±0.093  | 0.360±0.105  |\n| UATS     | 1.200±0.074  | 0.875±0.072  | 0.475±0.062  | 0.358±0.085  |\n| NCAT     | 1.183±0.101  | 0.872±0.064  | 0.417±0.084  | 0.264±0.106  |\n| CCAT     | 1.103±0.098  | 0.927±0.061   | 0.527±0.027   | 0.374±0.078   |\n| **CFAT** | **0.913±0.066** | **0.861±0.076** | **0.250±0.050** | **0.131±0.038** |\n\n[Table C – Dataset EXAM]\n\n\n| Method   | Step 5       | Step 10      | Step 15      | Step 20      |\n|----------|--------------|--------------|--------------|--------------|\n| Random   | 1.387±0.188  | 1.242±0.162  | 0.834±0.166  | 0.586±0.108  |\n| Fisher   | 1.182±0.176  | 0.916±0.152  | 0.681±0.072  | 0.314±0.029  |\n| UATS     | 1.178±0.121  | 0.820±0.102  | 0.449±0.093  | 0.291±0.049  |\n| NCAT     | 1.176±0.128  | 0.803±0.110  | 0.547±0.105  | 0.241±0.083  |\n| CCAT     | 1.183±0.152  | 0.888±0.096   | 0.568±0.077  | 0.283±0.024   |\n| **CFAT** | **1.018±0.058** | **0.735±0.075** | **0.332±0.051** | **0.127±0.020** |\n\n\n---\n\nAs shown, our method consistently outperforms CCAT in terms of MSE across all steps and datasets. We'll include these results in the next version of the paper for completeness. If you have any other baselines you'd recommend for comparison, we’d be more than happy to include them as well. Thanks again for your valuable feedback!\n\n\n> **Q3**:  Is there a better baseline or a dataset that we can experiment with? For example, we can think of JUNYI as a dataset.\n\n**A3**: Thank you for your suggestion. We agree that exploring additional datasets can help further validate the effectiveness and generalizability of our approach. In response, we have conducted experiments on the JUNYI dataset, and the results are presented below. We plan to include these results in the next version of the paper.\n| **Method**         | **Step@5**        | **Step@10**       | **Step@20**       |\n|--------------------|-------------------|-------------------|-------------------|\n| Random+IRT         | 71.61 / 73.94     | 73.11 / 76.54     | 75.46 / 79.90     |\n| Fisher+IRT         | 72.84 / 75.83     | 74.80 / 78.48     | 76.11 / 81.05     |\n| KL+IRT             | 72.15 / 74.59     | 74.19 / 78.42     | 75.50 / 80.98     |\n| MAAT+IRT           | 71.83 / 74.61     | 74.35 / 77.54     | 74.09 / 79.66     |\n| BECAT+IRT          | 74.57 / 79.34     | 76.02 / 80.61     | 76.89 / 81.48     |\n| NCAT+IRT           | 72.26 / 77.64     | 75.62 / 80.96     | 76.98 / 81.53     |\n| BOBCAT+IRT         | 72.22 / 78.23     | 75.48 / 80.22     | 76.98 / 81.67     |\n| **CFAT+IRT**       | **75.77 / 79.58** | **77.45 / 81.20** | **78.81 / 83.05** |\n\n---\n\n| **Method**             | **Step@5**        | **Step@10**       | **Step@20**       |\n|------------------------|-------------------|-------------------|-------------------|\n| MAAT+NeuralCD          | 72.81 / 75.31     | 75.06 / 77.61     | 75.62 / 80.96     |\n| BECAT+NeuralCD         | 77.11 / 80.28     | 78.33 / 83.00     | 79.02 / 84.00     |\n| NCAT+NeuralCD          | 75.33 / 79.55     | 77.59 / 81.36     | 78.44 / 84.16     |\n| BOBCAT+NeuralCD        | 76.09 / 80.24     | 77.42 / 82.79     | 78.47 / 84.06     |\n| **CFAT+NeuralCD**      | **78.23 / 82.41** | **80.51 / 83.27** | **81.91 / 86.54** |\n\n> **Q4**: However, in UAT and CCAT published at a similar time, there is a considerable difference in the baseline performance even though the same dataset and metrics are used. Why is this so?\n\n**A4**: Thank you for raising this question. Yes, although UAT and CCAT were published around the same time and used the same dataset and evaluation metrics on the surface, there are notable differences in their baseline performance. This discrepancy can be attributed to several key factors:\n\n- CCAT primarily focuses on ranking the examinees, and thus adopts Ranking Consistency metrics. In addition, the optimization strategies used in CCAT differ significantly: it employs MCMC-based methods and specific gradient descent, which are tailored to their specific modeling objectives.\n\n- Data Splitting Strategies: The data partitioning schemes differ between the two works. In our study, we follow a 7:2:1 split for training, validation, and testing, while UAT adopts a 6:2:2 split. Such differences can impact the model’s generalization and reported performance.\n\n- Data Preprocessing and Filtering: UAT is a data-driven method and applies its own data filtering criteria. Based on the released code and dataset details, it appears that UAT filters out a considerable number of examinees or items with insufficient interaction records. This can lead to a cleaner but less diverse dataset, which may inflate performance metrics.\n\n\n\n\n> **Q5**: If there can be differences in each experiment, it seems that sufficient experimental data is needed (such as mean or std).\n\n\n\n**A5**: Yes, we completely agree. In our paper, Figure 2 already presents the mean and standard deviation of MSE, with results averaged over 10 repetitions and error bars indicating the standard deviation. To further enhance the transparency of our findings, we will include the std of AUC/ACC metrics in the appendix in the next version.\n\n\nPlease feel free to reach out if you have any further concerns or would like to discuss in more detail!\n\n[1] Yu, Junhao, et al. \"A unified adaptive testing system enabled by hierarchical structure search.\" Forty-first International Conference on Machine Learning. 2024.\n\n[2] Liu, Zirui, et al. \"Computerized adaptive testing via collaborative ranking.\" Advances in Neural Information Processing Systems 37 (2024): 95488-95514.\n\n[3] Cheng, Ying. Computerized adaptive testing—new developments and applications. University of Illinois at Urbana-Champaign, 2008."}, {"review_id": "SEw6JQXt94", "reviewer": "Reviewer_W1GP", "summary": "This paper addresses the challenge of efficient and accurate human ability estimation in adaptive testing systems, which aims to select a small subset of questions tailored to an examinee’s ability level. However, existing methods rely on computationally intensive nested optimization, which limit scalability and robustness in large-scale, real-time testing environments. In response, the author(s) propose CFAT that quantifies the estimation error as first and second-order terms. The model bridges psychometric theory (e.g., IRT) with efficient computation, making it suitable for high-stakes assessments with massive question pools and diverse examinees.", "questions": "**Q1.** The paper's work is constrained by using IRT as the cognitive diagnosis model (CDM). Under IRT, the loss function is $\\mu$-strongly convex, which ensures the submodular property of the model and thus enables the use of greedy algorithms to find optimal questions. However, if other CDMs are adopted, such as neural network-based CDMs or the DINA model, does the submodular property still hold? Can greedy algorithms still be applied?\n\n**Q2.** Eq. (5) is derived by performing Taylor expansion on Eq. (4), where the authors introduce a complement set $Z$. Why not directly apply Taylor expansion to Eq. (3) instead?", "limitations": "The theoretical guarantees strictly rely on IRT’s convexity. For non-convex CDMs, submodularity remains unproven, limiting theoretical universality.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is easy to understand, and the proof seems to be correct.\n\n2. They propose the first closed-form solution for adaptive testing subset selection, eliminating nested optimization and enabling direct error minimization via gradient-Hessian alignment.\n\n**Weaknesses**\n\n1. The $\\epsilon$-submodularity proof relies critically on IRT's $\\mu$-strong convexity. For non-convex CDMs (e.g., DINA, complex neural CDMs), submodularity, and thus greedy optimality lacks theoretical guarantees.\n\n2. The design (e.g., Hessian stability term) assumes smooth and twice-differentiable losses. This may not hold for discrete CDMs (e.g., DINA) or models with non-differentiable components.\n\n3. The author(s) assumes fixed guessing/slipping probabilities ($\\pi_g$ and $\\pi_s$), which may not confirm with real-world behaviors.", "was_revised": true, "final_justification": "After reading the author(s)'s rebuttal (including for other reviewers), I think the (additional) experiments are good enough, which can support the current theoretical work. But the major point that makes me indecisive is the strong assumptions (e.g., submodularity). This assumption narrows general theoretical contributions. Hence, we just give a marginal acceptance rating.", "rebuttal": "We sincerely appreciate your positive feedback on the readability, theoretical clarity, and methodological contributions of our work.\n\n> **Q1**: if other CDMs are adopted, such as neural network-based CDMs or the DINA model, does the submodular property still hold? Can greedy algorithms still be applied?\n\n**A1**: That's a great and insightful question. To answer directly: when using more complex models like neural network CDMs or the DINA model, the submodular property generally does **not** hold. However, greedy algorithms can **still** be applied effectively in practice.\n\nFirst, we focus on IRT in the main text because it's the most widely used measurement models for adaptive testing, especially due to its simplicity and interpretability. It's the foundation for many large-scale tests like the GRE. This is why all the background, related work, and theoretical analysis in our paper are exclusively based on the IRT setting. Under IRT, the estimation objective has nice mathematical properties (e.g., strong convexity and bounded Hessian), which allow us to derive an approximate submodular structure. This is the basis for the theoretical guarantees in our Lemma and Theorem. \n\nIn contrast, models like neural networks are highly non-linear and non-convex. Their loss functions are generally not convex as you said, let alone strongly convex, so the theoretical results in our paper do not apply to these models. As a result, all the theoretical analysis in the paper is based solely on the IRT model.\n\nHowever, greedy algorithms can still be used as **heuristics** even without submodularity guarantees. In fact, many non-convex problems in practice have been shown to exhibit behavior similar to submodular functions, at least empirically [1][2]. If we can define some notion of \"informativeness\" or \"uncertainty\", greedy selection can still be a powerful strategy, e.g., active learning, coreset. Moreover, Table 1 demonstrates that even without theoretical guarantees, greedy selection still performs well in practice when applied to neural network methods. We'll highlight this point in the next version.\n\n\n> **Q2**: The design (e.g., Hessian stability term) assumes smooth and twice-differentiable losses. This may not hold for discrete CDMs (e.g., DINA) or models with non-differentiable components.\n\n\n**A2**: Yes, the majority of the design in our paper is grounded in IRT. This is because IRT remains the dominant framework in most large-scale testing, as discussed in motivation and related work section. As you noted, for discrete CDMs, there is currently a lack of reliable theoretical foundations in this domain. While our analysis centers on IRT, we do consider complex approaches as well. For example, to calculate Hessian in complex models like NeuralCDM, computing the exact Hessian is often infeasible due to the high dimensionality and nonlinearity of the parameter space. In these cases, we use a quasi-Newton approximation of $\\mathcal{H}^{-1}$, as described in Appendix A. \n\n> **Q3**: The author(s) assumes fixed guessing/slipping probabilities $\\pi_g$ and $\\pi_s$, which may not confirm with real-world behaviors.\n\n**A3**: Thank you for your comment. Yes. In real-world settings, guessing and slipping behaviors can vary across different examinees and items. In our work, we use fixed values for  $\\pi_g$ and $\\pi_s$ as a simplifying assumption to make the bias correction process more transparent and easier to interpret. Also, these probabilities are only used during the question selection stage, and not treated as item parameters that can be directly estimated. Our current approach aims to capture *general behavioral trends*, for example, $\\pi_s$ reflects an overall tendency for careless mistakes across the population. We agree that adapting or learning these probabilities dynamically could improve the model's realism. In future work, we plan to explore strategies that allow us to optimize $\\pi_g$ and $\\pi_s$ jointly during the question selection process.\n\n> **Q4**: Eq. (5) is derived by performing Taylor expansion on Eq. (4), where the authors introduce a complement set $Z$. Why not directly apply Taylor expansion to Eq. (3) instead?\n\n**A4**: Yes, performing Taylor expansion directly on Eq. (3) is mathematically equivalent to the approach we use in Section 3.1 with the introduction of complement set $Z$. Our intention here was to convey: rather than physically removing the influence of $Z$ from the item pool, we apply a soft adjustment by down-weighting its contribution. This idea aligns with the strategy discussed later in Section 3.3, where we handle noisy samples in set $S$ by down-weighting them instead of removing/replacing them. By keeping this treatment consistent across sections, we aim to improve the overall conceptual coherence of the method. To avoid confusion, we will clarify the equivalence of these two more explicitly in the revised version.\n\nThank you again for your careful review. If anything remains unclear, we'd be happy to discuss it further.\n\n\nReference:\n\n[1] Krause, Andreas, and Daniel Golovin. \"Submodular function maximization.\" Tractability 3.71-104 (2014): 3.\n\n[2] Sener, Ozan, and Silvio Savarese. \"Active Learning for Convolutional Neural Networks: A Core-Set Approach.\" International Conference on Learning Representations. 2018."}]

Author rebuttal:

Meta-review: The paper studies an interesting problem: given a (computerised) test give to a human, how to pick questions out of a predefined set in order to elicit as much information about the human's ability as possible. 

The main strength of the paper is the practicality of the proposed method. Greedy optimisation of a closed form objective is easy to implement and understand. 

The main weakness is that the assumptions made in the theoretical part are not very realistic (see review W1GP).

I lean towards acceptance because (1) the task is important and has potential for a lot of positive impact in education and (2) making idealistic assumptions is quite common in the ML community.

The authors were responsive the rebuttal phase, addressing both the reviewers' concerns and my request to compare with Bayesian approaches. The discussions with reviewers were focussed on (1) experimental evaluation (datasets / baselines), (2) the suitability of assumptions.
Decision: Accept (poster)