Title: A Closed-Form Solution for Fast and Reliable Adaptive Testing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Computerized Adaptive Testing

Human Assessment

Psychometrics

Ability Estimation

AI Education
Abstract: Human ability estimation is essential for educational assessment, career advancement, and professional certification. Adaptive Testing systems can improve estimation efficiency by selecting fewer, targeted questions, and are widely used in exams, e.g., GRE, GMAT, and Duolingo English Test. However, selecting an optimal subset of questions remains a challenging nested optimization problem. Existing methods rely on costly approximations or data-intensive training, making them unsuitable for today's large-scale and complex testing environments. Thus, we propose a Closed-Form solution for question subset selection in Adaptive Testing. It directly minimizes ability estimation error by reducing ability parameter's gradient bias while maintaining Hessian stability, which enables a simple greedy algorithm for question selection. Moreover, it can quantify the impact of human behavioral perturbations on ability estimation. Extensive experiments on large-scale educational datasets demonstrate that it reduces the number of required questions by 10% compared to SOTA methods, while maintaining the same estimation accuracy.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of efficient and accurate human ability estimation in adaptive testing systems, which aims to select a small subset of questions tailored to an examinee’s ability level. However, existing methods rely on computationally intensive nested optimization, which limit scalability and robustness in large-scale, real-time testing environments. In response, the author(s) propose CFAT that quantifies the estimation error as first and second-order terms. The model bridges psychometric theory (e.g., IRT) with efficient computation, making it suitable for high-stakes assessments with massive question pools and diverse examinees.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper is easy to understand, and the proof seems to be correct.\n\n2. They propose the first closed-form solution for adaptive testing subset selection, eliminating nested optimization and enabling direct error minimization via gradient-Hessian alignment.\n\n**Weaknesses**\n\n1. The $\\epsilon$-submodularity proof relies critically on IRT's $\\mu$-strong convexity. For non-convex CDMs (e.g., DINA, complex neural CDMs), submodularity, and thus greedy optimality lacks theoretical guarantees.\n\n2. The design (e.g., Hessian stability term) assumes smooth and twice-differentiable losses. This may not hold for discrete CDMs (e.g., DINA) or models with non-differentiable components.\n\n3. The author(s) assumes fixed guessing/slipping probabilities ($\\pi_g$ and $\\pi_s$), which may not confirm with real-world behaviors.\n\nQUESTIONS:\n**Q1.** The paper's work is constrained by using IRT as the cognitive diagnosis model (CDM). Under IRT, the loss function is $\\mu$-strongly convex, which ensures the submodular property of the model and thus enables the use of greedy algorithms to find optimal questions. However, if other CDMs are adopted, such as neural network-based CDMs or the DINA model, does the submodular property still hold? Can greedy algorithms still be applied?\n\n**Q2.** Eq. (5) is derived by performing Taylor expansion on Eq. (4), where the authors introduce a complement set $Z$. Why not directly apply Taylor expansion to Eq. (3) instead?\n\nLIMITATIONS:\nThe theoretical guarantees strictly rely on IRT’s convexity. For non-convex CDMs, submodularity remains unproven, limiting theoretical universality.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCFAT efficiently selects question subsets using a closed-form objective, outperforming meta-learning and reinforcement learning methods by an average of 2% in prediction accuracy while reducing the number of required questions by at least 15%. It achieves up to 12× faster selection than BECAT and consistently chooses high-discrimination questions aligned with examinees’ abilities, offering strong interpretability and generalizability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.The purpose of the research is clear and definite. The quality of the paper is also high.\n\n2.Not only are the experimental results excellent, but the theoretical support is also solid.\n\nWeaknesses\n1. Even considering the existing trends of the task, I have some doubts about whether it is a novel method that shows overwhelmingly good performance. I don't feel that it is a significant improvement over the existing baseline.\n\n2. It is questionable whether any comparisons with recent studies have been made. For example, we could easily find a comparable baseline, 'Liu, Zirui, et al. \"Computerized adaptive testing via collaborative ranking.\" Advances in Neural Information Processing Systems 37 (2024): 95488-95514.'\n\nQUESTIONS:\n1. Same as in the Weaknesses Section. Is there a better baseline or a dataset that we can experiment with? For example, we can think of JUNYI as a dataset.\n\n2. The experimental results presented in the BECAT paper and the experimental results presented in this study are almost the same. However, in UAT and CCAT published at a similar time, there is a considerable difference in the baseline performance even though the same dataset and metrics are used. Why is this so? If there can be differences in each experiment, it seems that sufficient experimental data is needed (such as mean or std).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the approximation limitations of existing approaches to adaptive testing, which often require iterative computations and hinder their applicability in online settings. To overcome this, the authors propose a closed-form solution for question selection that supports efficient ability estimation with bias correction. The method is evaluated across four dimensions: (1) student performance prediction, (2) ability estimation accuracy, (3) computational efficiency, and (4) reliability. Experimental results on three datasets show that the proposed approach outperforms all baseline methods in most cases.\n\nSTRENGTHS AND WEAKNESSES:\nMotivation: The paper clearly identifies a relevant research gap and provides a strong motivation for addressing the computational limitations of adaptive testing in online settings.\n\nLiterature Review: The related work section effectively introduces the necessary background and the context of existing methods.\n\nMethodology: The proposed approach is well-designed and thoughtfully constructed. The theoretical analysis and proofs included in the paper provide solid support for the method's validity.\n\nEvaluation: The proposed method is thoroughly evaluated on three datasets against eight baseline models, including random selection, classical approaches, and a variety of learning-based (e.g., active learning, meta-learning, reinforcement learning, and graph neural networks) and heuristic methods. The results show that the proposed method consistently outperforms these baselines in both student performance prediction and ability estimation tasks, whether integrated with classical IRT models or neural network-based approaches. Furthermore, the method demonstrates better computational efficiency, relatively effective alignment between selected questions and examinee ability distributions, and robust reliability under response noise caused by guessing and slipping behaviors.\n\nQUESTIONS:\n* Table 1: To improve clarity, it would be helpful to explicitly label the variants of the proposed method in the table (e.g., \"CFAT + IRT\" and \"CFAT + NeuralCDM\") so that readers can easily distinguish the configurations being evaluated.\n\n* In the evaluation section, Figure 2 presents results for the ability estimation and computational efficiency tasks but includes only a subset of the baseline methods. Could the authors clarify the rationale for selecting these specific baselines, and why the others were omitted?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

comments: SUMMARY:
The authors proposed a closed form solution to tackle the question selection problem in Adaptive Testing systems. The detailed theoretical analysis is provided and experiments demonstrate the effectiveness of the proposed method.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This work is well motivated. 
2. Related work are well studied and discussed.
3. The idea is neat, with extensive theoretical support.
4. The experimental results demonstrate the improvement of the proposed method.

Weaknesses:
1. The reviewer's first question lies in the invertible assumption of Hessian matrix. Since the problem setup is to find a subset of questions that can reflect students' ability across all questions, it indicates a low rank assumption. In such cases, how could we assume the Hessian matrix is invertible, and thus leads to the closed form solution?
2. The datasets used in the experiment section are not well introduced, e.g., statistics, distribution.
3. It's not clear how the method can be used in large scale applications.

QUESTIONS:
See above weaknesses.

LIMITATIONS:
See above weaknesses.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-08