Computerized Adaptive Testing via Collaborative Ranking
Authors: Zirui Liu
Yan Zhuang
Qi Liu
Jiatong Li
Yuren Zhang
Zhenya Huang
Jinze Wu
Shijin Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Data Mining
Intelligence Education
Computerized Adaptive Testing
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: As the deep integration of machine learning and intelligent education, Computerized Adaptive Testing (CAT) has received more and more research attention. Compared to traditional paper-and-pencil tests, CAT can deliver both personalized and interactive assessments by automatically adjusting testing questions according to the performance of students during the test process. Therefore, CAT has been recognized as an efficient testing methodology capable of accurately estimating a student’s ability with a minimal number of questions, leading to its widespread adoption in mainstream selective exams such as the GMAT and GRE. However, just improving the accuracy of ability estimation is far from satisfactory in the real-world scenarios, since an accurate ranking of students is usually more important (e.g., in high-stakes exams). Considering the shortage of existing CAT solutions in student ranking, this paper emphasizes the importance of aligning test outcomes (student ranks) with the true underlying abilities of students. Along this line, different from the conventional independent testing paradigm among students, we propose a novel collaborative framework, Collaborative Computerized Adaptive Testing (CCAT), that leverages inter-student information to enhance student ranking. By using collaborative students as anchors to assist in ranking test-takers, CCAT can give both theoretical guarantees and experimental validation for ensuring ranking consistency.

Computerized Adaptive Testing via Collaborative Ranking
Zirui Liu', Yan Zhuang', Qi Liu Jiatong Li', Yuren Zhang', Zhenya Huang',
Jinze Wu', Shijin Wang*
State Key Laboratory of Cogmitive Intelligence.
University of Science and Technology of China
2: Institute of Artificial Intelligence, Hefei Comprehensive National Science Center 3: iFLYTEK Co., Ltd
(liuzirui,zye cslii yr16699 mxwjzz]oo (ewwjz]/mmall Smailmmaalll.... ustc. edu. cn , hwwngnhyyyuusteeeeeeoaooo edu cn, sjwang36iflytei com
Abstract
As the deep integration of machine learning and intelligent education, Comput- erized Adaptive Testing (CAT) has received more and more research attention. Compared to traditional paper-and-pencil tests, CAT can deliver both personalized and interactive assessments by automatically adjusting testing questions according to the performance of students during the test process. Therefore, CAT has been recognized
efficient testing methodology capable of accurately estimating a student's s ability with a minimal number of questions, leading to its widespread adoption mainstream selective exams such the GMAT and GRE. However, just improving the accuracy of ability estimation is far from satisfactory in the real-world scenarios, since an accurate ranking of students is usually more im- portant (e.g., in high-stakes exams). Considering the shortage of existing CAT solutions in student ranking, this paper emphasizes the importance of aligning test outcomes (student ranks) with the true underlying abilities of students. Along this line, different from conventional independent testing paradigm among students, propose novel collaborative framework, Collaborative Computer- ized Adaptive Testing (CCAT), that leverages inter-student information to enhance student ranking. By using collaborative students as anchors to assist in ranking test-takers, CCAT can give both theoretical guarantees and experimental validation for ensuring ranking consistency.
Introduction
With the rapid advancements in computer science, online education has undergone significant transformation, reshaping and displacing traditional offline educational assessment techniques. In this evolving landscape, Computerized Adaptive Testing (CAT) 1 2 has emerged prominent methodology for standardized testing, widely adopted in selective exams such the GMAT 3 GRE 4. and TOFFL [S. Diverging from traditional paper-and-pencl tests, CAT offers personalized and interactive assessments, where the difficulty and characteristics of questions are continuously adapted based real-time responses. By aligning questions with current estimation of students' abilities, CAT refines the estimation process each iterative step [6) Upon test completion, final ability score shown in Figure I is provided as score report to students. This score plays pivotal role in influencing their educational and career prospects.
*Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

GRE Score Report
Ranking Consistancy
6.7
100
Truo 0.2
MBE
BECAT BOBCAT
Teat 0
NCAT
Test Step
(8)
Figure 1: (a) The score report provided by GRE and an example to show that low MSE cannot guarantee the correct ranking of students' testing results. (b) This line chart shows the performance of previous CAT methods in ranking, and can be seen that the method that performs state-of-the-art (BECAT) in accuracy may only achieve the effect of random selection in ranking.
However, while massive efforts have been made on optimizing the accuracy of ability estimation via [2], it crucial to underscore that accurate ability estimation does not inherently guarantee correct student ranking. As illustrated in Figure a). minimizing mean squared error (MSE) ability scores does not always translate into accurate rankings of students. In fact, even state-of-the-an art (SOTA) question selection algorithms with superior accuracy performance can exhibit inconsistencies in ranking performance, sometimes performing worse than random selection methods, as presented in Figure [bb. Meanwhile, the asynchronicity and independency between different students in the CAT test process significant technical challenge in achieving accurate ability ranking. This issue prevents the utilization of all students' testing information together for question selection to enhance ranking precision among students, thereby complicates the resolution of the ranking consistency issue in CAT.
improvements the question selection algorithms 77 8 9
[3 [4] is
To address this challenge, we propose novel framework Collaborative Computerized Adaptive Testing (CCAT), which introduces a collaborative learning [15] 16] approach that leverages data from collaborative students as ranking anchors. This framework facilitates interaction among test-takers, allowing for more robust ranking results. Importantly, we also present theoretical analysis that demonstrates how, with sufficient number of collaborative students, the ranking consistency error can be significantly reduced to an acceptable level. In summary, our contributions are:
To the best of our knowledge, this is the first research to unveil the ranking consistency dilemma inherent in CAT, by providing its formal definition and approximation. This discovery has enabled us to significantly refine the objectives of CAT, which is a vital advancement for its deployment in high-stakes examination contexts.
We introduce novel, collaboration- based methodology that enhances both question se- lection and ability estimation to minimize ranking inconsistency, providing theoretical guarantees for ranking consistency even with limited number of questions.
Our methodology general enough to integrate with existing question selection algorithms. Empirical results extensive real-world educational datasets proves the effectiveness of CCAT, manifesting in an average 5% rise in ranking consistency compared with other methods, and this improvement is more significant in the short test scenarios.
Related Work
CAT is designed to efficiently and accurately estimate students' abilities[2). It is widely employed in various competitive exams, including the GRE, CAT essentially operates in two stages: first, uses methods such as Item Response Theory (IRT) 17 to estimate students' abilities. Subsequently, uses these estimations to select the next question for each student. The following paragraphs separately outline Item Response Theory and common question selection algorithms used in CAT.

---

Item Response Theory (IRT). IRT is a psychological measurement theory predominantly em- ployed education to estimate students' abilities [17 18 19. It posits that examinees' abilities remain constant throughout test, and their performance depends solely on their ability and the information provided by the questions The standard model is the two-parameter logistic (2PL.) model: (0) = P(u, = 1) (a, b2)) )), where (x) Ite-= is sigmoid function and y/- = indicates a correct response to question j. The parameters aj. bj E IR represent the discrimination and difficulty of question j. These parameters are estimated by algorithms such as Markov Chain
Monte Carlo (MCMC) 220 21 and Gradient Descent (GD) [22 [ before testing. 0 E R represents the student's s ability, which is estimated using the maximum likelihood method at each step l:
0' arg max
Σ U/ In P3() + +(1 vイ) In (1 (1))(()).
kn
(1)
OEO
28 leveraging the rapid advancements in deep learning technologies (e.g., the neural networks) have significantly enhanced the accuracy of student leverages a non-negative fully connected neural network to capture the complex student-question interactions to achieve a more accurate estimation.
In recent years, the increasing studies 24 25 26
27
ability estimation. For example, NeuralCD (24)
Selection Algorithms. Research on selection algorithms can be categorized into two main ap- proaches: traditional rule-based algorithms and data-driven algorithms. Firstly, traditional question selection algorithms(299 [29] 30, 31 view CAT as parameter estimation problem. They calculate the information value of each question based on the student's current proficiency and select the question with the maximum information value[32], typically using metrics such as Fisher Information (FSI) (32 and Kullback-Leible Information (KLI) 33 Subsequently, in order to optimize the accuracy of the test result directly, researchers have proposed methods such as MAAT [4], BOBCAT (35) and
NCAT [6). which are based on active learning [37) meta-learning [38 and reinforcement learning [39. Recently, BECAT 440] proposes to use the ability estimated by student's 's full responses on the entire question bank as the true value and solve the CAT problem using a data efficiency method [4]]
In fact, in many exams, especially selective exams, the ranking of grades is usually one of the most important bases for employment. So we argue that the requirement of students in CAT is not necessarily a more precise estimation of their abilities on the test set. Rather, CAT should ensure that students with stronger abilities receive better rankings. Consequently, we establish the ranking consistency of CAT as our primary objective.
Ranking Consistency of CAT
We first assume that the testing step in CAT is uniformly T steps and all the selected questions come from question bank Q. The questions answered by each student constitute subset S C Q. For each step f, the student's s ability estimated by IRT is 6* and the student's 's final result is 8" when the test stops. For traditional CAT methods, the goal is that test results 0T should be close as possible to the true abilities of students 6* with fewer questions 40
min ||0T 11 |S|=T
where 6* is approximated by the abilities of students estimated by their full responses to the entire question bank Q [40). However, as previously mentioned, CAT often prioritize the issue ranking among students over merely improving the accuracy of 0",, For instance, if students learn that peer with lower true ability outperforms them in CAT, they may question the fairness of the exam Thus, we define the consistency of CAT ranking follows:
Definition 1. (Ranking Consistency of CAT) In computerized adaptive testing, the true abilities of two students are represented by 0 and 07. The testing results of these two students on subsets Si and S2 of question bank Q are denoted by 0f and 0%. The ranking consistency of testing demands that students with higher true abilities should also exhibit higher testing results:
max -T
P(O > 0% 10 > 05).
(3)
3

---

Question Selection Part
Ability Estimation Part
4
Select Algorithm
Ranking
8 O
& &
Ranking Ability
Responses
students:
XXXX VXVV
This question = diffieuit, and we comnot correctly answer You have no need waste time this question!
Collaborative Students
8
Question Answer Correctly
Question Anewer Inorrectly
Compare
Untested Question
Tested Student
O
X ?X?
IRT Estimation
IRT Ability 8*
Figure 2: The structure of CCAT framework. CCAT consists of two parts: question selection and ability estimation. The question selection part utilizes the performance of collaborative students in answering various questions to select appropriate questions for the tested student, and the ability estimation part ranks the tested student with collaborative students and uses the ranking the test result.
Given the varied performance, queries, and progress of the students undergoing testing, they remain independent during the CAT process. Consequently, impractical to intervene in ranking con- sistency by selecting questions based on each others' performance in the test. This complicates the direct optimization of this problem.
The CCAT Framework
To address the problem of ranking consistency, in this section, we first introduce the concept of collaborative students as anchors for the tested students. Then we elucidate their application in question selection and ability estimation. Finally, conducted theoretical analysis of the collaborative student method, demonstrating that while the ranking of the tested students among collaborative students may not be entirely accurate, the likelihood of achieving ranking consistency in CAT can reach at least 8 when a sufficient number of collaborative students are available.
Definition 2. (Collaborative Students) Collaborative students represent group with M students who are utilized as anchors to assist ranking test-takers 44 45 can be assumed that collaborative students have already completed answering all questions in the question bank Q, and their abilities on question bank Q or subset S(SS| = T) are 0" and e"" which can be obtained casily.
Due to the absence of information disclosure between any two students during the testing process, we cannot directly intervene their ranking relationship. Nonetheless, since the collaborative students answered every question from the question bank, we can hypothesize that each collaborative student will accompany the tested students in responding to the same questions during the test. This could facilitate the establishment of relationships among the tested students.
Specifically, when two students, and B, answer distinct sets of questions, say q1, 92, Q3 for student A and qs, 45. for student B, inconsistencies may arise due to the dissimilarity of the questions. However, each collaborative student can compare their performance with both students A and B. For instance, a collaborative student can assess her performance on questions qı, student A and on questions Q4, Qs- Q6 alongside student B. the collaborative student finds that her abilities exceed those of student A but fall short of student B, she will provide valuable information for accurately ranking students and
alongside
4.1 Problem Approximation
As previously mentioned, our goal is to establish the ranking relationship between tested students by comparing with collaborative students. Obviously, the first step in ensuring the ranking consistency among tested students to establish ranking consistency between the collaborative students and the tested students:
max P > 0% 10 > S).
S =

---

In Section 2, we outlined the estimation method for f in Item Response Theory, as presented in Equation (1). Utilizing this formula, we can derive the subsequent lemma, which aids simplifying the optimization objective.
Lemma 1. Given h1. D2. ... Tr. their testing abilities on S are ,",6" HT, which are estimated by IRT with param- eters at, bj. We can prove that if (6" - > then E a;(y: 5.) > O, vice versa
students, whose responses on S([S) T) are yt, U2." yr and
Lemma posits that if two students are tested on the same question subset, the term Σi az(y; - vg) can be used to replace 8 07 because they share the same sign (either positive or negative). This substitution leads to more streamlined formulation of the objective:
P(0": > 10 > S) P
Σ 9.E3
a.(yi v() > 0|5,0 0 > 6:
Σ
ai P(y > v0 02)
(5)
9:ES
=
Σ
R(q: 0r > 07):
9.5
where R(g./0" > =0)==aP(m P(y) 0 02), yf and yj represent the responses of collaborative students and tested students to question respectively. The above derivation assumes that all questions in the question bank Q are independent. and students with high abilities should perform well on each question. This formula indicates that for each tested student, answering questions that students with weaker abilities cannot answer correctly enhances ranking consistency.
Considering the asymmetry between collaborative students and tested students, we also need to consider the situation where collaborative students have stronger abilities than tested students:
P(OT < <0.'0- < 5)
Σ R(q.10" < ())) GLES
(6)
where R(gs/0" < P;: 1 < 07) Similar to equation (5), our objective
to shield students from being assessed on questions that students with higher abilities may struggle to answer accurately. By utilizing the constraints from formulas (5) and (6), we can select specific questions for the tested students based on their collaborative students:
Ct arg !EQUS.-1
P(O" < Ro < <0.) + P(O' > 0.)(( > 02)
Here S represents the subset of questions selected up to step with = S U {gr} where qa is the question selected at step t. This formula aims to find questions that collaborative students with higher abilities are likely to answer correctly, while tested students may struggle with. Meanwhile, it also identifies questions that collaborative students with lower abilities are unlikely to answer correctly, while tested students may respond correctly. The selection method enhances the performance of the originally strong students while diminishing that of weaker ones, aiding tested students in determining their ranking among collaborative students.
After testing, the tested students received their performance on S, as well as their ranking relation- ship with each collaborative student. In the study, we used the mean ranking relationship among collaborative students as the test results for the tested students:
( acl - vf) > iES
>0) (<a(k-v))>))
or= = [I (0" > 02) = E
(8)
where 7 (-) is the indicator function. Due to the uncertainty of the tested students' abilities and the incomplete responses from collaborative students during the testing process, we further approximate and elucidate the optimization problem in appendix section
5

---

Algorithm 1: The CCAT framework
Require: Q-question /TT estimattooo method.
Initialize: Random initialize tested student's ability 6%, initialize the question subset S. <- 0, the tested student's record Y 0 and collaborative students' records Y +- 01
1 for f = to T do
Select question:
Or - arg max mas P(O* < <0)) < <0.) + P(O > 0') > 0c)
St + S {
Get tested student's and collaborative students' answer:
Y Y U [vr). y +- y U ((vfr- ... -
Update students' estimate ability 0:0º arg minfge log pe pe V1))
5 Calculate tested student's rank in collaborative students: 0T + 1
Output: The student's final estimate ranking ability 0T. M Ei o o( a,(y% - -))))
4.2 Theoretical Analyses of CCAT
Through the above derivation and approximation, provide the selection algorithm and estimation method for CCAT, which can ensure high degree of consistency in ranking between collaborative and tested students. This ranking is then used to provide the test results for the tested students, denoted as have the following conclusion:
@r. Regarding the test result 0 in ability estimation,
Theorem 1. Given two students A and B, their relationship with collaborative students are T1,T2, ... .MMi T1, ,,, T; E {0, 1} indicating whether student A outperforms collaborative student in given test. Assuming the probability that student A outperforms the collaborative stu- dents is P(r, = 1) = Ç1 and student B outperforms the collaborative students is P(r: = 1) = C2-
Then the following conclusion be drawn:
In - collaborative students are provided, the prediction of ranking consistency will be at least 1
If >
(2) When the number of test questions T is small, the assessment of the ranking relationship between the tested students and collaborative students may yield inaccurate results. Assuming an ervor In collaborative students probability of p E (0, 0.5), we can still derive that if M > 2(1 (11-20
are provided, the prediction of ranking consistency will be least
can deduce that having sufficient number of collaborative students ensures consistent ranking of abilities among all tested students, even the presence rank errors between the tested and collaborative students. Meanwhile, Our question selection algorithm actually reduces the ranking error p by maximizing the ranking consistency between collaborative and tested students, thereby theoretically increasing the ranking consistency.
Drawing from Theorem
Algorithm [ outlines the pseudo-code for the CCAT framework. During the question selection phase, the complexity of our proposed question selection algorithm is D(QQTTM) as it involves selecting the most appropriate question from the question bank Q with a complexity of O(Q for each tested student. Here, T' denotes the total number of questions in the test, M is the number of collaborative students, and N is the number of students being tested. It can be observed that the time complexity of CCAT is comparable the inference speed of data-driven CAT methods. However, CCAT circumvents the time-consuming training process by storing collaborative students. Although this does increase spatial complexity, it significantly reduces the time required for training and eliminates the need for repeated training of models due to system changes.
5 Experiments
In this section, to demonstrate the effectiveness of CCAT on ranking consistency, we compare the per- formance of CCAT on the ranking consistency metric with other baseline methods on two real-world datasets. In addition, we conduct a case study to compare IRT and collaborative ability estimation and gain deeper insights on how collaborative ability estimation leads to ranking consisteney.

---

methods to directly 5.1 Expermental Setup
The goal is to ensure consistency the ranking of the test results of tested students on the subsets S and their abilities on all questions in question bank Q. In this study, we use the Kendall coefficient [46] between the abilities of tested students on the subsets S and on question bank Q, which we call intra-class ranking consistency:
Evaluation Method.
2
K
Σ
Uí5
N(N - 1)
ISKJJSN
{
(0, - -0) 0 20
U3
(0; -00-0310 03) (0:-0))00
0 <0
For any two students, if the ranking of their test results aligns with their true abilities, the metric record is 1. Conversely, if the ranking of test results diverges from their true abilities, the metric record is O.
Similarly, we can also examine the ranking consistency between the tested students and collaborative students, which we refer to as inter-class ranking consistency:
M
K
Unj:
(10)
MN
where M and N are the number of collaborative students and tested students. In addition, we also discuss AUC, ACC indicators in the main text, and NDCG indicator is used as a reference indicator
in appendix section D.2 47 48 49].
Dataset. We individually conduct experiments on two educational benchmark datasets, NIPS-EDU and JUNYI. NIPS-EDU [50] is a dataset compiled from student question interactions collected from Eedi and used in the NeurIPS 2020 Educational Challenge. JUNYI [51] is sourced from junyiacademy.org providing millions of response data from students enrolled in a course between 2018 and 2019. The rationale for selecting these two datasets is their extensive student population and the high volume of questions answered by each student, thus facilitating the construction of the collaborative student set. We filter out students who answer less than 50 times and questions that answered less than 50 times in the following experiment and then divide the dataset into a training dataset (Collaborative Students) and testing dataset (Tested Students) in 4:1 ratio. The code can be found in the github: https::/ /ttp///gtttttbbt com/ bi bigdata-
Compared Approaches. This article primarily focuses the discussion of ranking consistency in testing, and therefore, we employ IRT, which can provide practical significance results 0. As we know, Monte Carlo sampling (MCMC) and gradient descent (GD) methods estimate the IRT parameter ag b. In this experiment, we respectively employ the IRT model, estimated by both the GD and MCMC methods. and conduct question selection and student estimation. In terms of the question selection algorithm, we select the following SOTA algorithms as the baseline: Random Randomly select a question for students each step, which is benchmark to evaluate the improvement of other selection algorithms. FSI (32) and KLI [33] select the question with the highest Fisher/KL information, which measures how much information of students' abilities 0 can be obtained by answering a question. MAAT (34 utilizes active learning methods to measure the information each question brings testing. BECAT 40 regards CAT question selection as a coreset selection problem and provides an approximate solution strategy. BOBCAT (35] proposed Bilevel Optimization-bee -basdd framework to synchronously optimize the question selection algorithm and estimation model. NCAT (36] respectively utilizes the ideas of reinforcement learning, and uses
optimize the accuracy of CAT test results.
5.2 Results and Discussion
To prove the superiority of CCAT framework, we respectively compare various CAT question selection algorithms IRT estimated by GD and MCMC methods. The following conclusions obtained:
Intra-class Ranking Consistency Performance. Table | indicates that Method X consistently enhances ranking consistency at every step after employing collaborative ability estimation (X-C).
7

---

Table 1: The Performance of Different Question Selection Algorithms on Intra-class Ranking Consistency. Algorithm X-C means use algorithm X for question selection but use collaborative ability estimation proposed in CCAT as the testing result instead of the abilities estimated by IRT. CCAT (w/o C) means using the question selection algorithm but estimating the ability by IRT. The bold font represents significant improvement in statistics compared to the baseline.
(a) Intra-class Ranking Consistency Performance on IRT estimated by GD
Dataset
NIPS-EDU
JUNYI
Method Type
Step
10
15
20
10
15
20
BOBCAT
0.5770
0,6362
0.6572
0.6666
0.7104
0.7647
0.7882
0.8044
Random
0.7041
0.7434
0.7680
0.7856
0.6875
0.7350
0.7671
0.7914
FSI
0.7236
0.7889
0.8192
0.8321
0.7639
0.8284
0.8586
0.8740
Baseline
KLI
0.7328
0.7868
0.8142
0.8316
0.7748
0.8340
0.8623
0.8817
MAAT
0.6725
0.7095
0.7359
0.7535
0.6908
0.7465
0.7817
0.8113
NCAT
0.7611
0.8020
0.8266
0.8359
0.5198
0.6341
0.6803
0.7056
BECAT
0.7087
0.7542
0.7802
0.7957
0.7248
0.7712
0.7920
0.8030
Random-C
0.6988
0.7444
0.7715
0.7909
0.6862
0.7383
0.7734
0.7979
FSI-C
0.7340
0,8031
0.8339
0,8546
0.7736
0.8313
0.8623
0.8768
KLI-C
0.7399
0.7982
0.8304
0,8509
0,7813
0.8367
0.8671
0.8847
MAAT-C
0.6689
0.7175
0.7475
0.7603
0,7040
0.7822
0.8222
0.8464
Ours
NCAT-C
0.7691
0.8072
0.8317
0.8412
0.5049
0.6619
0.7194
0.7663
BECAT-C
0.7292
0.7959
0.8279
0.8438
0.7603
0.8295
0.8603
0.8769
CCAT (w/o C)
0.7320
0.7870
0.8177
0.8279
0.8026
0.8560
0.8819
0.8978
CCAT
0.7533
0.8081
0.8364
0.8543
0.8092
0.8647
0.8911
0.9066
(b) Intra-class Ranking Consistency Performance IRT estimated by MCMC
Dataset
NIPS-EDU
JUNYI
Method Type
Step
5
15
15
Random
0.7411
0.8061
0.8348
0.8540
0.6527
0.7759
0.8292
0.8600
FSI
0.7912
0.8570
0.8846
0.8975
0.8212
0.8820
0.9092
0.9257
Baseline
KLI
0.7821
0.8532
0.8804
0.8965
0.8124
0.8795
0,9082
0.9244
MAAT
0.6762
0.8083
0.8588
0.8843
0.7404
0.8506
0.8925
0.9161
NCAT
0.7766
0.8451
0.8710
0.8831
0.7430
0.8203
0.8526
0.8737
RECAT
0.7685
0.8441
0.8766
0.8958
0.7857
0.8699
0.9031
0.9225
Random-C
0,7531
0.8084
0.8363
0.8547
0.7511
0.8074
0.8429
0.8667
FSI-C
0.7933
0.8573
0.8848
0.8977
0.8226
0.8820
0,9090
0.9251
KLL-C
0.7839
0.8530
0,8805
0.8966
0.8146
0.8795
0,9079
0.9237
MAAT-C
0.6909
0.8090
0.8595
0.8848
0.7441
0.8512
0.8926
0.9157
Ours
NCAT-C
0.7923
0.8501
0.8725
0.8840
0.7829
0.8359
0.8615
0.8784
BECAT-C
0.7680
0.8449
0.8771
0.8961
0.7932
0.8706
0.9027
0.9217
CCAT (wlo C)
0.7982
0.8561
0.8832
0.8955
0.8190
0.8823
0.9098
0.9277
CCAT
0.8149
0.8635
0.8851
0.8969
0.8448
0.8875
0.9100
0.9273
This finding aligns with Theorem which subst antiates the effectiveness of collaborative ability estimation in CCAT. Furthermore, when comparing Method X-C, whether employing MCMC or GD methods for estimating IRT model parameters, our CCAT algorithm demonstrates superior perfor- mance in ranking consistency across two public datasets. Particularly, CCAT shows more significant improvement when fewer questions are tested, outperforming other methods. As the number of test steps increases, the FSI-C method improves ranking consistency more rapidly, ultimately achieving high level of consistency. This is attributed to the FSI method's ability to select questions with higher discrimination and uncertain responses, enabling the FSI-C method promptly adjust students with inconsistent ranking. However, due to the FSI method's s sensitivity to current abilities, it performs inadequately when fewer questions tested. These results confirm that the CCAT framework is generally effective in ranking for CAT, whether in terms of test duration or estimation model.
Inter-class Ranking Consistency Performance. After each baseline selection algorithm is com- pleted, we replace the original results obtained by directly using IRT for parameter estimation with the ranking results obtained from collaborative ability estimation. From Tables I and 2 it can be seen that there a positive correlation between the ranking consistency of the tested students among the collaborative students (Table 2 and the ranking consistency among the tested students (Table
8

---

Table 2: Inter-class Ranking Consistency Performance on IRT estimated by MCMC, which measures the accuracy of the collaborative ability estimation.
Dataset
JIPS-EDU
JUNYI
Step
Random
0.7798
0.8325
0.8590
0.8760
0.7651
0.8298
0.8648
0.8865
FSI
0.8258
0.8785
0.9013
0.9126
0.8575
0.9050
0.9249
0.9363
KI.I
0.8195
0.8758
0.8985
0.9119
0.8502
0.9028
0.9240
0.9353
MAAT
0.7242
0.8373
0.8807
0.9023
0.7830
0.8767
0.9069
0.9249
NCAT
0.8286
0.8697
0.8892
0.8994
0.8090
0.8604
0.8830
0.8972
BECAT
0.8045
0.8676
0.8948
0.9104
0.8287
0.8961
0.9204
0.9341
CCAT
0.8476
0.8839
0.9013
0.9116
0.8736
0.9082
0.9255
0.9373
RAND
FS1
MAAT
NCAT
BFCAT
CCAT
RORCAT
0.723
0.65
AUC GGD
ACC (GD)
0.690
0.675
0.5
0.555
0.500
Test Step
Test Step
0.75
0.055
AUC (MCMC)
ACC (MCMC)
0.70
0.600
9:000
0.75
Test Step
Test Step
Figure The performance on ACC and AUC of different question selection algorithms on the dataset NIPS-EDU for the IRT model estimated by MCMC and GD methods.
T when using the collahorative ability estimation method, especially, CCAT method a leading position in both two tables, and FSI method is only second to CCAT. This also explains why optimize the ranking consistency among the collaborative students in the above section.
ACC&AUC. Figure disspaass the metrics (ACC, AUC) obtained through various question selection algorithms on IRT, estimated by different methods. It is evident that CCAT, when compared to other CAT question selection algorithms, does not significantly differ in terms of ACC and AUC indicators. This suggests that CCAT maintains the accuracy of CAT test results while enhancing ranking consistency. Furthermore, IRT estimated by MCMC significantly outperforms that estimated by GD and BOBCAT. This also explains why the same question selection algorithm Table I performs better on the IRT model obtained through MCMC. Additionally, question selection algorithms proposed on GD, particularly those such as NCAT that utilize data-driven methods, not efficient for IRT estimated by MCMC. This implies that these methods may not be effective, but can compensate for the drawbacks of using GD estimate IRT. However, methods like BOBCAT, which concurrently train the IRT model alongside a question selection algorithm, may introduce bias into IRT model. As depicted in Figure 3 while it outperforms all gradient descent methods in specific optimization objectives (ACC@ 20). it may impact accuracy at other times and compromise the stability of the IRT model in ability estimation. This can result in suboptimal performance in ranking problems. Given the analysis above and the stability of the MCMC method, assert that

---

NIPS-EDU
JUNYI
Figure 4: Visualization of differences in abilities estimation by IRT method and CCAT method
more appropriate for IRT parameter estimation than the GD method, particularly when considering the ranking consistency of CAT.
Case Study. To demonstrate the superiority of CCAT and its mechanism, we select 10 student pairs from each dataset and conduct two visualization experiments as shown in Figures 4 This figure compares the ability gap between student pairs as estimated by the IRT and CCAT methods. Specifically, for each student pair, we subtract the estimated ability of the student with higher true ability from that of the student with lower true ability at each moment. A larger gap indicates better discrimination by the estimation method. When the value less than (red), it signals a ranking inconsistency at that point in time. Our findings show that, although the selection algorithm remains unchanged, CCAT produces greater discrimination and more accurate rankings, particularly when fewer testing steps are involved.
We also analyze the estimation results for collaborative students on these 20 student pairs, revealing that the collaborative ability estimation method essentially functions as voting process by collabo- rative students for the tested students. Additionally, we visualize how each collaborative student's judgment of the two students becomes progressively clearer as the number of test questions increases. For further details, please refer to Appendix D.2
6 Conclusion
This article explored the objectives of Computerized Adaptive Testing (CAT) from the perspective of students, reframing CAT challenges ranking tasks and proposing specific objectives for these tasks. To address the challenge students working independently, which limits influence on rankings during the testing process, introduced Collaborative Computerized Adaptive Testing (CCAT) framework. This approach leverages collaborative student interactions to assist in question selection and estimation during testing. Experiments on two real-world datasets demonstrated that CCAT improves ranking consistency. Despite these promising results, our method has inherent limitations, particularly with longer testing sequences. In future work, we aim refine our model address these limitations and enhance the robustness and effectiveness of the CCAT framework across diverse testing scenarios.
Acknowledgments and Disclosure of Funding
This research was supported by grants from the National Key Research and Development Program of China (Grant No. 2021 YFF0901003), the Key Technologies R & D Program of Anhui Province (No. 202423k09020039,, the University Synergy Innovation Program of Anhui Province (GXXT-2022- 042) and the Fundamental Research Funds for the Central Universities.
10

---

References
[1] Craig N Mills and Manfred Steffen. The gre computer adaptive test: Operational issues. In Computerized adaptive testing: Theory and practice, pages 75-99. Springer, 2000.
[2] Qi Liu, Yan Zhuang, Haoyang Bi, Zhenya Huang, Weizhe Huang, Jiatong Li, Junhao Yu, Zirui Liu, Zirui Hu, Yuting Hong, et al. Survey of computerized adaptive testing: machine learning perspective. arXiv preprint arXhv:2404.00712. 2024.
[3] Nathan R. Kuncel, Marcus Credé, and Lisa L. Thomas. A meta- veeta-nnaysiis of the predictive validity of the graduate management admission test (gmat) and undergraduate grade point average (ugpa) for graduate student academic performance. Academy of Management Learning and Education, 6:5 2007.
[4] WangZe and ChenJiliang. Cat: A case study of gre. 2004.
[5] Andrew T Barnette. The effect of the computerization of the toefl on the english language proficiency testing of international students at the university of mississippi. University of Mississippi, 2005.
[6] Wim J Van der Linden and Peter Pashley. Item selection and ability estimation in adaptive testing. In Computerized adaptive testing: Theory and practice, pages 1-25. Springer, 2000.
[7] Robert D Gibbons, Margarita Alegría, Li Cai, Lizbeth Herrera, Sheri Lapatin Markle, Francisco Collazos, and Enrique MaraiGa Successful validation of the cat-mh scales in a sample of latin american migrants in the united states and spain. Psychological assessment, 30(10): 2018.
[8] Soonwoo Kwon, Sojung Kim, Seunghyun Lee, Jin Young Kim, Suyeong An, and Kyuseok Kim. Addressing selection bias in computerized adaptive testing: A user-wise aggregate influence function approach. In Ingo Frommholz, Frank Hopfgartner, Mark Lee, Michael Oakes, Mounia Lalmas, Min Zhang, and Rodrygo I.. TT Santos, editors, Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023, pages 4674-4680, ACM, 2023.
[9] Hangyu Wang, Ting Long, Liang Yin, Weinan Zhang, Wei Xia, Qichen Hong, Dingyin Xia, Ruiming Tang, and Yong Yu. GMOCAT: A graph-enhanced multi-objective method for comput- erized adaptive testing. In Ambuj K. Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopulos, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye, editors, Proceedings the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pages 2279-2281 ACM, 2023.
[10] Junhao Yu, Yan Zhuang, Zhenya Huang, Qi Liu, Xin Li, Rui Li, and Enhong Chen. A unified adaptive testing system enabled by hierarchical structure search. In Forty-first International Con- ference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. Openkeview.net, 2024.
[11] Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Qingyang Mao, Shijin Wang, et al. Efficiently measuring the cognitive ability of Ilms: An adaptive testing perspective. arXiv preprint arXiv:2306. 230.11020..1 2023.
[12] Zheng Zhang, Le Wu, Qi Liu, Jiayu Liu, Zhenya Huang, Yu Yin, Yan Zhuang, Weibo Gao, and Enhong Chen. Understanding and improving fairness in cognitive diagnosis. Science China Information Sciences, 152106, 2024.
[13] Daniel O Segall. Computerized adaptive testing. Encyclopedia of social measurement, 1(429- 438):4, 2005.
[14] Mariana Silva, Matthew West, and Craig Zilles. Measuring the score advantage on asynchronous exams an undergraduate es course. In Proceedings of the 5 ACM Technical Symposium on Computer Science Education, pages 873-879, 2020.

---

[15] Dongyang Fan, Celestine Mendler- Dünner, and Martin Jaggi. Collaborative learning via prediction consensus. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages Curran Associates, Inc., 2023.
1988 2009,
[16] Chen Cheng, Gary Cheng, and John C Duchi. Collaboratively learning linear models with structured missing data. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 7529-7540. Curran Associates, Inc., 2023.
[17] Susan E Embretson and Steven Reise. Item response theory. Psychology Press, 2013.
[18] Zhemin Zhu, David Arthur, and Hua-Hua Chang. A new person-fit method based on machine learning in edm in education. Brilish Journal of Mathematical and Statistical Psychology, (5(3):61-637, 2022.
[19] Duc Nguyen and Anderson Ye Zhang. A spectral approach to item response theory. In Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 3881 8-38830. Curran Associates, Inc., 2022.
[20] Peter J. Green. Reversible jump markov chain monte carlo computation and bayesian model determination. Biometrika, 82:711-732, 1995.
[21] Christophe Andrieu and Johannes Thoms. A tutorial on adaptive memc. Statistics and computing, 18:343-373, 2008.
[22] Sebastian Ruder. An overview of gradient descent optimization algorithms. ArXiv, abs/1609.04777, 2016.
[23] Yan Zhuang, Qi Liu, Zhenya Huang, Zhi Li, Binbin Jin, Haoyang Bi, Enhong Chen, and Shijin Wang. A robust computerized adaptive testing approach in educational question retrieval. In Enrique Amigó, Pablo Castells, Julio Gonzalo. Ben Carterette. J. Shane Culpepper, and Gabriella Kazai, editors, SIGIR '22 The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 15, 2022, pages 116-426. ACM, 2022.
[24] Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang. Yu Yin, Shijin Wang, and Yu Su. Neuraled: a general framework for cognitive diagnosis. IEEE Transactions on Knowledge and Data Engineering, 2022.
[25] Haiping Ma, Manwei Li, Le Wu, Haifeng Zhang, Yunbo Cao, Xingyi Zhang, and Xuemin Zhao. Knowledge-sensee cognitive diagnosis for intelligent education platforms. Proceedings of the 1tt ACM International Conference on Information & Knowledge Management, 2022.
[26] Haiping Ma, Jinwei Zhu, Shangshang Yang. Qi Liu, Haifeng Zhang, Xingyi Zhang, Yunbo Cao, and Xuemin Zhao. A prerequisite attention model for knowledge proficiency diagnosis of students. Proceedings of the 3 ACM International Conference on Information & Knowledge Management, 2022.
[27] Junhao Shen, Hong Qian, Wei Zhang, and Aimin Zhou. Symbolic cognitive diagnosis via hybrid optimization for intelligent education systems. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, pages 14928-14936. Vancouver, Canada, 2024.
[28] Shuo Liu, Junhao Shen, Hong Qian, and Aimin Zhou. Inductive cognitive diagnosis for fast student learning in web-based online intelligent education systems. In Proceedings of the ACM Web Conference 2024, Singapore, Singapore, 2024.
[29] Lawrence M Rudner. An examination decision-theory adaptive testing procedures. annual meeting of the American Educational Research Association. 2002.
[30] Wim van der Linden. Bayesian item selection criteria for adaptive testing. Psychometrika, 63(2):201-216, 1998.
12

---

[31] Wim JJ Veerkamp and Martijn PF Berger. Some new item selection criteria for adaptive testing. Journal of Educational and Behavioral Statistics, 22(2):003-226, 1997.
[32] Frederic M. LoFrd. Applications of item response theory to practical testing problems. 1980.
[33] Hua-Hua Chang and Zhiliang Ying. A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20(3):213-229, 1996.
[34] Haoyang Bi, Haiping Ma, Zhenya Huang, Yu Yin, Qi Liu, Enhong Chen, Yu Su, and Shijin Wang. Quality meets diversity: A model-agnostie framework for computerized adaptive testing. In 2020 IEEE International Conference on Data Mining (ICDM), pages 42-51. IEEE, 2020.
[35] Aritra Ghosh and Andrew S. Lan. Bobcat: Bilevel optimization- computerized adaptive testing. ArXiv, 0bs//100800336, 2021.
[36] Yan Zhuang, Qi Liu, Zhenya Huang, Zhi Li, Shuanghong Shen, and Haiping Ma. Fully adaptive framework: Neural computerized adaptive testing for online education. Proceedings of the AAAI Conference on Artificial Intelligence, 36(4):47344477 Jun. 2022.
[37] Burr Settles. Active learning literature survey. 2009.
[38] Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14:5149-5169 2020.
[39] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Trans. Neural Networks, 9:1054-1054 1054, 1998.
[40] Yan Zhuang, Qi Liu, GuanHao Zhao, Zhenya Huang, Weizhe Huang, Zachary Pardos, Enhong Chen, Jinze Wu, and Xin Li. A bounded ability estimation for computerized adaptive testing. Advances in Neural Information Processing Systems, 36, 2023.
[41] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv: Machine Learning, 2017.
[42] Hua-Hua Chang. Psychometrics behind computerized adaptive testing. Psychometrika, 80(1):1-
20, 2015.
[43] Zheng Zhang, Qi Liu, Hao Jiang, Fei Wang, Yan Zhuang, Le Wu, Weibo Gao, and Enhong Chen. Fairlisa: Fair user modeling with limited sensitive attributes information. In Thirth- Conference on Neural Information Processing Systems.
[44] Xigui Yang. A historical review of collaborative learning and cooperative learning. TechTrends, 67(4))718-728, 2023.
[45] Suhrid Balakrishnan and Sumit Chopra. Collaborative ranking. In Proceedings of the fifth ACM international conference Web search and data mining, pages 143-152, 2012.
[46] Pranab Kumar Sen. Estimates of the regression coefficient based on kendall's tau. Journal of the American Statistical Association, 63:1379-1388, 1968.
[47] Yuting Zhang, Ying Sun, Fuzhen Zhuang, Yongehun Zhu, Zhulin An, and Yongjun Xu. Triple dual learning for opinion- based explainable recommendation, ACM Transactions on Information Systems, 42(3):1-27, 2023.
[48] Yining Wang, Yuanzhi Li, Wei Chen, L Wang, D Li, Weiyi He, Chen T.-Y, and Liu. A theoretical analysis of ndeg ranking measures. 2013.
[49] Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions Knowledge and Data Engineering, 33(1):100-11) 2019.
[50] Zichao Wang, Angus Lamb, Evgeny Saveliev, Pashmina Cameron, Yordan Zaykov, José Miguel Hernández-Loobtto Richard E Turner, Richard G Baraniuk, Craig Barton, Simon Peyton Jones, Simon Woodhead, and Cheng Zhang. Diagnostic questions: The neurips 2020 education challenge. arXiv preprint ar arXiv:2007.12061 2020.
13

---

[51] Haw-Shiuan Chang, Hwai-Jung Hsu, and Kuan- Ta Chen. Modeling exercise relationships in e-learning: unified approach. In Educational Data Mining, 2015.
[52] Jin Shi-gu. Application of lagrange mean value theorem. 2014.
[53] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding, pages 409-426, 1994.
[54] Shai Shalev-Shwarzz and Shai Ben-David. Understanding machine learning from theory algorithms. 2014.
[55] Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie- Yan Liu. A theoretical analysis of ndeg type ranking measures. In Conference on learning theory, pages 24.54. PMLR, 2013.
[56] Hamed Valizadegan, Rong Jin, Ruofei Zhang, and Jianchang Mao. Learning to rank by optimizing ndeg measure. Advances in neural information processing systems, 22, 2009.
[57] Jun Hu and Ping Li. Collaborative multi-objective ranking. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 1363-1372, 2018.
14

---

A Proofs of Lemma
Lemma 1. Given two students, whose responses on S([S) T) are y1, 92. yr and 91. 02. Dr. their testing abilities on S are ,",0" 0", which are estimated by IRT with param- eters a,, a,,b, by We can prove that if "0" -0") > 0, then (Ei= a.(y: - 7.) > 0, vice versa.
Proof. Since the abilities of t" and 0Th are the maximum likelihood estimation of IRT in Formula 2, they meet the following conditions:
1l L
Σ (PPP((( vi) Σ
T - - =
a:
00
1=1
According to the Lagrange mean value theorem [52]. the following derivation can be derived:
T
Σ a(ui - 0n) = Σ -P)(')
P(() - 8²).
Since ((( = a, Pi(G)) - P(G:)) and P.(G)) < 1, it implies that:
Σ a.ly - ū.) = Σ a P())(1 -F - )²
Due to Σ/- a; - 0.) and TT_ Hr shared positivity or negativity:
T
Σ all ū.) > 0 AT g >0
B Proofs of Theorem
Theorem 1. Given two students A and B, their relationship with collaborative students are T2, ... Faa; P1, f2,.... "M, r, (0, 1} indicating whether student A outperforms collaborative student in a given test. Assuming the probability that student A outperforms the collaborative stu- dents is P(r; 1) Çi and student B outperforms the collaborative students is P(f: = 1) Then the following conclusion can be drawn:
If) >
2((1-Ca)² collaborative students provided, the prediction of ranking consistency will be least 5.
In
(2) When the number of test questions is small, the assessment of the ranking relationship between the tested students and collaborative students may yield inaccurate results. Assuming error
In probability of p E (0, 0.,), we can still derive that if M > 21-2pj collaborative students provided, the prediction of ranking consistency will be least 1 8.
Proof. Assuming that the ranking abilities of two students are 07 87. Without loss of generality, we suppose Si > (2. We define the random variable X, as the relationship between two students' ranking, where X; F2 Fz.
Suppose, we have M collaborative students, and define X 1 DM X₁. Obviously,
B 0 = =X
Σ (1-
((1-(a) = (1-2p)((i--a))
M
According to the Hoeffding's 's inequality
53 54 we have:
P(O < < = P(O /5 E
03
(G- -()()))a)))) cxp(-2M((() <
Setting 8 = exp( - 21 - 2C) 22
we have when M is larger than 21-220
In
P(O < 0g < 6, which means the prediction error is small than 8.
15

---

C Implementation Details of CCAT
Due to the incomplete information in the test, we also made the following approximations:
(1) Approximate Collaborative Students. Since there no collaborative student in the real world assumed, P((0)) to supplement the
who has completely completed all the questions answer for question i, which means:
1
V = 1
0
y 0 yf = None
(11)
Po(O')
Based this, if yf is not provided, [if 1) should be replaced to P.(0)) and [tf 0)
should be replaced to 1 - Pi()%) in question selection part, and in ability estimation part, (Eies a; (y - yf) > 0) can be approximated as a (Eies ar(ys - vfr)) and can be applied regardless of whether there is P.(02) of supplementation not.
(2) Approximate Outperform Probability. In our method, we need to select questions by using the information on whether tested students outperform the collaborative students [(0" >> 02). However, the ground truth 0* and 0 is unknown when testing. So we proposed using 0e and 0 to approximate O" and 0 for each step L. Considering that there is certain error between time and the actual state, we use the sigmoid function a(0' 6e)) to approximate I( > 8²), which means the more tested students are ahead of collaborative students at step the higher the likelihood that their true abilities surpass those of the collaborative students. Through the above approximation, the question selection algorithm can be rewritten follows:
4 arg
max R(g,, 0" > 0-10--1) + R(g:, 0" < 1) QIERIS
(12)
where R(q;, 0" > 0- = a. P(y; = 010") Eyrec Vio (Eres a:(y: - -((i) and R(gi, 0" < 0210--1) = P,y = 110t) Eyreec(1 - v') (Eies a, / w.)]]. P(y: = 010"), P(y 110*) can be calculated by IRT method and C' is the set of collaborative students.
D Details of Experiments
D.1 Statistics of the datasets.
Table 3: Statistics of the datasets
Dataset
NIPS-EDU
JUNYT
#Students
4,914
8,852
#Questions
900
702
#Response logs
1,382,173
801,270
#Response logs per student
281.27
90.52
#Response logs per question
1,535.75
1,141.41
D.2 Detailed Evaluation Method
Statistic for Ranking Consistency. For CAT tasks, there are many methods that are sensitive to the initial abilities of students, including Random, FSI, KLI, MAAT, BECAT, and CCAT proposed in this article. However, data-driven methods such as BOBCAT and NCAT are often insensitive to the initial abilities of students. Therefore, this study randomly initialized the initial abilities of students 5 times and counted the mean and standard deviation of the ranking consistency of each question selection algorithm, as shown in Tables 4 and 5 can be seen that although the current abilities of students are used in the selection process, CCAT is almost not affected by the initialization of student abilities. This indicates that CCAT not only performs well in ranking consistency but also is more stable compared other strategics.
16

---

Table 4: The Detail Performance of different question selection algorithms on NIPS-EDU Algorithm X-C means use algorithm X for question selection but use collaborative ability estimation proposed in CCAT as the testing result instead of the abilities estimated by IRT. The bold font represents significant improvement in statisties compared the baseline.
(a) Intra-class Ranking Consistency Performance on IRT estimated by GD
Datasct
NIPS-EDU
Step
5
10
15
20
Random
0.7041
0.007
0.7434 + 0.005
0.7680
0.007
0.7856
0.004
FSI
0.7236
0.004
0.7889 + 0.003
0.8192
0.002
0.8321
0.002
KLI
0.7328
0.004
0.7868 + 0.005
0.8142
0.003
0.8316 = 0.002
MAAT
0.6725
0.001
0.7095 = 0.002
0.7359
0.002
0.7535 = 0.001
BECAT
0.7087 0.007
0.7542 + 0.004
0.7802
0.005
0.7957
0.005
CCAT (w/o C)
0.7320 0.002
0.7870 + 0.002
0.8177
0.002
0.8279
0.002
Random-C
0.6988 + 0.008
0.7444 = 0.004
0.7715 + 0.005
0.7909
0.004
FSI-C
0.7340 + 0.005
0.8031 + 0.003
0.8339 + 0.002
0.8546
0.001
KLI-C
0.7399 + 0.003
0.7982 + 0.003
0.8304 + 0.002
0.8509 = 0.001
MAAT-C
0.6689 + 0.002
0.7175 - 0.003
0.7475 + 0.002
0.7603 0.002
BECAT-C
0.7292 + 0.006
0.7959 = 0.003
0.8279 0.002
0.8438 0.007
CCAT
0.7533 0.000
0.8081 + 0.001
0.8364 0.000
0.8543 0.000
(b) Intra-class Ranking Consistency Performance on IRT estimated by MCMC
Datasct
NIPS-EDU
Step
5
10
15
20
Random
0.7411
0.005
0.8061 + 0.005
0.8348
0.004
0.8540
0.005
FSI
0.7912 1: 0.005
0.8570 = 0.003
0.8846
0.001
0.8975
0.001
KLI
0.7821 0.005
0.8532 = 0.003
0.8804
+ 0.001
0.8965
0.002
MAAT
0.6762 0.005
0.8083 = 0.007
0.8588
0.004
0.8843
0.002
BECAT
0.7685 0.005
0.8441 + 0.002
0.8766
0.002
0.8958
0.002
CCAT (w/o
0.7982 0.001
0.8561 = 0.001
0.8832
0.001
0.8955
0.000
Random-C
0.7531 0.004
0.8084 + 0.005
0.8363
0.004
0.8547 = 0.004
FSI-C
0.7933 0.005
0.8573 + 0.003
0.8848
0.001
0.8977 = 0.001
KLI-C
0.7839 0.006
0.8530 + 0.003
0,8805
0.001
0.8966
0.002
MAAT-C
0.6909 0.005
0.8090 + 0.003
0.8595
0.004
0.8848
0.002
BECAT-C
0.7680 0.004
0.8449 + 0.001
0,8771
0.002
0.8961
0.001
CCAT
0.8149 + 0.002
0.8635 + 0.001
0.8851 0.001
0.8969 = 0.000
(c) Inter-class Ranking Consistency Performance IRT estimated by MCMC
Dataset
NIPS-EDU
Step
5
10
15
20
Random
0.7798 +: 0.003
0.8325 1 0.003
0.8590 + 0.002
0.8760
0.002
FSI
0.8258 + 0.003
0.8785 + 0.002
0.9013 + 0.001
0.9126 0.001
KLI
0.8195 + 0.003
0.8758 + 0.002
0.8985
0.001
0.9119 0.001
MAAT
0.7242 = 0.004
0.8373 = 0.002
0.8807
0.002
0.9023
0.001
BECAT
0,8045 1 0.003
0.8676 1 0.001
0.8948 1 0.001
0.9104
0.001
CCAT
0.8476 + 0.001
0.8839 + 0.000
0.9013 + 0.000
0,9116 0.000
17

---

Table 5: The Detail Performance of different question selection algorithms on JUNYL. Algorithm X-C means use algorithm X for question selection but collaborative ability estimation proposed in CCAT as the testing result instead of the abilities estimated by IRT. The bold font represents a significant improvement in statisties compared the baseline.
(a) Intra-class Ranking Consistency Performance IRT estimated by GD
Dataset
JUNYI
Step
5
10
15
20
Random
0.6875 1 0.008
0.7350 + 0.005
0.7671
0.003
0.7914
0.003
FSI
0.7639 + 0.004
0,8284 + 0.003
0.8586
0.002
0.8740
0.002
KLI
0.7748 + 0.002
0.8340 0.001
0.8623
0.001
0.8817 0.001
MAAT
0.6908 + 0.000
0.7465 + 0.000
0.7817
0.000
0.8113 0.000
BECAT
0.7248 + 0.003
0,7712 0.003
0.7920
0.8030
0.003
CCAT (w/o
0.8026 + 0.001
0.8560 + 0.001
0.8819 = 0.000
0.8978 0.000
C)
Random-C
0.6862 + 0.008
0.7383 + 0.007
0.7734 0.004
0.7979
0.003
FSI-C
0.7736 + 0.005
0.8313 0.003
0.8623
0.002
0.8768
0.002
KLI-C
0.7813 1 0.001
0.8367 1 0.002
0.8671
0.001
0.8847
0.002
MAAT-C
0.7040 + 0.000
0.7822 + 0.000
0.8222
0.000
0.8464
0.000
BECAT-C
0.7603 + 0.003
0.8295 + 0.002
0.8603
0.002
0.8769 0.001
CCAT
0.8092 + 0.001
0.8647 + 0.000
0.8911 = 0.000
0.9066 + 0.000
(b) Intra-class Ranking Consistency Performance on IRT estimated by MCMC
Dataset
JUNYI
Step
10
Random
0.6527 + 0.002
0.7759 + 0.005
0.8292 + 0.002
0.8600 + 0.002
FSI
0.8212 + 0.003
0.8820 + 0.001
0.9092 + 0.001
0.9257
0.000
KLI
0.8124 + 0.003
0.8795 + 0.002
0.9082 0.001
0.9244
0.001
MAAT
0.7404 + 0.008
0.8506 + 0.001
0.8925
0.001
0.9161
0.001
BECAT
0.7857 + 0.002
0.8699 + 0.001
0.9031
0.001
0.9225
0.000
CCAT (w/o
0.8190 0.001
0.8823 + 0.001
0.9098
0.000
0.9277
0.000
Random-C
0,7511 + 0.005
0.8074 + 0.005
0.8429 + 0.002
0.8667 + 0.002
FSI-C
0,8226 + 0.002
0.8820 + 0.001
0.9090 = 0.001
0.9251 + 0.001
KLI-C
0.8146 + 0.002
0.8795 + 0.001
0.9079 = 0.001
0.9237 + 0.001
MAAT-C
0.7441 + 0.003
0.8512 + 0.001
0.8926 + 0.001
0.9157 0.001
BECAT-C
0.7932 + 0.002
0.8706 + 0.001
0.9022
0.001
0.9217 + 0.000
CCAT
0.8448 + 0.007
0.8875 + 0.000
0.9100 0.000
0.9273 0.000
(c) Inter-class Ranking Consistency Performance on IRT estimated by MCMC
Dataset
JUNYI
Step
15
20
Random
0.7651 +: 0.003
0.8298 + 0.004
0.8648 +: 0.001
0.8865
0.001
FSI
0,8575 + 0.001
0.9050 +: 0.000
0.9249
: 0.000
0.9363
0.000
KLI
0,8502 + 0.002
0.9028 0.000
0.9240 0.000
0.9353
0.000
MAAT
0.7830 + 0.002
0.8767 0.000
0.9069
+ 0.001
0.9249
0.001
BECAT
0.8287 0.001
0.8961
0.001
0.9204
+ 0.001
0.9341
0.000
CCAT
0.8736 0.001
0.9082 0.000
0.9255 +0.000
0.9373 0.000
18

---

RANDOM
MPAT
NCAT
BECAT
CCAT
NDCGBA60
0.905
NDCG
6.97
0.990
0066
0.915
006
01531
0194
09905
0.95
0.99
002
Test Step
Test Step
Figure The performance on NDCG of different question selection algorithms on the dataset NIPS-EDU for the IRT model estimated by MCMC method.
as an important metric for ranking problems in recommendation systems, is also used reference metric for CAT ranking problems. Specifically, at each moment of the test, CAT provides students with an ability estimation, while selection exams can be seen as recall of students. Specifically, we assume that 60% of students will be admitted or eliminated, which means recalling the top 60% of students (NDCG@ A60%) and recalling the bottom 60% of students (NDCG @ D60%). From Figure 5] it can be seen that CCAT, CAT method proposed for ranking problems, also performs outstandingly in recall tasks, indicating that the CCAT method can provide more fair selection for selective exams.
NDCG. NDCG
(a) -0.23 -0.39
(b) 0.36 0.12
(c) -0.11 0.27
-0.72 -0.93
(c) -0.37 -1.03
(f) -0.36 vs -0.58
(g) 0.32 vs -0.65
(h) -0.05 vs -0.84
(i) -0.43 -0.82
( 0.43 -0.05
Figure 6: Rating Chart for different students pair estimated by collaborative students in NIPS-EDU.
19

---

(a) 0.91 vs 0.67
(b) -0.91 vs -1.53
(c) 0.58 vs -0.07
(d) -0.99 vs -1.43
(c) 0.22 -0.71
(1) 0.04 vs -0.45
(g) -0.76 -1.37
(h) 0.69 -0.29
(i) 0.86 vs -0.08
(i) 0.69 vs -0.24
Figure 7: Rating Chart for different students pair estimated by collaborative students in JUNYI.
Case Study Supplement. Figures fillsstaee the responses of collaborative students within each pair. Each point's 's coordinates denote the comparative performance of the student pairs relative to individual collaborative students. The intensity of the point's ' color corresponds to the response time, with darker hues indicating later responses.
can be seen that CCAT determines the ranking students each moment by comparing the number of collaborative students in the upper and lower triangles. The light-colored points the figure are mainly distributed the center, while the dark ones are distributed around, indicating that as the number of test questions increases, each collaborative student's 's judgment of the two students gradually changes from vague to clear. can be found that the collaborative ability estimation method is essentially collaborative student voting for tested students, and the collaborative student union in the upper left lower right corner of the figure will ultimately distinguish the two students.
Based on Figure
20

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We briefly elaborated the abstract that this article investigates the issue of ranking in CAT, and described the main contributions of this study in the last paragraph of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
21

---

Justification: In the experimental section and appendix section D, we discussed the limita- tions of our method under long test durations and the disadvantages of the GD method in ranking consistency problems. Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiscless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: In sections and of the appendix of this article, we provide the proof process for Lemmal and Theorem I, respectively. In section C, we present all the hypotheses used to construct collaborative students and help with question selection.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
22

---

We have introduced the dataset used in the experimental section and appendix section D of the article, and included the complete code and some data the supplemental materials to reproduce the results of the article. Guidelines:
Justification:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code for the experimental part of this article is included in the supplemental materials, and the datasets used our experiments are public datasets
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.cm cc/
Souidess/ooeSuumissst.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
23

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: This article introduces the method of data splits in appendix section D, and the implementation method of this article has been detailed in section of the main text and appendix section C. In addition, this article is based on theoretical derivation, there no technical details such as hyperparameters, optimizers, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have provided detailed information on the experiment in appendix section D and provided statistical information on the experiment, such as the number of tests and the variance obtained.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma emror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
24

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiment??
Answer: [Yes]
Justification: The resources used in the experiment are introduced in appendix section of the article.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have read and promise that our research conform with NeurolPS ethical standards in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: fact, the fundamental purpose of proposing the issue of CAT for ranking in our work is to address the issue of educational equity.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
25

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The code used the article all original and the dataset used open-source, which can be used after being referenced.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pwppppaipphheeeecc twooee. com/ datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
This document provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
26

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "1h8NrZxUjL", "reviewer": "Reviewer_74zz", "summary": "The study deals with CAT( Computer Adaptive Testing) via Collaborative Ranking.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed CCAT algorithm demonstrates superior performance in ranking 259 consistency across two public datasets. Particularly, CCAT shows more significant improvement 260 when fewer questions are tested, outperforming other methods.", "weaknesses": "The study does not discuss in sufficient detail the following: limitations, generalization possibility and the application of the algorithm as part of instructional design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:** The study does not discuss in sufficient detail the following: limitations, generalization possibility and the application of the algorithm as part of instructional design.\n\nThank you for your valuable feedback. We apologize that we have shown in the experimental and appendix sections, rather than divided it into paragraphs for description and we will supplement this section in future versions of our paper:\n\nLimitations:\n\nAs mentioned in section Experiment,  we have stated limitations in line 262 in this version that CCAT may not perform as well on long test sequences as methods that directly optimize capabilities. To make it clearer, we will separate statements of limitations into a new section in the new version of this paper.\n\nGeneralization Possibility:\n\nOur method performs particularly well when the number of CAT test questions is small. Additionally, since CAT is inherently a cold start problem, our research may be applicable to other fields facing similar challenges. For example, personalized recommendation systems, healthcare diagnostics, and user behavior prediction are areas where cold start problems are prevalent, and our approach could potentially be generalized to improve performance in these fields.\n\nApplication as Part of Instructional Design:\n\nIn instructional design, the CCAT algorithm can be integrated to create adaptive learning paths tailored to individual student abilities. By dynamically adjusting the difficulty and selection of questions based on ongoing assessments, educators can provide a more personalized and effective learning experience. However, careful consideration must be given to the practical implementation and potential limitations discussed above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YuPuzdsuP3", "reviewer": "Reviewer_piWg", "summary": "This paper proposes a new perspective on Computerized Adaptive Testing (CAT) by framing it as a task of ranking students. The authors define CAT as a ranking problem and present a feasible optimization algorithm to address this. Extensive experimental results demonstrate that this method significantly improves the consistency of student ranking scores compared to the baseline system.", "questions": "See above.", "limitations": "The paper and its supplementary materials clearly illustrate its limitations. Due to the need for collaborative students to be stored, the algorithm’s time and space complexity should be more thoroughly discussed. Providing detailed analyses and potential optimization strategies could strengthen the paper.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper explores a previously overlooked issue in CAT and human assessment domain—student ranking consistency. It redefines the CAT task within this context. The solution proposed is interesting and interpretable, aligning well with real educational scenarios.\n2. In terms of technical implementation, this paper utilizes existing students as ranking anchors to enhance selection and evaluation methods. It provides a theoretical basis, demonstrating that the algorithm can reduce ranking consistency errors to an acceptable level.\n3. Experiments on real-world datasets have shown that the proposed method improves ranking consistency by an average of 5% compared to baseline selection methods.", "weaknesses": "1. This paper has rich and convincing experiments, but I want to know why Table 1 only shows the results of BOBCAT in the Table 1(a). Furthermore, this paper does not explain why BOBCAT has such a significant difference in performance on this task.\n2. In Table 1 (a), NCAT performs best on the NIPS-EDU dataset at T=5, but this result is not  discussed in this paper.\n3. This paper mainly analyzes the question selection and estimation of the CCAT method. In fact, according to my understanding, the construction of collaborative students is long-term and complex in real educational scenarios. Is there a method to ensure the effective construction of collaborative students?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments! To address your concerns, below we prudently justify the details of our proposed method and experiments.\n\n> **Q1:** This paper has rich and convincing experiments, but I want to know why Table 1 only shows the results of BOBCAT in the Table 1(a). Furthermore, this paper does not explain why BOBCAT has such a significant difference in performance on this task.\n\nAs a bilevel optimization model, BOBCAT optimizes both the IRT model and the question selection strategy. This means it employs a separate GD model and does not require training an additional IRT model through the MCMC method. Consequently, only one result for BOBCAT is shown in Table 1(a). The significant difference in BOBCAT's performance can be attributed to its two-layer optimization approach, which leads to peak accuracy at specific times (as illustrated in Appendix Figure 4, ACC), but poor performance at other times. Additionally, training the IRT model can lead to instability in estimating real ability, ultimately resulting in poor performance in ranking metrics.\n\n> **Q2:** In Table 1 (a), NCAT performs best on the NIPS-EDU dataset at T=5, but this result is not discussed in this paper.\n\nThe CAT ranking problem we are studying can essentially be seen as an degradation problem for accuracy. When optimizing for accuracy, we may randomly achieve higher or lower ranking consistency. As observed in Table 1, although NCAT performs well on the NIPS-EDU dataset with T=5, its performance on the JUNYI dataset is even worse than that of random selection. This indicates that the NCAT method may not be effective in consistently optimizing ranking problems. We will include a discussion of these observations in the revised version of our paper to provide a comprehensive analysis of NCAT's performance across different datasets.\n\n> **Q3:** This paper mainly analyzes the question selection and estimation of the CCAT method. In fact, according to my understanding, the construction of collaborative students is long-term and complex in real educational scenarios. Is there a method to ensure the effective construction of collaborative students?\n\nCurrently, the selection and construction of collaborative students are relatively simple processes. Generally speaking, in real CAT systems such as GRE, a group of students will take tests in advance to generate test records [1]. If CAT is viewed as a long-term process, while CAT increases students' learning efficiency, it also leads to the sparsification of student data. As data becomes sparser, the effectiveness of CAT and IRT may be impacted. To date, only one study has addressed the bias in IRT data [2]. Therefore, more research is needed to ensure the long-term construction and maintenance of collaborative students.\n\nReference: \n\n[1] Computerized adaptive testing: Theory and practice[M]. Dordrecht: Kluwer Academic, 2000.\n\n[2] Kwon S, Kim S, Lee S, et al. Addressing Selection Bias in Computerized Adaptive Testing: A User-Wise Aggregate Influence Function Approach[C]//Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 2023: 4674-4680.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tBkrnLtwHR", "reviewer": "Reviewer_M6H4", "summary": "The paper proposes an algorithm for performing computerized adaptive testing that handles and accounts for student rankings in the item recommendaiton.", "questions": "- Could the authors describe how their work differs from collaborative filtering approaches where estimation of item difficulty and user ability can be done through previous users and their attempted items? \n\n- It seems strange to me that the collaborative records used for testing are being simultaneously updated at every step t (rf. Algorithm 1). Why do we need to update the student ranks online? \n\n- Theorem 1 seems more like a statement about the collaborative filtering approach in estimating item difficulty and user ability, and is not really about estimating the ranking. Could the authors cleave the effects of items and users on the IRT estimation from the ranking estimation in their Theorem? \n\n- While Figure 3 points to areas of improvements, I also noticed areas where their method does worse: e.g., on the second row of the NIPS-EDU. Could the aggregate difference be reported instead of the heat map visualization?", "limitations": "The authors do not provide a separate Limitations section in their paper. While the authors state they provided the limitations in Appendix D and experiments, these seem to be more like \"findings\" from empirical observations than limitations of their work/framing/approach.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- I like the idea of incorporating additional information in the collaborative filtering approach (while I don't quite understand why you need to intermittently rank students during CAT)", "weaknesses": "My main critique of the paper is the work's motivation for connecting ranking and CAT: Ranking seems like a completely separate task from CAT, and would happen after CAT. The paper's introduction would benefit from a concrete example where online updates in collaborating filtering is important when administering tests. It seems like there would be issues during testing such as handling the nonstationarity of question difficulties and forcing students to start and complete every question at the same time before proceeding onto the next question. It's not clear why you need intermediate ranks amongst students as they are getting tested.\n\nLooking at Alg 1, it feels like the work should use/estimate student rankings not for the sake of estimation/collaborative filtering, but rather selecting questions that best *differentiate* students -- so distinguish their abilities to further diagnose students. But I'm curious what the authors originally had in mind with connecting online ranking estimations with CAT, because currently... I don't quite see how the algorithm assumptions can hold in the real world (see paragraph above).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on our manuscript. We sincerely appreciate your effort in evaluating our work. Below, we address each of your comments in detail:\n\n> **Q1**: My main critique of the paper is the work's motivation for connecting ranking and CAT: Ranking seems like a completely separate task from CAT, and would happen after CAT.\n\nYou are right. Ranking typically occurs after the CAT process, but it is inherently interconnected with CAT and is also influenced by CAT . Existing CAT methods focus solely on the accuracy of abilities (Figure 4), overlooking the importance of ranking in CAT (Figure 1). This oversight can lead to potential biases and inequities in selection tests. Our research aims to address this gap by enhancing the testing process.\n\n> **Q2:** Could the authors describe how their work differs from collaborative filtering approaches?\n\nCF and CCAT share some underlying principles, but they are fundamentally different in their applications and objectives. CF is a recommendation technique that estimates item preferences based on previous users' behaviors, while CCAT adapts the test items based on the examinee's ability level, which is estimated dynamically during the test.\n\nMoreover, CF aim to suggest items based on user preferences (**ranking items**), whereas CAT necessitating precise estimation of students' abilities with minimal interaction (**ranking users**). In CF, the item ranking of a user is fixed, but in CAT, selected questions affect the ability evaluation through the answers, impacting the ranking. This variability makes CAT tasks more challenging than typical CF problems.\n\n> **Q3:** It seems like there would be issues during testing such as handling the nonstationarity of question difficulties and forcing students to start and complete every question at the same time.\n\nWe would like to clarify that in traditional CAT models, the difficulty of questions is pretrained before testing and is fixed throughout the testing process. This assumption is based on the fact that the difficulty of a question is determined by its inherent attributes [1].\n\nMeanwhile, students do not need to wait for other students before proceeding onto the next one. Each student, upon completing a question, will compare their performance with a group of \"collaborative students\", which are defined as students who have already answered questions in question bank in Definition 1. Generally speaking, in real CAT systems such as GRE, a group of students will take tests in advance to generate test records [2], which indicates that our define and hypothesis is reasonable. \n\n> **Q4:** Why are intermediate ranks needed amongst students as they are getting tested?\n\nWe do not dynamically update students' rankings during the CAT process. We only update students' abilities after each question (like other CAT) and use the comparison results between the tested students and collaborative students for rankings after testing.\n\n> **Q5:** Why are collaborative records used for testing being simultaneously updated at every step t (ref. Algorithm 1)?\n\nCollaborative records are not updated every round. Instead, we extract useful records from the collaborative records corresponding to the questions that the current student has answered.\n\n> **Q6:** Could the authors cleave the effects of items and users on the IRT estimation from the ranking estimation in their Theorem?\n\nYes. Collaborative students are only a prerequisite of Theorem 1 rather than what it focuses on. Theorem 1 can be understood as a process where collaborative students vote for Student A and Student B (A is better than B). It claims that as long as there are enough collaborative students, A's votes will surpass B's votes. This explains why we utilize collaborative students to vote for ranking the students being tested.\n\n> **Q7**: Could the aggregate difference be reported instead of the heat map visualization?\n\nYes. We originally hoped to visually demonstrate the advantages of CCAT's results compared to IRT's results through a heatmap. Below is the aggregate difference representing the average improvement of each student in 20 steps using CCAT compared to IRT (positive value means improve and negative value means decline):\n\n| Aggregate Difference | Average | 1     | 2     | 3    | 4     | 5    | 6    | 7     | 8    | 9    | 10   |\n| -------------------- | ------- | ----- | ----- | ---- | ----- | ---- | ---- | ----- | ---- | ---- | ---- |\n| NIPS-EDU             | 0.18$\\uparrow$    | -0.15 | 0.18  | 0.22 | 0.19  | 0.13 | 0.24 | 0.22  | 0.49 | 0.06 | 0.16 |\n| JUNYI                | 0.004$\\uparrow$   | -0.08 | -0.09 | 0.11 | -0.12 | 0.26 | 0.09 | -0.21 | 0.00 | 0.07 | 0.01 |\n\nOur method may not achieve better ranking relationships among all students, but from the table and various experimental results, our method can indeed improve the overall ranking consistency of students.\n\n> **Q8:** While the authors state they provided the limitations in Appendix D and experiments, these seem to be more like \"findings\" from empirical observations than limitations of their work.\n\nSorry for the confusion. As mentioned in the Experiment section, we have stated limitations in line 262 that CCAT may not perform as well on long test sequences as methods that directly optimize capabilities. To make it clearer, we will separate statements of limitations into a new section in the new version of this paper.\n\nReference:\n\n[1] Wainer H, Dorans N J, Flaugher R, et al. Computerized adaptive testing: A primer[M]. Routledge, 2000.\n\n[2] Computerized adaptive testing: Theory and practice[M]. Dordrecht: Kluwer Academic, 2000.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vL4ApxUK9L", "reviewer": "Reviewer_cZqj", "summary": "This paper addresses a real-world problem in AI education: improving the accuracy of student rankings by selecting different questions during the exam process. It proposes a question selection method based on collaborative students and provides theoretical guarantees. The experimental results have demonstrated the effectiveness of its method in ranking consistency.", "questions": "See Weaknesses part", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. To my knowledge, the perspective of this paper is novel. It starts from a real exam scenario and defines the CAT problem as a ranking problem, which has not been solved in previous CAT research. This means that based on this work, numerous ranking methods may be incorporated into CAT problems.\n\n2. The paper demonstrates the method's superior ranking accuracy through experiments on real-world datasets. The approach exhibits general applicability across CAT systems estimated by IRT or GD.\n\n3. The logic of this paper is clear, and the supplement materials include all theoretical proofs and several additional experiments, making this paper easy to follow.", "weaknesses": "1. Can you clarify the detail of \\theta^T_c? Furthermore, since the true abilities of \"collaborative students\" are known, why use the abilities of collaborative students at T-moment instead of their true abilities.\n\n2. The method proposed in the paper seems to have value primarily for educational research. While this is an important domain, the paper could benefit from discussing the potential applicability in other fields or providing more insights into how the proposed approach could be generalized beyond the education sector. \n\n3. Ranking is a common issue in recommendation systems. After defining CAT tasks as ranking problems in this paper, what are the similarities and differences between CAT and recommendation tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback!  Regarding the questions you raised, we have carefully considered each point and have made following responses:\n\n> **Q1:** Can you clarify the detail of $\\theta^T_c$? Furthermore, since the true abilities of \"collaborative students\" are known, why use the abilities of collaborative students at T-moment instead of their true abilities.\n\nThe parameter $\\theta^T_c$ represents the ability assessment results obtained by collaborative students who have answered the same T questions as the students being tested. This approach ensures that the collaborative students' ability assessments are based on a comparable set of questions, providing a more accurate basis for scoring the tested students.\n\nUsing the true abilities of collaborative students ($\\theta^*_c$) for each student would mean that the collaborative students would lose their sensitivity to the question selection process. This would hinder the optimization of ranking through question selection, as the adaptive nature of the test would be compromised.\n\n> **Q2:** The method proposed in the paper seems to have value primarily for educational research. While this is an important domain, the paper could benefit from discussing the potential applicability in other fields or providing more insights into how the proposed approach could be generalized beyond the education sector.\n\nOur method performs particularly well when the number of CAT test questions is small, addressing the cold start problem inherent in CAT. This characteristic suggests that our research may be applicable to cold start problems in other fields as well. For instance, our approach could be valuable in personalized recommendation systems, where limited initial data is a common challenge. Additionally, it could be useful in any domain that requires adaptive testing or assessment with sparse initial data.\n\n> **Q3:** Ranking is a common issue in recommendation systems. After defining CAT tasks as ranking problems in this paper, what are the similarities and differences between CAT and recommendation tasks?\n\nCF and CCAT share some underlying principles, but they are fundamentally different in their applications and objectives. CF is a recommendation technique that estimates item preferences based on previous users' behaviors, while CCAT adapts the test items based on the examinee's ability level, which is estimated dynamically during the test.\n\nMoreover,CF aim to suggest items based on user preferences (**ranking items**), whereas CAT necessitating precise estimation of students' abilities with minimal interaction (**ranking users**). In CF, the item ranking of a user is fixed, but in CAT, selected questions affect the ability evaluation through the answers, impacting the ranking. This variability makes CAT tasks more challenging than typical CF problems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p7yj8cVbv2", "reviewer": "Reviewer_gvzk", "summary": "In this paper, the authors first discovered the inconsistency issue in existing Computerized Adaptive Testing (CAT) solutions for estimating the latent abilities of students, that is the higher accuracy of ability estimation (with a lower MSE) does not necessarily guarantee the ranking consistency of students’ abilities. Based on this discovery, the authors then proposed a novel Computerized Adaptive Testing framework CCAT inspired by collaborative ranking. Specifically, CCAT uses collaborative students as anchors to assist in test-question selection and estimation in testing. More importantly, the authors provide a theoretical analysis of the upper bound of ranking consistency error for collaborative students, which verifies that with an adequate number of collaborative students, the ranking consistency error can be reduced to an acceptable level. Through experiments on two real-world datasets, the authors demonstrated that CCAT can achieve the best ranking consistency.", "questions": "1. How can we get the collaborative students? Are these collaborative students the same as all the testing students?\n\n2. From Equation (5), the response of the students belongs to 0 or 1, which means the student's responses studied in this paper are either right or wrong, what if the rating of the student responses has more choices (like the values in the range of [0,1])? Does the CCAT solution still work in this scenario?\n\n3. How to determine the parameter T (testing round) in CCAT?", "limitations": "The authors have adequately addressed the limitations", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Computerized Adaptive Testing is a cross-cutting research direction between artificial intelligence and the testing area, with broad applications in the real world. In this paper, the authors first discovered the inconsistency issue in existing Computerized Adaptive Testing (CAT) solutions for estimating the latent abilities of students, and therefore, the research motivation is both original and significant.\n\n2. The proposed Collaborative Computerized Adaptive Testing framework CCAT exploits the idea of collaborative students to address the incomparable test-answering behavior problem of different students. This idea is quite novel and differs from traditional CAT solutions and collaborative ranking solutions.\n\n3. Both the theoretical analysis and the experimental validation seem to be solid and convincing.", "weaknesses": "1. It is not easy to illustrate the main ideas of CAT as it contains both the Ability Estimation Part and the Question Selection Part. Therefore, the readability of this paper can be further improved especially for the readers without any background of CAT.\n\n2. The main idea of the CCAT algorithm can be explained in more detail in the main text. For instance, how can we get the collaborative students?\n\n3. More references about collaborative ranking are recommended to be included and discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on our manuscript. We sincerely appreciate your time and effort in evaluating our work, and we appreciate you for the opportunity to explain and articulate our work.\n\n> **Q1:** It is not easy to illustrate the main ideas of CAT as it contains both the Ability Estimation Part and the Question Selection Part. Therefore, the readability of this paper can be further improved especially for the readers without any background of CAT.\n>\n> **Q2:** The main idea of the CCAT algorithm can be explained in more detail in the main text.\n\nWe appreciate your observation regarding the difficulty in illustrating the main ideas of CAT, particularly as it involves both the Ability Estimation Part and the Question Selection Part. To address this, we will optimize the content of our article to enhance its readability, especially for readers who do not have a background in CAT. We will ensure that our explanations are more accessible and clear, incorporating additional context and examples where necessary.\n\n> **Q3:** How can we get the collaborative students? \n>\n> **Q4:** Are these collaborative students the same as all the testing students?\n\nIn the experimental phase, collaborative students can be constructed by splitting the dataset and supplementing it with the predicted values from the IRT, as detailed in Appendix Section C. We believe that there is no essential difference between these collaborative students and testing students, only that collaborative students have answer records while testing students do not. In real exam scenarios, generally speaking, in real CAT systems such as GRE, a group of students will take tests in advance to generate test records[1]. Moreover, we can also use students who have previously taken exams as collaborative students. \n\n> **Q5:** More references about collaborative ranking are recommended to be included and discussed.\t\n\nWe appreciate the importance of providing a comprehensive context for our work and will include additional references related to collaborative ranking in future versions of our paper. This will help situate our research within the broader academic discourse and offer readers a more thorough understanding of the field.\n\n> **Q6:** From Equation (5), the response of the students belongs to 0 or 1, which means the student's responses studied in this paper are either right or wrong, what if the rating of the student responses has more choices (like the values in the range of [0,1])? Does the CCAT solution still work in this scenario?\n\nIndeed, changing the range of question values implies that the traditional IRT would no longer be applicable, and consequently, the CCAT algorithm cannot be directly used in its current form. However, we believe that our approach remains universal. Specifically, for any given evaluation model, we can still optimize the accuracy of ability ranking among collaborative students by enhancing the question selection process. Ultimately, collaborative student voting can be employed to rank the tested students effectively.\n\n> **Q7:** How to determine the parameter $T$ (testing round) in CCAT?\n\nIn general, for research purposes in CAT problems, the parameter $T$ is often set to fixed values such as 5, 10, 15, or 20. These fixed values provide a standardized way to compare different methods and results. However, in practical application scenarios, the termination of T can be more dynamic. It can be based on specific indicator change amplitudes, such as the ability change value or ranking change value. The testing rounds can be terminated once these changes are less than a predefined threshold, ensuring that the testing adapts to the candidate's performance and achieves optimal efficiency.\n\nReference:\n\n[1] Computerized adaptive testing: Theory and practice[M]. Dordrecht: Kluwer Academic, 2000.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper argues that the ranking among students in tests are more important than the accurate estimation of student abilities, and proposes an algorithm called CCAT to optimize the ranking consistency on students instead of minimizing estimation errors on student abilities like existing work. Experiment results on two datasets show that CCAT can achieve higher ranking consistency than baselines and the improvement is particularly bigger with fewer questions. 

The proposed approach is considered novel and the improvement is considered significant by most reviewers. 

The authors are encouraged to incorporate reviewers’ feedback and suggestions into the final version to further improve the paper.
Decision: Accept (poster)