Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper develops a spectral algorithm for estimating the item parameters in the Rasch item response model.

The authors show that the proposed estimator is consistent, and they provide finite-sample error bounds that are (near-)optimal.

Empirically, the authors show that their approach performs favorably in comparison to existing estimators: Predictive performance is on par or better, and the algorithms typically runs faster than competing approaches.
strengths_and_weaknesses: Strengths:

- The Rasch model is widely used, and the problem addressed by this paper is very relevant to the NeurIPS community.
- The theoretical contributions are comprehensive and address a number of important questions that remain mostly open for other estimators for the Rasch model: consistency, finite-sample error, optimality.
- In practice, the algorithm is simple to understand and to implement, and performs favorably. As such, the paper's contributions also relevant to practitioners.
- Beyond the specific results presented by the author, this paper also provides a bridge between the literature on spectral algorithms for the Bradley-Terry model and the Rasch model. I expect this paper to unlock further research on the topic.
- The experimental evaluation is excellent. The illustration of failure cases (Fig. 1) for various estimators is helpful in understanding the trade-offs that different methods have. The real-data experiments are thorough, and also include datasets and metrics where there is not necessarily a clear advantage for their proposal, giving a balanced picture of the empirical benefits.
- The paper is well-written and easy to follow. Throughout the paper, the authors strike a good balance between building intuition and formalizing their claims.

Altogether, the combination of these strengths make for a significant contribution that will have impact on ML researchers and practioners.

Weaknesses

- Theorems 3.5 and 3.6 appear to be missing something. Clearly setting $\beta^* = 0$ and $T(X) = 0$ achieves zero error (and satisfies the unbiasedness assumption). Perhaps $\beta^*$ needs to be bound, e.g., by taking the supremum over a certain range. Similarly, it is not clear what the expectation is over (I read it to be over $X$ but this could be made explicit).
- There is no discussion about the existence and uniqueness of the estimator, i.e., of the stationary distribution of (3). This needs to be formalized more rigorously. Under what conditions on `X` does the spectral estimate exist? How do Algorithms 1 & 2 handle cases where this condition is not satisfied?
- The similarities and differences to existing pairwise & spectral approaches for the Rasch model should be emphasized in the main text. Reading through Appendix D, it is apparent that the differences are major, but this was not clear in the main text (beyond the brief comment in the introduction about dense vs. sparse matrices). I strongly suggest making use of any additional space to move parts of Appendix D to the main text. Finally, statements like that of line 135, "our algorithm instantiates the general spectral approach [...]" are slightly misleading, given prior work.

These weaknesses are relatively minor and I believe they can be addressed in a camera-ready version.

Small comments:

- I am surprised that Agarwal et al. [2] is not cited in Section 4, "accelerating the spectral algorithm".
- Paragraph starting on line 50: an explicit expression for the CMLE would be helpful; it is hard to understand what the "likelihood conditioned on $\{ s_l \} $" looks like.
- l. 60: later -> latter
- l.116 and 204: practicioners -> practitioners
questions: - Why is the MMLE so bad in the Top-K case, depsite achieving good AUC and log-likelihood? (I don't understand the caption of Table 1).
- Can you comment to what extent the proofs of Thms 3.1 and 3.2. mirror the proofs in Rank Centrality paper for the BTL model?
limitations: Limitations are adequately addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: UPDATE: I increased my score based on the authors' response for noted in my counter-response.

The paper presents a novel representation of the statistical correlation between items in a 1PL (Rasch) IRT model that allows faster and (empirically) more accurate estimation of the underlying item difficulties (beta).  By embedding the items in a Markov Chain and calculating its stationary distribution, the authors are able to prove finite sample complexity bounds, which I believe is a first for such a method.  They provide extensions that speed up learning by using different normalization constants for each item and also add a regularization term to handle sparsity in real datasets.  Their empirical results show either parity or improvement against existing methods with an order-of-magnitude speedup against the most-accurate competitor.
strengths_and_weaknesses: The paper provides a very thorough analysis of the spectral approach in the Rasch model case.  I have some questions below about the differentiation to other spectral methods and some terms and definitions but overall those are minor and I think the analysis in the Rasch case is first rate.  The theorems look correct to my reading and exploring the connection between a MC's  stationary distribution and the overall item difficulty parameters is a clever choice.  I also liked the real-world practitioner extensions with variable normalization and regularization which make the algorithm truly applicable to the always-way-to-sparse item response data.

 However, I was disappointed that the authors did not mention or attempt to extend their approach to more modern IRT models including 3PL (where items have 3 axes of rating, not just difficulty) or multivariate IRT (where there are multiple thetas, for instance a student’s ability adding fractions and their ability to calculate percentages).  These more modern models are often calibrated using machine learning techniques (see below) and have far more intense computational burdens than Rasch so it is disappointing to not see at least a proposed extension to those cases.

More specifically, if we move to just 2-PL, how would the edges in the spectral embedding (the probability of transitions) represent both item parameters?  Could this be done using a factored Markov chain or some other more complicated graphical model?  Or is the spectral approach limited to only a single item-difficulty parameter? 

In the case of MIRT, we have seen recent successes of machine learning methods that use pairwise item comparisons to advance the very difficult problem of calibrating IRT models, see:
“Multidimensional Item Response Theory in the Style of Collaborative Filtering” (https://link.springer.com/article/10.1007/s11336-021-09788-9) as an example.  Since computational burdens are much larger with MIRT I would like to know if the computational advances in the Rasch have repercussions in that space?

Returning to the Rasch case and the results in the paper, I found the spectral embedding clever and a unique statistical approach for solving this problem.  I very much appreciate the finite-sample complexity bounds in the paper.  There are, however, are several places where I would like to see clarifications:

The authors cite several other spectral methods but do not make clear if any of the theoretical results come from those papers. Is Proposition 2.1 specific to the current paper or was it proved in one of the earlier works?

For Theorem 3.3 (where m grows) the authors need to describe to this audience why JMLE fails in this case and what is different about their algorithm that makes it succeed.

Table 1 needs to be reformatted to fit the margins.  Also, the authors should make a bigger deal about the speedup compared to MMLE.  It’s not just faster, the new method is an order of magnitude quicker.

I found Section 6 unhelpful and it seemed to be trying to justify why this paper fits at NeurIps by describing tangentially related problems.  I know there are connections to general logistic regression problems here but this section would be better used describing more complex IRT models like MIRT and how the approach could be used there.  In terms of fit at Neurips, I think the paper is ok as a statistical learning theory approach that models a psychological phenomenon although ultimately this result belongs in a Psychometrics journal to be really picked up by practitioners.
 
Minor points:

Lines 29 and 30: citations needed for these claims

Line 31: “Traditionally, the goal of estimation under the Rasch model is to recover the item parameters” … No, the goal is to assess the mastery parameters theta – at the end of the day we want to know whether students have certain skills or not.  Calculating beta values are a means to that end but they are not the goal.

questions: From the review, how does the new approach expand our capabilities in the 2PL/3PL or MIRT case?  

The authors cite several other spectral methods but do not make clear if any of the theoretical results come from those papers. Is Proposition 2.1 specific to the current paper or was it proved in one of the earlier works?

For Theorem 3.3 (where m grows) the authors need to describe to this audience why JMLE fails in this case and what is different about their algorithm that makes it succeed.

In accordance with Neurips guidelines on AI ethics, the authors should provide in their response a more detailed description of potential biases in this method compared to existing IRT approaches and clearly spell out possible negative societal impacts.
limitations: Beyond the missing extensions above I do think a discussion of ethics is necessary for this paper to be published, given Neurips's dedication to the topic.  I understand the work here is purely theoretical so I didn't flag it for ethics review below, but the application domain for this IRT method is evaluating children.  So any systematic errors or biases in the new approach need to be spelled out much more clearly.  The authors should provide in their response a more detailed description of potential biases in this method compared to existing IRT approaches and clearly spell out possible negative societal impacts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an inference algorithm based on a spectral estimation approach for estimation of the item parameter in the IRT model with finite-sample guarantees. The authors provide detailed convergence analyses and demonstrate the utility of the proposed estimator via both simulated and real-world datasets, on which the proposed approach achieves performance comparable to other estimators (e.g., MLE) at a fraction of the computational cost.
strengths_and_weaknesses: The paper makes interesting theoretical contributions to the inference algorithm for IRT model. The algorithm is simple to implement and has finite-sample guarantees. The analyses are sound and the experiments corroborate with the analytical results. The proposed method is also competitive with other inference algorithms. I do not find major weaknesses; some suggestions are outlined in the "limitations" part.


questions: Will and how does the algorithm generalize to settings in which student/user parameters need be estimated instead of sampled from a prior?
limitations: It seems that the proposed approach is applicable to only 1PL IRT model (Rasch model). Although still widely used, there are other, potentially more power, variants of the IRT models (e.g., 2 or 3 PL IRT models) and it is not quite clear whether or how the proposed algorithm generalizes to other IRT variants. This is a potential limitation, although not a main concern.

The proposed approach relies on prior distribution on the student parameter, i.e., the student parameters are not estimated. For applications where only item parameter estimations are needed, the proposed approach might be sufficient. In other, also common, settings where the student parameters also need be estimated, it is unclear how the proposed approach would be applicable.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a spectral estimation algorithm for item parameters in the 1PL item response (aka Rasch) model. The authors conducted theoretical analyses of the proposed algorithm in several scaling regimes, proposes a variant of their algorithm under practical constraints, and conducted experiments on real data to show that their method is on-par with existing algorithms in terms of estimation accuracy while being more computationally efficient. 
strengths_and_weaknesses: Strengths: 
- Although spectral estimation has been thoroughly studied, its application to the Rasch model is novel
- I liked the fact that the authors provided a version of their algorithm in practice where one can use a different normalization factor d for every item. This is important and likely helps a lot in terms of Markov chain convergence - a uniform d can be really bad for some items on real world datasets
- I have not checked the proofs but the theoretical results and discussions seem sound

Weaknesses:
- ML is only one family of estimation methods but there are others, say posterior mean etc. I get that the proposed methods may be better than M/J/CMLE, which have their own problems for sure, but how would its accuracy/runtime compares to Bayesian methods is not clear. See below. 
- The experimental settings and results are not clear. See below. 
- The presentation could be improved in several places. See below. 

questions: 1. Main concern: experiments. The experimental settings and results (on real data) are really underdiscussed, even in the supplementary material. The overall takeaway is a solid one - the proposed spectral algorithm is on-par with existing algorithms in terms of accuracy and is much faster, which is perfectly fine. However, some details give me a sense that some baselines are not properly treated. Some things that the authors can do to help are:
- at least discuss what Bayesian estimation methods would do; more accurate when the prior is right? Too inefficient on large real world datasets? A simulation study where you know the real item parameter prior or experiments on one real world dataset would certainly help. 
- many datasets are not discussed (LSAT, UCI, 3GRADES) where the number of items is very very small. Here, AUC is the same for all methods. The experimental results are not thoroughly discussed; I'm a little unsure about the huge variation in terms of ranking performance since after all, these are different algorithms for the same underlying statistical model. The 0.0 results for top-K accuracy for some methods are a bit suspicious; it's not totally surprising, but more discussion is definitely needed. Otherwise, it's just really hard to properly evaluate how beneficial the proposed algorithm is in practice. 
- AUC scores for most methods are pretty close on most datasets, which makes perfect sense. However, the proposed method is obviously worse on BX; why?

2. Some minor aspects of the presentation could be improved:
- in the abstract, it's probably more accurate to call the proposed algorithm "item parameter estimation" instead of "item estimation"
- line 32 "educational testing"
- line 37JMLE does not really have to use an alternating maximization algorithm, right? The problem is jointly convex in the item and ability parameters
- I would like to see Theorem 3.1 and 3.3 written with ||beta - beta*||_2/sqrt(m) on the left hand side - this quantity makes more sense in practice
- Need to zoom into the small n regions in Figure 1. Lines are overalpping at times and it's hard to tell which is which. 
limitations: This field is not really applicable to this paper
review_title: null

Meta-review: metareview: This is a strong paper with interesting theoretical results and important practical contributions (much faster parameter learning than prior methods with little performance drop-off). All reviewers agreed it was above the bar for acceptance to NeurIPS.
confidence: Certain
Decision: Accept