Title: Generalized Top-k Mallows Model for Ranked Choices
Venue: NeurIPS 2025
Accepted: yes
Keywords: Mallows Model

choice modeling

top-k sampling
Abstract: The classic Mallows model is a foundational tool for modeling user preferences. However, it has limitations in capturing real-world scenarios, where users often focus only on a limited set of preferred items and are indifferent to the rest. To address this, extensions such as the top-$k$ Mallows model have been proposed, aligning better with practical applications.
In this paper, we address several challenges related to the generalized top-$k$ Mallows model, with a focus on analyzing buyer choices. Our key contributions are: (1) a novel sampling scheme tailored to generalized top-$k$ Mallows models, (2) an efficient algorithm for computing choice probabilities under this model, and (3) an active learning algorithm for estimating the model parameters from observed choice data. These contributions provide new tools for analysis and prediction in critical decision-making scenarios.
We present a rigorous mathematical analysis for the performance of our algorithms. Furthermore, through extensive experiments on synthetic data and real-world data, we demonstrate the scalability and accuracy of our proposed methods, and we compare the predictive power of Mallows model for top-$k$ lists compared to the simpler Multinomial Logit model.


Peer reviews:
comments: SUMMARY:
This paper proposes inference methods for generalized Mallows model using top-K ranking data, in which each item has an associated weight. Inference is built on efficient sampling of rankings by reducing the computation time of the method by Chierichetti et al. (2018a). Two methods are then proposed for estimating choice probabilities given model parameters and estimating model parameters given choice data.

STRENGTHS AND WEAKNESSES:
Strengths: Extensive theoretical analysis and synthetic data experiments are provided. Real dataset experiment shows improvement against the Multinomial Logit (MNL) model. 

Weaknesses:

A notation table would have been very helpful to improve clarity, particularly as the proofs in the Appendix have some inconsistencies (e.g., caligraphic P should be standard P in line 815, Equation 6 numerator should be A^bar). Further, terminology is inconsistent such as items, samples, products meaning the same thing. 

While it is important to outperform MNL, this observation is somewhat expected with respect to the literature (e.g. Desir et al.). It is more interesting to compare the proposed algorithms in terms of predictive accuracy and computation time to existing efficient top-k Mallows model inference algorithms like Chierichetti et al. or Fotakis et al. 

Hyperparameters p and beta were tuned on the test set (rather than a separate validation set), which is unfair against the MNL model.

QUESTIONS:
N/A

LIMITATIONS:
Please see above.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies top-K Mallows model, a variant of the ground truth-based statistical model, for sampling rankings. The paper builds on a paper by Chierichetti et al. titled “Mallows models for top-k lists”. First, the paper provides the “PROFILE BASED REPEATED INSERTION METHOD”, i.e.,  sampling procedure for the generalized version of the Mallows model. Second, the paper introduces the “DYNAMIC PROGRAMMING FOR CHOICE PROBABILITIES”, i.e., a method for calculating the choice probabilities functions. Third, the paper provides the “BUILD CENTER FROM CHIOCES”, i.e., a learning algorithm for inferring the ground truth of Mallows. Finally, the authors provide experimental analysis, demonstrating the behavior of the top-k Mallows model. They use synthetic data and real-life Sushi model (one of the most frequently used dataset in computational social choice).\n\nSTRENGTHS AND WEAKNESSES:\n[Significance]\n\nThe Mallows model is widely used, and the generalization to the top-*k* variant seems like a natural and valuable extension that one would want to have.\n\n[Originality]\n\nThe originality of this paper is somewhat limited, although I don't see that as a major drawback.\n\n[Quality]\n\nThe overall quality of the paper is good, with some minor issues that should be addressed:\n\n- Lines 138/139 “This concept has been extended to top-k lists, where it no longer forms a true metric but retains useful mathematical properties.” This statement is too vague. There are various ways in which a function can fail to be a true metric—for example, becoming a pseudometric, semimetric, or quasimetric. It seems that the function described here is a semi-metric as it fails the triangle inequality. Please clarify explicitly in what way the function deviates from being a true distance.\n- The citations occasionally feel somewhat arbitrary. While the paper includes an extensive list of works related to the Mallows model, several cited papers seem only vaguely relevant. It would be helpful if the authors reviewed the bibliography more carefully to ensure relevance and accuracy. For example, ”Liu, A. and Moitra,” appears twice. (Side comment: it might be worth taking a look at “Properties of the Mallows Model Depending on the Number of Alternatives: A Warning for an Experimentalist”)\n- “the generalized Mallows model for top-k lists provides a more realistic framework\nfor understanding user preferences”—while this is quite a strong statement, there is little evidence supporting it in the paper.\n- The authors state that “the Multinomial Logit (MNL) (Bradley & Terry, 1952) model being the most widely used due to its simplicity and interpretability.” Please provide justification for that claim. Additionally, clarify the context in which the model is considered “most widely used”. For example, in the context of computational social choice, the “Guide to Numerical Experiments on Elections in Computational Social Choice” suggests that the impartial culture model is the most widely used.\n\n[Clarity]\n\nClarity of the paper could be improved. \n\n- The font size in Figures 1 and 2 is far too small. As a result, the plots are difficult to read and fall short of NeurIPS presentation standards. Please increase the font size and improve the overall readability of the figures.\n- The “Preliminaries and Definitions” section looks like it is almost copy pasted from the “Mallows models for top-k lists” paper.\n- The authors refer to Kendall’s tau distance, by writing: “Kendall”, “Kendall tau, and “Kendall Tau”. Please unify the name throughout the paper.\n- Line 26: “The Mallows model Mallows (1957)” is awkward. Please rephrase this sentence or use a different citation command.\n- Line 167: A misplaced comma should be moved one line above\n- Line 179: “method for permutations Doignon et al” is awkward. Please rephrase this sentence or use a different citation command.\n- Typo in line 115: CHIOCES\n\nQUESTIONS:\nQ1: In what practical use-case the p parameter would be set to a value larger than 1? (Intuitively I would say that p should fall in [0,1] interval).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a generalized Mallows model for top-k preference lists over n items that allows item-specific weights and extends the Kendall-tau distance to partial rankings. Three tasks are considered: sampling a top-k list according to the model. It takes  O(k*2^k+klogn) time; computing the choice probabilities, which takes O(2^k*k^3*∣A∣^2) time, where A is the assortment; and an active learning algorithm that learns the central ranking with O(log n) assortments.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n\n\nThe main contribution of this paper is the introduction of a probabilistic model for top-k lists, which extends the Generalized Mallows Model of Fligner and Verducci (1986). The analysis of the sampling procedure and the derivation of choice probabilities build naturally on prior work, particularly the RIM method and the dynamic programming framework. The derivations can be difficult, given the complex combinatorial structure, and hence the paper has merits in terms of technical contributions. Overall, the paper is clearly written.\n\n\nWeakness:\n\n\n\n1. (Computational complexity) It is unclear how computationally tractable the proposed GMM for top-k lists is in practice. For example, according to Theorem 3.1, even sampling a top-k list can take O(k·2^k) time, which is a very weak guarantee when k is close to n. This contrasts with the RIM method, which remains efficient at least when k = n, and the sampling procedure of Feng and Tang (2021), which is very efficient albeit based on a different variant of the Mallows model. Unsurprisingly, computing choice probabilities also appears challenging when k is large; see Theorem 4.1.\n\n\n2. (Parameter learning from choice data) Given the discussion above, it is only natural to expect that learning (i.e., estimating the model parameters) from choice data would be even more difficult. I therefore read Section 5 with interest, but found the results somewhat underwhelming for several reasons:\n\n\n    - 2a First, the analysis assumes that the algorithm has full control over the assortments. In fact, I believe BUCCHOI only picks assortments of size at most two. From a practical perspective, however, the observer may only be endowed with observed data and then the question is how to infer the model parameters (e.g., the central ranking) in which the assortment history is arbitrary (i.e., not controlled by the observer). Methodologically, this is precisely the challenge that much of the prior literature (e.g., Desir et al. 2022; Feng and Tang, 2021) was trying to address, but this paper effectively sidesteps it. If the assortments are only pairwise, there is already a **very** vast body of work on passive and/or active learning of ranking and/or best-item identification from pairwise comparisons. (So even in the active learning setting, it is unclear how strong the sample complexity guarantee of \\Theta(\\log n) is.)\n\n\n    - 2b Another important question is that the model brings richness by introducing a few new parameters and the paper remains relatively silent about how to determine them. Based on Section 5, the underlying data are choice data (rather than ranked or top-k list data). This makes the value of k essentially a modeling parameter, but it is unclear how to \"learn\" the right k. In addition, one of the key distinctions of this model compared to the original Mallows model is the introduction of the \"weights\" w. However, it remains silent how these w parameters should be calibrated. If making them uniformly one, does the proposed model go back to some more basic version of TopKGMM?\n\nQUESTIONS:\n1 It is claimed that \"once setting w = 1, we obtain Equation (TopKMM).\" But that equation is actually parameterized by w. Is there a typo? \n\n2. Why omit Mixed-MNL or Plackett–Luce mixtures? This is usually included in the literature so explanation of the exclusion would be helpful. \n\n3. It is unclear what the 100-sushi data tries to mimic in practice. If the authors are just trying to mimic a situation where the observer can only see single choices, then the value of \"k\" is only a hyper-parameter to be tuned. If the authors are actually using the fact the the raw data is based on top-10 lists, why not directly train the model based on the top-10 lists?\n\n\nApart from that, I do not have questions beyond the \"weakness\" comments.  That said, in the rebuttal, if the authors can come up compelling advantages of the newly proposed model/method compared to what has been proposed in the literature, I am happy to re-evaluate.\n\nLIMITATIONS:\nYes (after various clarifications of the rebuttal process).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel sampling strategy, an efficient algorithm, and a parameter learning method for generalized top-k Mallows models. Specifically, traditional Mallows models only fit the full permutation circumstances, while practical scenarios require Mallows models for top-k assortments, which are called generalized Mallows models. The proposed method extends the generalized Mallows models from three perspectives: RIM for generalized Mallows models, choice probabilities calculation for top-k list, and the choice centers construction. The authors provide the theoretical guarantee and empirical validation to support the effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nI am totally unfamiliar with the topic and could only provide some reviews on formatting and illustration. My understanding of technical details might be wrong, and the authors are welcome to figure out mistakes within my review. \n\n# Strength\n\n1. Clarity. The authors clearly state their motivation, method, and experimental results. The background and notations for the studied problem, as well as the practical usage scenarios, are sufficiently illustrated to understand the problem and setting of the work. \n\n2. The authors provide theoretical guarantees for the proposed method. \n\n3. Extensive experimental results are provided to support effectiveness. \n\n\n\n# Weakness \n\n1. The text in several figures is too small to read. \n\n2. The grammar in some places might be problematic; please check. For example, around Line 23, the authors wrote, \"Based on Plackett-Luce (PL) model the Multinomial Logit (MNL) model has been suggested\", where there should be a comma between \"model\" and \"the\". \n\n3. A holistic comparison with current methods may help to signify the importance of the proposed method. The authors illustrated their contributions by iterating the components, where there is a lack of an overview of the systematic advancement of current methods. The authors may illustrate how they achieve more usable/suitable generalized Mallow models by proposing three novel components.\n\nQUESTIONS:\n1. How to use the proposed method for practical problems? In other words, can the authors provide a usage example/algorithm (or maybe it is one of the presented algorithms in the Appendix)? \n\n2. How do centers of choices help to enhance the framework?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a generalized Mallows model for top-k lists and develop efficient algorithms\nrelated to generating samples from it, learning its parameters and finding choice probabilities.\n\nIn particular, they propose a sampling scheme with running time exponential in k (the length of the top-k list) and polynomial in n, improving upon the previously best sampling scheme, and adapted to the generalized top k Mallows model. They give a dynamic programming algorithm to compute the choice probabilities\nAnd finally provide a method to learn the center of the top-k generalized Mallows model.\n\nExperimentally, they check the runtime of their algorithms and also show that that the test error on the Sushi data set is lower as compared to the Multinomial Logit Model.\n\nSTRENGTHS AND WEAKNESSES:\nI don't feel particularly convinced by several parts of the paper and I feel that I struggle to understand its relevance -  please see my questions for several things I am unclear about. The authors could much improve the motivation in the paper by offering concrete examples in which their particular model is more useful/ important/ relevant than other ranked choice models.\n\nThe experiments are down for the top-k mallows model not its generalized version and thus do not support the version introduced by the authors in this paper . This begs the question where one would use the weighted version.\nAlso the TopKGMM mixture models are computed by using the Kendall-Tau distance K^p, and then used to compare TOPKGMM with MNL, and I worry that this inherently is advantageous towards TOPKGMM. At the very least the authors should discuss this choice. Figure 1 and the meaning of the test error is further not clear to me.\n\nI am not convinced that this paper solves an open question regarding a new RIM for top-k Mallows. My impression would be that RIM is a very nice and simple method, and something is RIM-like in so far as it is still simple, not because it uses insertions in some way.\n\nLine 164:\nI would be useful to explain in words to the reader what I, P and Q mean.\nFor P_3(tau): are not 4,7,8 all greater than 3 and all of 3,4,7,8 satisfy that they ar in \\bar(\\tau)? Thus, shouldn't \nP_3(\\tau)=3? Similarly P_4(\\tau) seems wrong?\nLine 45: \"efficient\" - how is exponential in k an efficient algorithm?\nLine 159: Should the index in the summation include \\tau^*? What is \">\"? Is it the same as the curly version?\n\nMinor comments:\n\nLine 158: It would be helpful for the reader if you introduced the general inversion vectors before you introduce the special case, to understand what they are.\nLine 115: Chioces -> Choices\nLine 124: full stop missing\nLine 163: \"For example,..\" This is not a full sentence\nLine 167L \",\" should be on the previous line\nLine 210: later -> latter\n\nLine \"219\": One cannot evaluate the computational complexity of an algorithm through experiments..\n\nQUESTIONS:\nWhy is Generalized Mallows sensible? Why would you only have weights attributed to the higher element?\nIs this well motivated or does it simply make the math easier?\n\nWhat is the sample complexity of doing the same things as you do in the paper for the Plackett-Luce model? In particular, the Plackett Luce model seems mode amenable to top-k rankings, since its sampling algorithm exactly sampled in order of ranking.What is the main different between Plackett Luce and Mallows, especially if in Plackett Luce once uses exponential weights?\n\n\nMinor Questions:\n\nWhy do you say that two items in tau bar are incomparable as opposed to assuming they are a tie (mathematically)?\nLine 29: \"High predictive Power of the Mallows model...\": What do you mean by this?\n\n\n\nLine 33: Can you reference concrete examples where platforms show user specific top k relevant items and the k\nIs the same across all users?\n\nWhat is \\tau^* in Equation 1? Do you mean \\tau'?\n\nLIMITATIONS:\nThe algorithms are exponential in the parameter k, thus not practicable for larger k. It is not clear why the generalized version for top-k Mallows is needed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}