Identity testing for Mallows model

Abstract: In this paper, we devise identity tests for ranking data that is generated from Mallows model both in the \emph{asymptotic} and \emph{non-asymptotic} settings. First we consider the case when the central ranking is known, and devise two algorithms for testing the spread parameter of the Mallows model. The first one is obtained by constructing a Uniformly Most Powerful Unbiased (UMPU) test in the asymptotic setting and then converting it into a sample-optimal non-asymptotic identity test. The resulting test is, however, impractical even for medium sized data, because it requires computing the distribution of the sufficient statistic. The second non-asymptotic test is derived from an optimal learning algorithm for the Mallows model. This test is both easy to compute and is sample-optimal for a wide range of parameters. Next, we consider testing Mallows models for the unknown central ranking case. This case can be tackled in the asymptotic setting by introducing a bias that exponentially decays with the sample size. We support all our findings with extensive numerical experiments and show that the proposed tests scale gracefully with the number of items to be ranked.


Reviews and author responses:

The authors propose asymptotic and non-asymptotic test for the hypothesis $\phi = \phi_0$ where $\phi$ is the spread parameter of a Mallows model. To do that, they first prove that we can derive a non-asymptotic test from an asymptotic one (Prop 4.1), then show that an optimal unbiased UMP test can be defined is too computationally costly when the number of items to order ($m$) is large (Theorem 5.1), due to the calculation of the probability in Eq (2).  To alleviate this limitation, they propose to estimate the most likely spread parameter $\hat{\phi}$ which allows them to efficiently approximate the quantity in Eq (2) and control the deviation (Theorem 5.4). Finally, the authors address testing for spread with unknown central ranking, by showing that the average rank aggregation is asymptotically close to the central ranking. Finally, they confirm the efficiency of their method in experiments.  **Main points:**

I believe the paper provides a detailed and well-structured response to a specific and interesting theoretical problem.
I have two main concerns with the work:

1) I do not neglect your contribution to ranking models, but I believe there is a lack of motivation for this problem in your paper when presenting to a machine-leaning conference. The related problems of ranking aggregation does not suffer from this, since ranking aggregation is a practical problem in many situations. Can you identify a significant (or several) practical problems where testing for the spread value of a Mallows model is important?

2) I am not perfectly familiar with reference [r1] (added below), but they assess the goodness of fit of parameters to the Mallows Model in particular, a problem very related to testing,  and they also deal with the case of large $m$, as you do in your paper. Could you explain why your contributions are significant wrt their propositions?

**Details:**

* Why is the set of all permutations on $m$ sometimes $S_m$ (e.g. l. 209), sometimes $\mathbb{S}_m$ (e.g. l. 122)?
* What is $\Omega(1/\epsilon^2)$ in l. 183? Is $\Omega$ supposed to be the cardinal of $\Omega$ here?

[r1] Assessing Goodness of Fit of Ranking Models to Data, Cohen and Mallows, 1983

-------------- AFTER READING THE OTHER REVIEWS AND THE RESPONSES

The authors adequately addressed the points I raised and I raised my score to weak accept.
However, I still believe that the reviews that we (the reviewers) were unable to put the theoretical contribution in perspective, which requires a thorough knowledge of ranking models (which is rare due to the focus on score-based approaches). It would be interesting to have the additional opinion of someone that is very familiar with testing ranking models. Nothing to declare

 Thank you  for your constructive  feedback.

* Regarding the motivation of the contribution: There are several ML-related applications of hypothesis testing for the Mallows model. For example, consider an online system which recommends news or some other content to users. The number of clicks of each user induces a ranking over the topics, for example, some users are more interested in politics than sports, etc. When we launch a new recommendation system using A/B test, we would like to know whether the new system has some impact on the user interest ranking distribution compared to the current system. This problem can be viewed as a hypothesis testing problem, where the null hypothesis is the ranking distribution induced by the current production model and the alternative hypothesis is the data generated by the user in the test cohort. However, in our paper, we focused on devising testing algorithms and their properties which can be better demonstrated on synthetic data.
* Regarding the difference between $S_m$ and $\mathbb{S}_m$: it is a typo, thanks for pointing it out.
* $\Omega$ is the Bachmann–Landau notation. $f(x) \in \Omega( g(x))$ means that $f$ is bounded below by $g$ asymptotically. Sorry for not defining it, but we thought that this notation has already become a standard in the Machine Learning community. That is why we did not define it in the submission. We will add this definition to the final version.
* Regarding paper by Cohen and Mallows (1983): that paper presents a goodness of fit test for ranking data in a way the ranking data is handled as a sample from a discrete distribution with $m!$ parameters. That is why their analysis is restricted to $m=3$ and $m=4$. The authors presented a goodness of fit test of pairwise marginals of various ranking models for larger $m$ including Mallows model, Placket-Luce and the Thurstonian model, but their focus was to decide which model fits better to the data in terms of pairwise marginals. Whereas we were interested in testing the parameters of Mallows model with large $m$ values. Finally, Cohen and Mallows does not provide finite sample complexity analysis and provably UMPU tests. We will add this reference to the final version.   


 Thank you for the positive feedback. We agree with the reviewer that these references are indeed relevant and we will make sure that they will be added to the final version.


 Thank you for the positive feedback!
* The algorithms presented in Section 5.3 can be applied to test the spread parameter. If one wants to test whether the central ranking is the same, i.e. $d_K(\pi_0, \pi)>0$, then the average ranking approach can be used as it is described in Section 6.1 and 6.2.
* Regarding the Mahonian number: $N(\ell,n)$ is the Mahonian number in (2) which is denoted by $N_k(m)$ as well. We will fix this discrepancy in the notation in the final version.


 Thank you for the positive feedback!
* We will make sure that the figures will be better formatted in the final version.
* Regarding $\chi^2$ approximation: the $\chi^2$ approximation is what we would get once the limit of the corresponding central limit theorem is reached. When this happens is not clear theoretically due to the asymptotic nature of CLT. This is why we include the experiments to compare the non-theoretically rigorous convergence of the $\chi^2$ approximation with the theoretically rigorous approach which result is presented in Subsection 7.1.


Paper provided a UMPU test on Mallows model's spread parameter. And based on that paper introduced many variations and discussed their properties. Paper contributes to the testing literature by providing a UMPU test and associated math tools.  Paper has contributed the following
- A UMPU test on the based point hypothesis testing, based on which some finite sample variations and other sorts such as central ranking unknown is discussed, with mathematical properties provided
- Careful simulation is provided with decent performances even with small samples
- Paper is easy to follow and clearly written

This is a decent work and I believe is a good addition to the venue. 

- Figure 2 is hard to read. Venue format requirement may played a role here. Maybe increase the graph size and / or change data presentation is going to be better when submitting long ver to journals.
- Do we have some assumptions on $\chi^2$ approximations? A brief discussion on this would be good for users avoiding corner cases in practice. not applicable to this work

The paper studies the problem of identity testing for the Mallows model, which is a popular parametrized distribution over rankings. It gives identity testers for the non-asymptotic and asymptotic setting which are sample optimal but not necessarily computationally efficient. It also gives a efficient identity tester, with weaker guarantees, and also extends the results to the setting where the central ranking is not unknown.

  The paper studies the problem of identity testing for the Mallows model which is a well studied family of distributions over rankings i.e, given a set of samples (rankings) $\pi_1,\ldots,\pi_n$ drawn from $M(\pi_0,\phi)$ with $\pi_0$ being the central ranking and $\phi$ being the spread parameter, the objective is distinguish where $\{\pi_i\}$'s are drawn from $M$ or from a distribution which is $\epsilon$-far from $M$ in the total variation metric. The main contributions of the paper are that it gives testers for the asymptotic and the non-asymptotic settings, which are then shown to be sample complexity optimal. The paper then extends these results to the setting where the central ranking $\pi_0$ is not unknown.

The main technical challenge here is that naïve application of identity testers from present literature will result in algorithms whose sample complexity have prohibitive dependence on the size of the ranking $m$, and therefore a more fine tuned approach is needed which exploits the properties of the distribution function of the mallows model. Using a careful argument, the authors first explicitly characterize the analytic form of an unbiased asymptotic tester, which is then subsequently used to give a (computationally expensive) sample complexity optimal tester for the asymptotic setting. They also give a computationally efficient tester which is based on the testing-by-learning approach, using recent results on learning Mallows block models.

Overall I think this is an interesting direction to explore in the setting of property testing, and the paper provides a nice set of first such results for the setting of Mallows model.

Strengths:
-------------------------------
> Gives new results for an interesting but under-explored direction in property testing.

> Approach could be useful in other settings where the distribution is supported on combinatorially large number of objects

Weaknesses
-------------------------------
> The guarantee of the efficient algorithm i.e, Theorem 5.4 seems arguably weaker than that of Theorem 5.2.

Comments/Questions for the Authors:
----------------------------------

> Line 33: that is -> that are

> Line 62: from computational -> from a computational

> Line 79: is devised -> was devised

> Line 129: access the -> access to the

> Remark 1: where is $N_k$ used in the definition? Reference for Mahonian number?

> Section 5.3: I am a bit confused about the main result of this section, as far as I understand, Alg. 2 gives a tester for the spread 
parameter, but then it's not clear if it immediately yields an $(\epsilon,\delta)$-identity tester as well? For e.g., how is it dealing with $(\pi,\phi)$ pairs where $\phi = \phi_0$, but $d_K(\pi_0,\pi)$ is large?

 Yes

This paper is concerned with the hypothesis testing of Mallows ranking models, which is parametrized by a central ranking and a spread parameter. The authors proposed two different methods: one based on the classical Type I error, and the other based on an optimal learning approach. The authors presented a complete theory regarding testing Mallows rankings, and also provided empirical results to corroborate the theory.   This paper considers the problem of hypothesis testing for a popular ranking model -- Mallow's ranking model. The paper fills in the gap of testing procedures for a large random discrete object (ranking with a large size of items), which is important for the practical use. The authors then proposed two different approaches: one through the classical test framework involving significance level and power, and the other based on an optimal learning procedure. Both approaches were treated rigorously, and the results are scientifically correct. Empirical results were also provided, and showed that the proposed optimal learning testing is valid for testing a Mallows ranking with $1000$ items (this is considered to be large in testing ranking models). 

Overall, I like the paper and enjoy reading it. Below I provide a few more references that the authors could add, and compare with their results. 
1. Some nice development of Mallows model, and related models were made in late 80s by Fligner and Verducci, see "Distance based ranking models", Journal of the Royal Statistical Society. Series B (Methodological), pp. 359–369, 1986, and "Multistage ranking models", Journal of the American Statistical association, 83(403):892–901, 1988.

2. Regarding testing Mallows model with unknown central ranking, the authors indicated that the MLE is NP hard, and proposed to use the average rank instead. In particular, they proved in Lemma 6.1 a 'rate of convergence' of the average rank estimate. I would like to point out a parallel result for the MLE, which could be found in "Mallows Ranking Models: Maximum Likelihood Estimate and Regeneration", Proceedings of the 36th International Conference on Machine Learning (ICML 2019) PMLR 97, 6125-6134.  There Theorem 3.2 provides a lower and an upper bound for $\mathbb{P}(\pi_{MLE} \ne \pi)$. It may be worth pointing out that Lemma 6.1 is much better for large random models, since the rate is $m^2 \exp(-Cn)$ while for the MLE the rate is $m! \exp(-Cn)$.
 The authors have adequately addressed the limitations and potential negative societal impact of their work.


Meta-review: Mallows model are a popular distribution of rankings. This papers studies the problem of testing whether the underlying distribution is equal to a known Mallows model or is far from it. This is a clean question, and is a novel avenue for applying goodness-of-fit testing. The paper is above the bar for NeurIPS. 

Decision: accept