Private and Non-private Uniformity Testing for Ranking Data

Abstract: Róbert Busa-Fekete, Dimitris Fotakis, Emmanouil Zampetakis

Reviews and author responses:

 Thanks for the feedback and the clarifications! 

The paper studies uniformity testing for certain structured distributions on large domains. The classic uniformity testing problem is to decide, using sample access, whether an unknown distribution over a domain of size $n$ is the uniform distribution or $\epsilon$-far from the uniform distribution in total variation distance. The sample complexity is known to be $\theta(\sqrt n)$ for this general problem and hence is not feasible when $n$ is large unless we restrict the class of distributions under consideration. The present paper considers the class of distributions known as the Mallows model: this is a class of exponential distributions on rankings (set of permutations over a domain). More precisely, given sample access to a distribution from the Mallows model: decide whether it is the uniform distribution or $\epsilon$-far in total variation distance from the uniform distribution. The paper presents both theoretical as well as experimental results on this problem. On the theoretical side, it proves upper bounds on the sample complexity in  various settings including for differentially private testing. On the experimental side, it implements and tests these algorithms for various parameter settings.   The paper presents several theoretical as well as experimental results. However, this reviewer is concerned about the significance of the problem studies as well as the results.  While Mellows class of distributions may be useful in several settings, it is not at all clear whether a uniformity testing algorithm with privacy or without privacy for this class of distributions is a significant research question. Thus the result might not be of interest to a broad NeurIPS audience. 

Update After Rebuttal

While, as presented in the paper the motivation for investigating this specialized class of distributions for uniformity testing is weak, the paragraph in the rebuttal gives much-needed motivation for the problem considered.  I recommend that the authors add a similar paragraph in the next version. Also, the interesting paper by Mukerjee cited is missing in the submitted version.  The author/authors state that "I hope there is no negative social impact of my work" in the checklist document. 

This work considers uniformity testing of ranking (or permutation) of a discrete set with and without differential privacy (CDP and LDP). The ranking distributions are assumed to belong to the Mallows model family. The authors designed algorithms for these cases and proved their sample complexities. Synthetic experiments are also conducted for the algorithms.  Strength:
This work provides a comprehensive understanding for the sample complexity upper bounds in uniformity testing of ranking distributions in the Mallows model family. Algorithms are designed for non-private, CDP and LDP settings. The algorithm sample complexities scale as the domain size instead if the number of permutations. 

Overall the writing is clear.

Major concerns:
1. For CDP setting, it is claimed that TRUNC is better than the naive approach when $m$ is small. However, when $m=10$, TRUNC seems to have a worse performance. Could the authors explain the discrepancy here?
2. What are the time and space complexities for the algorithms?
3. For CDP and LDP the maximum domain size is only 1000. Experiments for larger domains could be done at least for CDP. 


Minor concerns:
1. $\phi_\epsilon$ seems to be hard to compute in practice. Could the authors characterize the relationship between $\phi_\epsilon$ and $\epsilon$, or provide a method to emprically calculate this? 
Or perhaps characterize the problem as testing between $\phi=1$ and $\phi\le1-\varepsilon$, instead of TV distance?
2. This work does not include lower bounds (only a lower bound for a special case which can be reduced to testing Bernoulli random), though it is OK not to have them for this submission.

Other comments:
1. The sample complexity bounds that go to 0 with $m$ (such as$O(1/m^{1/2})$) are not very accurate since you mentioned that it is impossible to do testing with only one sample. You should add a constant term. 
2. For the LDP experiments, what is $k$?

 Yes.

 Thanks very much for your response. You addressed my concerns and I have increased my score. 

However, for the sample complexity expression, I think you should still add a constant if the expression goes to 0 and the minimum samples required is 2. When the constant dominates, it cannot be viewed as hidden inside the big O notation.

 Thank you for the positive feedback. 

The main technical difficulty was to handle the arbitrary central ranking of the alternative class. Note that the tester does not know the central ranking, and it doesn't even estimate it. But what is happening is that the testing algorithm picks a random ranking $\pi$ (Line 2, Algorithm 2) and it uses that ranking to compute the pairwise statistics. Under the null hypothesis, this random ranking $\pi$ does not play any role, since the expected value of the pairwise statistics are all 0 with any ranking $\pi$. But it turned out that, under the alternative hypothesis, this random selection of ranking ensures that a constant fraction of the expected value of the binary statistics are deviating significantly from $0$ with high probability, and thus the alternative hypothesis is easy to distinguish from the null hypothesis.


 Thank you for the review. 
Since the reviewer has no concern regarding the technical content of our submission, but only its relevance to the NeurIPS audience, we would like to point out that several influential papers have been published by the NeurIPS conference on property testing and on Mallows models. In addition to this, the following applications may address the concern of the reviewer, which application is of interest to the ML community which focuses on fairness: 
* Significant attention has been attracted to machine learning systems and their ability to discriminate against minorities, historically disadvantaged populations, and other protected groups when allocating resources (e.g., loans) or opportunities (e.g., jobs). For example, the Fair Lending Act -- a commonly used fairness requirement introduced in the USA -- is explicitly created to impose such unbiasedness on financial companies. This policy even has to be respected in audience targeting in online advertising. Uniformity testing for ranking data can be applied to practical settings where we want to collect/provide evidence that our ranking system has no bias towards some group of users. First, one filters the data for cases with individuals having similar economical backgrounds. The obtained data is then partitioned based on time, and for each class within the partition the groups of interest (e.g., age groups, gender groups, ethnic groups, etc.) are ranked due to the number of successful applications or amount of loan received (proportional to the size of the group). The lending policy of a company is fair if this ranking samples (the sequence of ranking through time) cannot be distinguished from uniform sampling. This is indeed where uniformity testing presented in our submission can be applied. 

In addition, a very interesting application was presented by Mukherjee 2016 in Ann. Statist. where a draft lottery data was analyzed from a point of bias, where indeed the hypothesis to be tested was whether the data is uniform.


 Thanks for the positive feedback.
Regarding major comments:
1) The proper claim should be small $m$ and when $\phi$ is close to 1. We will make this claim more accurate.
2) Our testing algorithms are based on computing $m/2$ pairwise statistics, thus the space complexity is $O(m)$, and computing the pairwise statistics is again linear in m for a ranking, therefore the time complexity is equal to sample complexity times $m$.
3) We will add these results to the appendix.

Regarding minor and other comments:
1) Regarding $\phi_\epsilon$: the total variation distance between two Mallows model with the same central ranking can be computed in quadratic time in m, since the permutations can be grouped by their Kendal distance from the central ranking and then the total variation distance boils down to computing $\sum_{i=0}^{m*(m-1)/2} \vert1-\phi^i \vert * M(i,m)$ where $M(i,m)$ is the $i$th Mahonian number of order $m$ and the normalization $Z(\phi)$. The Mahonian number can be computed based on recursion (Knuth 1962, Vol 3) and the normalization of the Mallows model can be computed in closed form (Fligner and Verducci 1986). Furthermore, the total variation distance for Mallows model from the uniform distribution is monotone in $\phi$, thus binary search can be applied to find $\phi_\epsilon$. We will add a remark on that. In our experiments, we assessed the power of algorithms for a given sample size, which is a more realistic scenario in practice. 
2) This constant is hidden by the big O notation, nevertheless we agree with the reviewer that at least two rankings are needed to carry out the test.
3) In the LDP experiments k is 1000 samples. In general, the LDP algorithms require much more samples than the central DP algorithms, since its privacy guarantee is very strict.


This paper studies the problem of (private) uniformity testing in the ranking problem, where the data in the alternative hypothesis is assumed to be generated from the Mallows models. 

The main contributions of this paper are:

1. Devising two uniformity test algorithms accompanied with performance guarantees, for large and general values of $m$.

2. Devising uniformity test algorithms with performance guarantees for central and local DP.  I found the paper well-written, and somewhat motivated. The introduction is comprehensive and the setup and learning problem are clear. The strong side of this paper is its qualitive contribution. In terms of the technical contributions, I read all the proofs and they seem okay; most of the proofs are somewhat standard and straightforward, unless I am missing something. Saying that, Algorithm 2 and its analysis are interesting and more technical. It would be great if the authors could point out the main difficulties encountered in their proofs as well as the novel arguments/techniques used to establish their results.  

I recommend accepting the paper, mainly due to the fact that I find the setting interesting. 
 

 Yes.

Meta-review: This paper considers the problem of testing uniformity of rankings vs a Mallows model, given sample access to these rankings. The paper presents novel algorithms for this problem, both private and non-private. It is a nice contribution to the literature. As one of the reviewers pointed out, it can be helpful to add some remarks in the paper about lower bounds for this problem. 

Decision: accept