Identity testing for Mallows model

1 Introduction . Identity testing of discrete distributions [ 9 , 11 ] is one of the most fundamental problem which consists of answering a yes-or-no question about the closeness of some explicitly given distribution to an unknown distribution from which random samples are observed . This testing problem have been studied in the classical statistical literature [ 16 ] , and several tests have been devised in the asymptotic regime which family hypothesis tests are often referred to as goodness-of-fit tests and are routinely used in data analysis [ 10 ] . In this work , we extend the identity testing setup to a more general domain that includes rankings over m items which can be viewed as a discrete distribution testing problem over a domain with size m ! . As a consequence , identity testing of ranking data without any assumption is not feasible , since lower bound is known for identity testing for discrete distribution [ 26 , 11 ] that is ⌦ ( p n ) where n is the domain size . Therefore we study an important subclass of ranking distributions , which is introduced by [ 20 ] and also known as the exponential family on rankings . The model has two parameters , the central ranking ⇡0 2 Sm and the spread parameter 2 [ 0 , 1 ] . Based on these , the probability of observing a ranking ⇡ 2 Sm is proportional to d ( ⇡ , ⇡0 ) , where d is a ranking distance , such as the number of discordant pairs , a.k.a Kendall ’ s tau distance . There are many applications of the Mallows model in Machine Learning , to name a few , label ranking [ 18 ] , online learning [ 3 ] , recommendation systems [ 29 , 15 ] , clustering [ 23 ] . The focus of this study is to devise identity tests for Mallows model that are scalable with the number of items m. Identity testing is a central problem in analysing output of ranking systems where the goal is to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . decide whether the output ranking data deviates from some expected behaviour , or is biased towards some group of object to be ranked or it is indeed fair ( [ 24 , 20 , 32 ] and see Chapter 3-4 of [ 22 ] ) . The hypothesis testing literature that is related to Mallows model , and more generally to ranking distributions , such as Plackett-Luce [ 27 , 19 ] or Babington-Smith [ 22 ] , is not extensive . In our work , we partially fill this gap by devising hypothesis tests for the Mallows model . We focus on identity testing problem , where we assume that the ground truth parameters of a Mallows model are given ; this corresponds to the null hypothesis . Our goal is to decide whether the data we observe is generated by the null model or by an alternative one . We show that this kind of goodness-of-fit testing task can be tackled in the traditional asymptotic setting as well as in non-asymptotic scenario in an efficient way regardless whether the central ranking is known . We can summarize our main results as follows : • We devise a Uniformly Most Powerful Unbiased ( UMPU ) test for asymptotic case to test spread parameter with known central ranking ⇡0 . • We show that , in general , the UMPU test can be converted into a sample-optimal non-asymptotic test which result may be of independent interest . Based on this result , we come up with an optimal non-asymptotic test , however it is hard to apply even for medium sized data . • Next , we propose a non-asymptotic test that is optimal for small and easy to compute . • We also consider the case when the central ranking ⇡0 is not given , but needs to be estimated , and devise tests for both testing setups . • We demonstrate the versatility of our algorithm running with large m on synthetic data , and we show that for large m > 70 to test the spread parameter , only one single sample is enough . The paper is organized as follows . Related work is presented in Section 2 . Then , we recall the Mallows model and introduce notations . In Section 4 , we describe the asymptotic and non-asymptotic testing setups and present our result on the relation of these two settings . In Section 5 , we present our algorithms for testing the spread parameter . In Section 6 , we present tests where both parameter of Mallows model are tested . Experiments are in Section 7 and finally conclude the paper in Section 8 . 2 Related work . The asymptotic testing has a long history [ 16 ] dating back to Pearson ’ s fundamental work . To come up with a asymptotic test for Mallows model is more challenging from a computational point of view than methodological point of view . Testing ranking distribution can be viewed as multinomial testing problem and thus applied , for example , [ 8 ] , however the domain size is m ! , so this approach becomes hard to apply even for small m. The non-asymptotic testing has also long history [ 9 , 6 , 5 ] and references therein , including testing for discrete data . Nevertheless , sample optimal tests for wide range of parameter had been only devised recently [ 11 ] which was further strengthened in [ 31 ] by showing that this lower bound is instance optimal . Those results imply a sample complexity of order ⇥ ( p m ! / '' 2 ) in our ranking setting if we do not exploit the structure induced by Mallows model . In the non-asymptotic setting , testing and learning are related problems . There are several so-called “ testing by learning ” algorithm which first learns a model , and then decides whether the learnt model is close to the null model or far enough from it which requires a tight lower bound on the total variation distance . Optimal learning , however , does not necessarily imply optimal testing in general , as it is the case for identity testing of discrete distributions ( see Subsection 6.6 of [ 5 ] for further discussion ) . Nevertheless , it turned out that for identity testing of Mallows model this is indeed to be the case for a wide range of parameters , since interestingly , upper bound for sample complexity of optimal parameter learning of Mallows model requires asymptotically as many samples as it is needed to show that two models are far from each other in terms of total variation distance in a certain parameter regime . Optimal learning is recently devised for Mallows model by [ 4 ] which result we rely on . The testing literature is very limited for ranking distribution in general . Mallows came up with an approximate solution , in [ 20 , Section 11 ] , to test uniformity against a single parameter Mallows model in the non-asymptotic setting . His approach relies on a normal approximation of the sufficient statistic . Cohen and Mallows [ 7 ] presents a goodness of fit test for ranking data in a way the ranking data is handled as a sample from a discrete distribution with m ! parameters . That is why their analysis is restricted to m = 3 and m = 4 . In addition to this , the authors presented a goodness of fit test for pairwise marginals of various ranking models for larger m including Mallows model , Placket-Luce and the Thurstonian model , but their focus was to decide which model fits better to the data in terms of pairwise marginals . Whereas we were interested in testing the parameters of Mallows model with large m values . There are some recent paper for testing ranking data , but these works are not related to parametric ranking models . For example , [ 21 , 17 ] uses statistics based on permutation kernel functions for testing the equality of two ranking distributions in the asymptotic setting . Finally , there is a very recent paper which testing identity of preference matrices which are marginals of ranking distributions [ 28 ] . 3 Preliminaries and Notation . Single Parameter Mallows Model . The Mallows model or , more specifically , Mallows - distribution is a parametrized , distance-based probability distribution that belongs to the family of exponential distributions R = { M , ⇡ | 2 [ 0 , 1 ] , ⇡ 2 Sm } with probability mass function p , ⇡0 ( ⇡ ) = d ( ⇡ , ⇡0 ) /Z ( , ⇡0 ) where and ⇡0 are the parameters of the model : ⇡0 2 Sm is the location parameter also called center ranking and 2 [ 0 , 1 ] the spread parameter . Moreover , d ( · , · ) is a distance metric on permutations , which for our paper will be the Kendall tau distance , that is , the number of discordant item pairs dK ( ⇡ , ⇡0 ) = P 1i < jm I { ( ⇡ ( i ) ⇡ ( j ) ) ( ⇡0 ( i ) ⇡0 ( j ) ) < 0 } . The normalization factor in the definition of the model is equal to Z ( , ⇡0 ) = P ⇡2Sm p , ⇡0 ( ⇡ ) . When the distance metric d is the Kendall tau distance we have the identity Z ( , ⇡0 ) = Z ( ) =Qm 1 i=1 Pi j=0 j . Observe that the family of distributions as stated is not an exponential family because of the location parameter ⇡0 . If we fix the permutation parameter then the family R ( ⇡0 ) = { M , ⇡0 | 2 [ 0 , 1 ] } is an exponential family with natural parameter ✓ = ln . From now on , if we use the neutral parametrization , we shall write ✓ , i.e . Z ( ✓ ) = Qm 1 i=1 Pi j=0 e j✓ . The log partition function is denoted by ↵ ( ✓ ) = logZ ( ✓ ) . In Appendix A , we review the basic properties of exponential families which we use in our work . Metrics between distributions . Let p , q be two probability measures in the discrete probability space ( ⌦ , A ) then the total variation distance between p and q is defined as dTV ( p , q ) = 1 2 P x2⌦ |p ( x ) q ( x ) | = maxA2A |p ( A ) q ( A ) | , and the KL-divergence between p and q is defined as DKL ( p||q ) = P x2⌦ p ( x ) ln ⇣ p ( x ) q ( x ) ⌘ . 4 Testing ranking distributions . We shall consider two types of identity tests : asymptotic and non-asymptotic tests . In the asymptotic case , a set of observations and a significance level ↵ > 0 are given in advance , and then our goal is to come up with a test that maximizes the power , i.e . the probability of rejection , if the alternative hypothesis is true , subject to the given level of significance ↵ , i.e . the probability of rejection must be below or equal to ↵ if the null hypothesis is true [ 16 ] . More concretely , assume that we are given a parametric family of ranking distribution R = { M✓|✓ 2 ⌦ } where ⌦ denotes the set of parameters . The observation consists of n rankings Dn = { ⇡1 , . . . , ⇡n } from a ranking distribution M. The null hypothesis is H0 : M 2 R0 where R0 ⇢ R. As an alternative hypothesis , we consider H1 : M 2 R1 ( ⇢ R ) such that R0 \R1 = ; . Then the test is a function f : SnM 7 ! { 0 , 1 } where 0 corresponds to the acceptance , and 1 to the rejection , such that probability of rejection E [ f ( Dn ) ] ↵ whenever M 2 R0 where the expectation is with respect to M. Our goal is to find a test f for which the power f ( M ) = E [ f ( Dn ) ] for all M 2 R1 is as large as possible . We will deal with randomized test so we consider tests in the form of f : SnM 7 ! [ 0 , 1 ] where the output means the probability of rejection . A test f is called most powerful for a given ↵ and M0 2 R1 , if there is no test f 0 such that f ( M0 ) < f 0 ( M0 ) . A test is uniformly most powerful ( UMP ) if it is most powerful for any distribution from the alternative hypothesis class . Furthermore , a test is unbiased , if supM2R0 f ( M ) ↵ infM2R1 f ( M ) . The unbiased UMP test is referred to as UMPU test . In case of non-asymptotic setup , the input is a tolerance parameter `` > 0 and significance parameter 2 ( 0 , 1 ) , and we assume that the tester has sample access to the unknown distribution M. An ( `` , ) non-asymptotic testing algorithm outputs a sample size n and a test function f : SnM 7 ! { 0 , 1 } such that , generating Dn from M , we have the following guaranties for f : 1. if the null hypothesis H0 is true , then it outputs reject ( f ( Dn ) = 1 ) with probability at most , i.e . E [ f ( Dn ) ] 2. if M 2 R1 such that dTV ( M , R0 ) > `` , then it outputs reject ( f ( Dn ) = 1 ) with probability at least 1 where dTV ( M , Ri ) = infM02Ri dTV ( M , M0 ) Given `` and , we say that a testing algorithm is sample-optimal for R0 versus R1 , if no algorithm can have the same confidence guarantee using less samples . If a testing algorithm sample-optimal for any `` and , we say that it is uniformly sample-optimal ( USO ) . One can show that the existence of UMPU test implies the existence of sample-optimal non-asymptotic test as follows with a particular ( `` , ) . Proposition 4.1 If there exists a UMPU test fn with a given significance level ↵ for R0 versus R1 for sample size n , n = 1 , 2 , . . . , then one can define a sample-optimal non-asymptotic testing algorithm that outputs n and f on input = ↵ and '' f = sup M2R1\R ( f ) dTV ( M , R0 ) ( 1 ) where R ( f ) = { M 2 R1 : f ( M ) 1 } . The proof of Proposition 4.1 is deferred to Appendix B . The opposite direction does not hold , since a non-asymptotic test does not guarantee anything for any M 2 R1 such that dTV ( M , R0 ) `` .

Summary: This paper is concerned with the hypothesis testing of Mallows ranking models, which is parametrized by a central ranking and a spread parameter. The authors proposed two different methods: one based on the classical Type I error, and the other based on an optimal learning approach. The authors presented a complete theory regarding testing Mallows rankings, and also provided empirical results to corroborate the theory.