Privacy Implications of Shuffling

1 INTRODUCTION . Differential Privacy ( DP ) and its local variant ( LDP ) are the most commonly accepted notions of data privacy . LDP has the significant advantage of not requiring a trusted centralized aggregator , and has become a popular model for commercial deployments , such as those of Microsoft ( Ding et al. , 2017 ) , Apple ( Greenberg , 2016 ) , and Google ( Erlingsson et al. , 2014 ; Fanti et al. , 2015 ; Bittau et al. , 2017b ) . Its formal guarantee asserts that an adversary can not infer the value of an individual ’ s private input by observing the noisy output . However in practice , a vast amount of public auxiliary information , such as address , social media connections , court records , property records , income and birth dates ( La Corte , 2019 ) , is available for every individual . An adversary , with access to such auxiliary information , can learn about an individual ’ s private data from several other participants ’ noisy responses . We illustrate this as follows . Problem . An analyst runs a medical survey in Alice ’ s community to investigate how the prevalence of a highly contagious disease changes from neighborhood to neighborhood . Community members report a binary value indicating whether they have the disease . Next , consider the following two data reporting strategies . Strategy 1 . Each data owner passes their data through an appropriate randomizer ( that flips the input bit with some probability ) in their local devices and reports the noisy output to the untrusted data analyst . Strategy 2 . The noisy responses from the local devices of each of the data owners are collected by an intermediary trusted shuffler which dissociates the device IDs ( metadata ) from the responses and uniformly randomly shuffles them before sending them to the analyst . Strategy 1 corresponds to the standard LDP deployment model ( for example , Apple and Microsoft ’ s deployments ) . Here the order of the noisy responses is informative of the identity of the data owners – the noisy response at index 1 corresponds to the first data owner and so on . Thus , the noisy responses can be directly linked with its associated device/account ID and subsequently , auxiliary information . This puts Alice ’ s data under the threat of inference attacks . For instance , an adversary1 may know the home addresses of the participants and use this to identify the responses of all the individuals from Alice ’ s household . Being highly infectious , all or most of them will have the same true value ( 0 or 1 ) . Hence , the adversary can reliably infer Alice ’ s value by taking a simple majority vote of her and her household ’ s noisy responses . Note that this does not violate the LDP guarantee since the inputs are appropriately randomized when observed in isolation . Additionally , on account of being 1The analyst and the adversary could be same , we refer to them separately for the ease of understanding . public , the auxiliary information is known to the adversary ( and analyst ) a priori – no mechanism can prevent their disclosure . For instance , any attempts to include Alice ’ s address as an additional feature of the data and then report via LDP is futile – the adversary would simply discard the reported noisy address and use the auxiliary information about the exact addresses to identify the responses of her household members . We call such threats inference attacks – recovering an individual ’ s private input using all or a subset of other participants ’ noisy responses . It is well known that protecting against inference attacks that rely on underlying data correlations is beyond the purview of DP ( Kifer & Machanavajjhala , 2014 ; Tschantz et al. , 2020 ) . Strategy 2 corresponds to the recently introduced shuffle DP model , such as Google ’ s Prochlo ( Bittau et al. , 2017b ) . Here , the noisy responses are completely anonymized – the adversary can not identify which LDP responses correspond to Alice and her household . Under such a model , only information that is completely order agnostic ( i.e. , symmetric functions that can be computed over just the bag of values , such as aggregate statistics ) can be extracted . Consequently , the analyst also fails to accomplish their original goal as all the underlying data correlation is destroyed . Thus , we see that the two models of deployment for LDP present a trade-off between vulnerability to inference attacks and scope of data learnability . In fact , as demonstrated in Kifer & Machanavajjhala ( 2011 ) , it is impossible to defend against all inference attacks while simultaneously maintaining utility for learning . In the extreme case that the adversary knows everyone in Alice ’ s community has the same true value ( but not which one ) , no mechanism can prevent revelation of Alice ’ s datapoint short of destroying all utility of the dataset . This then begs the question : Can we formally suppress specific inference attacks targeting each data owner while maintaining some meaningful learnability of the private data ? Referring back to our example , can we thwart attacks inferring Alice ’ s data using specifically her households ’ responses and still allow the medical analyst to learn its target trends ? Can we offer this to every data owner participating ? In this paper , we strike a balance and propose a generalized shuffle framework that meets the utility requirements of the above analyst while formally protecting data owners against inference attacks . Our solution is based on the key insight : the order of the data acts as the proxy for the identity of data owners as illustrated above . The granularity at which the ordering is maintained formalizes resistance to inference attacks while retaining some meaningful learnability of the private data . Specifically , we guarantee each data owner that their data is shuffled together with a carefully chosen group of other data owners . Revisiting our example , consider uniformly shuffling the responses from Alice ’ s household and her immediate neighbors . Now an adversary can not use her household ’ s responses to predict her value any better than they could with a random sample of responses from this group . In the same way that LDP prevents reconstruction of her datapoint using specifically her noisy response , this scheme prevents reconstruction of her datapoint using specifically her households ’ responses . The real challenge is offering such guarantees equally to every data owner . Bob , Alice ’ s neighbor , needs his households ’ responses shuffled in with his neighbors , as does Luis who is a neighbor of Bob but not of Alice . Thus , we have n data owners with n distinct , overlapping groups . Our scheme supports arbitrary groupings ( overlapping or not ) , introducing a diverse and tunable class of privacy/utility trade-offs which is not attainable with either LDP or uniform shuffling alone . For the above example , our scheme can formally protect each data owner from inference attacks using specifically their household , while still learning how disease prevalence changes across the neighborhoods of Alice ’ s community . This work offers two key contributions to the machine learning privacy literature : • Novel privacy guarantee . We propose a novel privacy definition , dσ-privacy that captures the privacy of the order of a data sequence ( Sec . 4.2 ) and formalizes the degree of resistance against inference attacks ( Sec . 4.3 ) . dσ-privacy allows assigning an arbitrary group , Gi , to each data owner , DOi , i ∈ [ n ] . For instance , the groups can represent individuals in the same age bracket , ‘ friends ’ on social media , or individuals living in each other ’ s vicinity ( as in case of Alice in our example ) . Recall that the order is informative of the data owner ’ s identity . Intuitively , dσ-privacy protects DOi from inference attacks that arise from knowing the identity of the members of their group Gi ( Sec . 4.3 ) . Additionally , this grouping determines a threshold of learnability – any learning that is order agnostic within a group ( disease prevalence in a neighborhood – the data analyst ’ s goal in our example ) is utilitarian and allowed ; whereas analysis that involves identifying the values of individuals within a group ( disease prevalence within specific households – the adversary ’ s goal ) is regarded as a privacy threat and protected against . See Fig . 1 for a toy demonstration of how our guarantee allows tuning the granularity at which trends can be learned . • Novel shuffle framework . We propose a novel mechanism that shuffles the data systematically and achieves dσ-privacy . This provides a generalized shuffle framework that interpolates between no shuffling ( LDP ) and uniform random shuffling ( shuffle model ) in terms of protection against inference attacks and data learnability . 2 RELATED WORK . The shuffle model of DP ( Bittau et al. , 2017a ; Cheu et al. , 2019 ; Erlingsson et al. , 2019 ) differs from our scheme as follows . These works ( 1 ) study DP benefits of shuffling whereas we study the inferential privacy benefits , and ( 2 ) only study uniformly random shuffling where ours generalizes this to tunable , non-uniform shuffling ( see App . A.15 ) . A steady line of work has studied inferential privacy ( Kasiviswanathan & Smith , 2014 ; Kifer & Machanavajjhala , 2011 ; Ghosh & Kleinberg , 2016 ; Dalenius , 1977 ; Dwork & Naor , 2010 ; Tschantz et al. , 2020 ) . Our work departs from those in that we focus on local inferential privacy and do so via the new angle of shuffling . Older works such as k-anonymity ( Sweeney , 2002 ) , l-diversity Machanavajjhala et al . ( 2007 ) , Anatomy ( Xiao & Tao , 2006 ) and others ( Wong et al. , 2010 ; Tassa et al. , 2012 ; Xue et al. , 2012 ; Choromanski et al. , 2013 ; Doka et al. , 2015 ) have studied the privacy risk of non-sensitive auxiliary information or ‘ quasi identifiers ’ . These works ( 1 ) focus on the setting of dataset release , whereas we focus on dataset collection , and ( 2 ) do not offer each data owner formal inferential guarantees , whereas we do . The De Finetti attack ( Kifer , 2009 ) shows how shuffling schemes are vulnerable to inference attacks that correlate records to recover the original permutation of sensitive attributes . A strict instance of our privacy guarantee can thwart such attacks ( at the cost of no utility , App . A.3 ) . 3 BACKGROUND . Notations . Boldface ( such as x = 〈x1 , · · · , xn〉 ) denotes a data sequence ( ordered list ) ; normal font ( such as x1 ) denotes individual values and { · } represents a multiset or bag of values . 3.1 LOCAL DIFFERENTIAL PRIVACY . The local model consists of a set of data owners and an untrusted data aggregator ( analyst ) ; each individual perturbs their data using a LDP algorithm ( randomizers ) and sends it to the analyst . The LDP guarantee is formally defined as Definition 3.1 . [ Local Differential Privacy , LDP Warner ( 1965 ) ; Evfimievski et al . ( 2003 ) ; Kasiviswanathan et al . ( 2008 ) ] A randomized algorithmM : X → Y is -locally differentially private ( or -LDP ) , if for any pair of private values x , x′ ∈ X and any subset of output , Pr [ M ( x ) ∈ W ] ≤ e · Pr [ M ( x′ ) ∈ W ] ( 1 ) The shuffle model is an extension of the local model where the data owners first randomize their inputs . Additionally , an intermediate trusted shuffler applies a uniformly random permutation to all the noisy responses before the analyst can view them . The anonymity provided by the shuffler requires less noise than the local model for achieving the same privacy .

Summary: The paper proposes a privacy model in between the guarantees of local DP and shuffle DP. It does so by shuffling each user with a group of "similar" users as opposed to all users. As a result, it achieves better utility than LDP. The group that the user belongs to is publicly available, hence the adversary can tell that the user is one of k particular users but not which one. Hence, the "shuffler" instead of choosing a random uniform  permutation chooses a permutation that satisfies this per-group shuffle. It is intractable to choose a permutation with these properties, hence the authors propose a heuristic method for choosing such permutation.