SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Abhishek Singh<sup>1</sup>, Praneeth Vepakomma<sup>1</sup>, Vivek Sharma<sup>1,2,3,*</sup>, Ramesh Raskar<sup>1</sup>  
<sup>1</sup>MIT Media Lab, <sup>2</sup>MGH, Harvard Medical School, <sup>3</sup>Sony AI
Cloud-based machine learning inference is an emerging paradigm where users query by sending their data through a service provider who runs an ML model on that data and returns back the answer. Due to increased concerns over data privacy, recent works have proposed Collaborative Inference (CI) to learn a privacy-preserving encoding of sensitive user data before it is shared with an untrusted service provider. Existing works so far evaluate the privacy of these encodings through empirical reconstruction attacks. In this work, we develop a new framework that provides formal privacy guarantees for an arbitrarily trained neural network by linking its local Lipschitz constant with its local sensitivity. To guarantee privacy using local sensitivity, we extend the Propose-Test-Release (PTR) framework to make it tractable for neural network queries. We verify the efficacy of our framework experimentally on real-world datasets and elucidate the role of Adversarial Representation Learning (ARL) in improving the privacy-utility trade-off. The source code and other details are available at tremblierz.github.io/posthoc.
While training ML models privately has seen tremendous progress[1, 45, 10, 26] in the last few years, protecting privacy during the inference phase remains elusive as these models get deployed by cloud-based service providers. Cryptographic techniques[43, 31, 40, 27] address this challenge by computing over encrypted data. To combat the high computational cost of encryption techniques, several recent works [66, 65, 5, 56, 57] have proposed Collaborative Inference (CI) – an alternate paradigm where users share a lower dimensional embedding of raw data where task-irrelevant information is suppressed. However, CI techniques are currently incompatible with formal privacy frameworks and their evaluation is currently based on empirical reconstruction attacks. The central issue in CI is the usage of Deep Neural Networks (DNNs) making it unsuitable for formal and worst-case privacy analysis. In this work, we take the first step towards a formal privacy framework for CI that will enable a more rigorous evaluation and a reliable understanding of the CI techniques.
The key aspect of any CI algorithm is an obfuscator (typically a DNN), which is trained to encode a user's private data such that an attacker can not reconstruct the original data from its encoding. Hence, CI techniques typically evaluate the privacy of their representations by empirically measuring the information leakage using a proxy adversary. However, existing works [57, 66, 18, 56] show that a proxy adversary's performance as a measure of protection could be unreliable because a future adversary can come up with a stronger attack. Alternately a few existing CI techniques have used theoretical tools [20, 69, 4, 68, 62, 5, 39] for measuring information leakage empirically. However, most of these works analyze specific obfuscation techniques and lack formal privacy definitions. Therefore, we first introduce a privacy definition applicable to CI by adopting a different threat model from that of differential privacy (DP)[12] because (as we explain in detail in Sec 2) protecting membership inference is at odds with achieving a non-trivial privacy-utility trade-off in CI.
*Part of this work was carried out while he was at MIT and MGH, Harvard Medical School.
Our threat model for the reconstruction attack is motivated by the intuition that the obfuscator discloses coarse-grained and filters out fine-grained information for preventing reconstruction. We hypothesize that to do so, the obfuscator should act as a contractive mapping, and as a result, increases the stability of the (obfuscation) function in the local neighborhood of data. We formalize this intuition by introducing a privacy definition in the metric space of data and experimentally validate it in Sec 5. Our privacy definition is an instantiation of metric-DP by Chatzikokolakis et al. Instead of evaluating the global Lipschitz constant of DNNs, we evaluate the Lipschitz constant only in the local neighborhood of the user's sensitive data. We then extend the Propose-Test-Release (PTR)[11] framework to formalize our local neighborhood based measurement of the Lipschitz constant.
We adopt the PTR framework due to its posthoc and data-adaptive nature – as the privacy of encoding is evaluated in the neighborhood of the private user input and after the obfuscator has been trained. This approach is different from conventional privacy mechanisms where the mechanism is fixed and the perturbation does not depend on data. Existing seminal works in data-adaptive privacy such as Propose-Test-Release (PTR) [11], smooth sensitivity [41] and Lipschitz extensions [48] have exploited this posthoc approach to enhance the utility of the mechanisms. Data adaptive mechanisms are an important area of study as quoted in a recent workshop report on the challenges in Differential Privacy [9] - "Going data-adaptive is a crucial step in bringing DP algorithms to an acceptable level of utility in applications". In the context of our work – the benefit of using the PTR framework is two-fold i) Utility - the obfuscator is expected to behave better at samples that match the data distribution from its training dataset are sampled, and ii) Computation - instead of computing global sensitivity, we only need to compute sensitivity in the small neighborhood around private data points. While the original PTR is intractable due to a well-known [9] expensive computation step, we design a new formulation that substitutes the expensive computation step with a tractable alternate.
Computational tractability is generally the main concern that makes CI incompatible with formal privacy because giving worst-case guarantees for obfuscator is generally difficult. While training and convergence guarantees for DNNs remain elusive, we find that the certified robustness community has made rapid progress in the past few years in giving worst-case guarantees during the inference stage of DNNs. Specifically, we utilize recent efficient formulations for the Lipschitz constant computation of DNNs [25, 55]. We measure the stability of the learned obfuscator model using the Lipschitz constant and link it with the local sensitivity under our privacy definition. We bridge the gap between the Lipschitz constant and our extended PTR framework by proposing a binary search algorithm that computes this Lipschitz constant multiple times in the neighborhood of sensitive data. This bridge makes our work amenable to any differentiable obfuscator with ReLU non-linearity.
The scope of our paper is to provide privacy guarantees against reconstruction attacks for existing CI techniques, i.e., our goal is not to develop a new CI algorithm but rather to develop a formal privacy framework compatible with existing algorithms. A majority of the CI techniques protect either a sensitive attribute or reconstruction of the input. We only consider sensitive input in this work. Furthermore, we only focus on protecting the privacy of data during the inference stage, and assume that ML models can be trained privately. Our privacy definition and the mechanism are built upon  $d_{\mathcal{X}}$ -privacy [8] and PTR [11]. Existing instantiations of  $d_{\mathcal{X}}$ -privacy include geoindistinguishability [2] and location-dependent privacy[32] that share a similar goal as ours of sharing coarse-grained information. Our work differs because we use DNN queries and high-dimensional datasets. We refer the reader to the supplementary for a detailed literature review.
In Sec 2 we begin with the preliminaries of DP and its variant for metric spaces. Then, we introduce the privacy definition relevant for CI in Sec 3. Next, we design our framework by extending PTR and proving its privacy guarantees in Sec 4. In Sec 5 we experimentally demonstrate the feasibility of our framework. We now give the summary of our contributions:
1. Proposing a privacy definition that formalizes reconstruction privacy in the context of CI. Protecting against membership inference (meaningfully) is not possible in CI and hence traditional DP cannot be applied directly as discussed in Sec 2.
2. Extending the (PTR) algorithm to guarantee privacy using the local Lipschitz constant. The standard PTR can not be applied due to its intractability for queries beyond median and mode (see - Page 152, 2nd paragraph of Dwork et al. and Sec 4).
3. Experiments 1) to verify efficacy for both utility and computability, 2) ablation on different design choices, 3) empirical reconstruction attacks, and 4) identifying the role ARL techniques play in improving privacy-utility trade-off in CI.
Figure 1: Posthoc Privacy framework: We project a high dimensional data instance to a lower dimensional embedding. The goal of the embedder is to measure a semantically relevant distance between different instances. The embedding is fed to the Obfuscator that compresses similar inputs in a small volume. In traditional ARL, the obfuscated instance is shared with the untrusted service provider without any formal privacy guarantee. In this work, by analyzing the stability of the obfuscator network, we perturb the obfuscated instance to provide a formal privacy guarantee.
Differential privacy[12] is a widely used framework for answering a query,  $f$ , on a dataset  $\mathbf{x} \in \mathcal{X}$  by applying a mechanism  $\mathcal{M}(\cdot)$  such that the probability distribution of the output of the mechanism  $\mathcal{M}(\mathbf{x})$  is similar regardless the presence or absence of any individual in the dataset  $\mathbf{x}$ . More formally,  $\mathcal{M}$  satisfies  $(\epsilon, \delta)$ -DP if  $\forall \mathbf{x}, \mathbf{x}' \in \mathcal{X}$  such that  $d_H(\mathbf{x}, \mathbf{x}') \leq 1$ , and for all (measurable) output  $S$  over the range of  $\mathcal{M}$
$$
\mathbb {P} (\mathcal {M} (\mathbf {x}) \in S) \leq e ^ {\epsilon} \mathbb {P} (\mathcal {M} (\mathbf {x} ^ {\prime}) \in S) + \delta ,
$$
where  $d_H$  is the hamming distance. This definition is based on a trusted central server model, where a trusted third party collects sensitive data and computes  $\mathcal{M}(\mathbf{x})$  to share with untrusted parties. In local-DP[28], this model has been extended such that each user shares  $\mathcal{M}(\mathbf{x})$ , and the service provider is untrusted. Our threat model is the same as local DP. However, unlike local DP, data is not aggregated over multiple individuals. Specifically, every sample is evaluated independently by the service provider and there is no aggregation involved. For ex.- a user shares a face image to receive an age prediction from the service provider, here the answer to the query depends exactly on the user's input. With traditional local DP, it is "impossible" to achieve good utility and privacy at the same time because any two samples (no matter how different) are neighboring databases, more formally,  $d_H(\mathbf{x},\mathbf{x}') \leq 1$ ,  $\forall \mathbf{x},\mathbf{x}' \in \mathcal{X}$  for ML inference. Informally, this notion of neighboring databases under the standard DP definition would imply that the outcome for any two samples should be almost indistinguishable no matter how different the samples are. This privacy definition could be too restrictive for our ML inference application where the data instance necessarily needs a certain degree of distinguishability to obtain utility from the service provider. This observation is formalized in the impossibility result of instance encoding[7] for private learning. To subside this fundamental conflict between the privacy definition and our application, we look at the definition of  $d_{\mathcal{X}}$ -privacy[8] that generalizes the DP definition to a general distance metric beyond hamming distance as follows:
$$
\mathbb {P} (\mathcal {M} (\mathbf {x}) \in S) \leq e ^ {d _ {\mathcal {X}} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right)} \mathbb {P} (\mathcal {M} \left(\mathbf {x} ^ {\prime}\right) \in S), \tag {1}
$$
here  $d_{\mathcal{X}}(\mathbf{x}, \mathbf{x}^{\prime})$  is a function that gives a level of indistinguishability between two datasets  $\mathbf{x}$  and  $\mathbf{x}^{\prime}$ . DP can be viewed as a special case of  $d_{\mathcal{X}}$ -privacy by keeping  $d_{\mathcal{X}}(\mathbf{x}, \mathbf{x}^{\prime}) = \epsilon d_{H}(\mathbf{x}, \mathbf{x}^{\prime})$ . Choosing a different distance metric yields a stronger or weaker privacy guarantee. Due to the impossibility of achieving both privacy and utility with hamming distance, we move towards a weaker privacy definition by only focusing on achieving privacy against reconstruction attacks instead of membership inference. Intuitively, we choose a distance metric that provides indistinguishability only within a neighborhood of similar looking samples.
3 Privacy Definition and Threat Model
To formalize reconstruction privacy, we hypothesize that semantically similar points are close to each other on a data manifold, i.e. semantically similar samples are closer in a space where distance is defined as geodesic on the data manifold. Therefore, one way to bound the reconstruction of  $\mathbf{x}$
is by making it indistinguishable among semantically similar points. The extent of reconstruction privacy would therefore depend upon the radius of the neighborhood. We formalize it with a privacy parameter  $R$  that allows a user to control how big this indistinguishable neighborhood should be.
Definition 1. A mechanism  $\mathcal{M}$  satisfies  $(\epsilon, \delta, R)$ -reconstruction privacy if  $\forall \mathbf{x}, \mathbf{x}' \in \mathcal{X}$  s.t.  $d_{\mathcal{X}}(\mathbf{x}, \mathbf{x}') \leq R$  and  $S \subseteq \text{Range}(\mathcal{M})$
$$
\mathbb {P} (\mathcal {M} (\mathbf {x}) \in S) \leq e ^ {\epsilon} \mathbb {P} (\mathcal {M} \left(\mathbf {x} ^ {\prime}\right) \in S) + \delta . \tag {2}
$$
Note that the above equation is exactly the same as  $(\epsilon, \delta)$ -DP except for the definition of neighboring databases. In this way, our privacy definition can be seen as a mix of standard DP and  $d_{\mathcal{X}}$ -privacy.
3.1 Threat and Attack model
Our threat and attack model is the same as previous collaborative inference works [66, 65]. Specifically, we focus on attackers that aim to reconstruct the original sample from its encoding. That is, given an encoding  $\mathcal{M}(\mathbf{x})$  obtained from a sample  $\mathbf{x}$ , an attacker will attempt to recover the original sample  $(\mathbf{x})$ . Our threat model is the same as local DP. Specifically, all side information including access to the obfuscator model, embedder model, and the framework in Fig 1 is known to the adversary and it can interact with them as much as it wants. The attacker can use this information to improve its attack algorithm. We (informally) note that an attacker always has a prior about raw data  $\mathbb{P}(\mathbf{x})$  and it will update its belief based on the posterior  $\mathbb{P}(\mathbf{x}|\mathcal{M}(\mathbf{x}))$ . The goal of our privacy mechanism is to ensure that the distance between the prior and posterior distribution is bounded. Our definition indeed enjoys such an interpretation due to being an instantiation of  $d_{\mathcal{X}}$ -privacy. A more formal description of this Bayesian formulation is given by Chatzikokolakis et al. in Theorem 4. We emulate such attackers in Sec 5 by training an ML model on a dataset of  $\mathcal{M}(\mathbf{x})$  as input and  $\mathbf{x}$  as output, then we evaluate its reconstruction error on encodings of previously unseen  $\mathcal{M}(\mathbf{x})$ .
3.2 Comparison with Differential Privacy
Conceptually, the usage of hamming distance in DP for neighboring databases provides a level of protection at the sample level. Such a privacy requirement can be at odds with the goal of prediction that necessarily requires discrimination between samples belonging to different concept classes. We relax this dichotomy by changing the distance metric to account for reconstruction privacy. Intuitively, an attacker can only obtain an accurate reconstruction of data up to a neighborhood in the space of data. Therefore, the size of the neighborhood is a privacy parameter  $R$  such that a higher value of  $R$  provides higher privacy. This privacy parameter is equivalent to the group size [12] used often in the DP literature. By default, this value is kept at 1 in DP but can be kept higher if correlated individuals (multiple samples, family) are present in a database instead of a single individual. We state the equivalence between the group privacy definition and standard DP informally -
Lemma 2.2 in [60]: Any  $(\epsilon, \delta)$ -differentially private mechanism is  $(R\epsilon, Re^{(R-1)\epsilon}\delta)$ -differentially private for groups of size  $R$ .
This lemma also applies to our proposed definition. However, we emphasize that privacy parameters of  $(\epsilon, \delta)$ -DP mechanism can not be compared trivially with a  $(\epsilon, \delta, R)$ -reconstruction privacy mechanism because same value of  $\epsilon$  and  $\delta$  provide different levels of protection due to different definitions of neighboring databases. We experimentally demonstrate this claim in Sec. 5.
3.3 Choice of  $d_{\mathcal{X}}(\mathbf{x},\mathbf{x}^{\prime})$  and its impact on the privacy guarantees offered
Our privacy definition is an instantiation of  $d_{\mathcal{X}}(\mathbf{x}, \mathbf{x}^{\prime})$  that permits a broad class of distance metrics for the privacy definition. Since the goal of our work is to protect against reconstruction -  $\ell_1$  or  $\ell_2$  distance is a reasonable choice. However, the space over which  $\ell$  norm should be considered is a challenging question. Existing papers have used metrics such as  $\ell_1$  or  $\ell_2$  norm in the raw data space [21, 6, 17, 3] for guaranteeing reconstruction privacy when training ML models. However, for high-dimensional datasets typically considered in CI such as images, guaranteeing reconstruction in ambient dimensions can lead to unintended privacy guarantee. In our work, we consider pre-processing the samples by projecting them into an embedding space, where samples that are semantically closer and farther have  $\ell$  norm smaller and higher respectively. We evaluate the privacy-utility trade-off in our experiments for both ambient space and embedding space.
4 Privacy Mechanism
Our goal is to design a framework that can provide a formal privacy guarantee for encodings informally privatized in CI. However, CI algorithms use non-linear neural networks trained on non-convex objectives making it difficult to perform any worst-case analysis. Therefore, we take a posthoc approach where we reason about privacy after the model is trained. Specifically, we apply the PTR mechanism by [11]. Applying PTR directly to our query (ARL) is not computationally feasible because PTR requires estimating local sensitivity at multiple points whereas evaluating the local sensitivity of a neural network query is not even feasible at a single point. Therefore, we design a tractable variant of PTR that utilizes the local Lipschitz constant estimator to compute privacy related parameters. We refer the reader to supplementary for background on the Lipschitz constant estimation and PTR.
CI algorithms consist of three computational blocks in the training stage: 1) obfuscator  $(f(\cdot))$  that generates an (informally private) representation  $(\tilde{\mathbf{z}})$  of data, 2) proxy adversary that reconstructs the data from the representation produced by the obfuscator, and 3) classifier that performs the given task using the obfuscated representation. The classifier and proxy adversary are trained to minimize the task loss and reconstruction loss, respectively. The obfuscator is trained to minimize the task loss but maximize the reconstruction loss. This setup results in a min-max optimization where the trade-off between task performance and reconstruction is controlled by a hyper-parameter  $\alpha$ . Note that a few CI techniques[42, 44, 61] do not require a proxy adversary but still learn an obfuscator model using other regularizers.
At a high level, our framework applies the mechanism  $\mathcal{M}$  such that the final released data  $\hat{\mathbf{z}} = \mathcal{M}(\mathbf{x})$  has a privacy guarantee discussed in Eq 2. Like PTR, we start with a proposal  $(\Delta_{LS}^{p})$  on the upper bound of the local sensitivity of  $\mathbf{x}$ . To test the correctness of  $\Delta_{LS}^{p}$ , we compute the size of the biggest possible neighborhood such that the local Lipschitz constant of the obfuscator network in that neighborhood is smaller than the proposed bound. Next, we privately verify the correctness of the proposed bound. We do not release the data (denoted by  $\perp$ ) if the proposed bound is invalid. Otherwise, we perturb the data using the Laplace mechanism. Next, we discuss the framework and privacy guarantees in more detail.
Global Sensitivity and Lipschitz constant of a query  $f: \mathcal{X} \to \mathcal{Y}$  are related in the  $d_{\mathcal{X}}$ -privacy framework. Global sensitivity of a query  $f(\cdot)$  is the smallest value of  $\Delta$  (if it exists) such that  $\forall \mathbf{x}, \mathbf{x}' \in \mathcal{X}$ ,  $d_{\mathcal{Y}}(f(\mathbf{x}), f(\mathbf{x}')) \leq \Delta d_{\mathcal{X}}(\mathbf{x}, \mathbf{x}')$ . While global sensitivity is a measure over all possible pairs of data in the data domain  $\mathcal{X}$ , local sensitivity  $(\Delta_{LS})$  is defined with respect to a given dataset  $\mathbf{x}$  such that  $\forall \mathbf{x}' \in \mathcal{X}$ ,  $d_{\mathcal{Y}}(f(\mathbf{x}), f(\mathbf{x}')) \leq \Delta_{LS}(\mathbf{x}) d_{\mathcal{X}}(\mathbf{x}, \mathbf{x}')$ . We integrate the notion of similarity in a neighborhood (motivated in Sec 2) by defining the local sensitivity of a neighborhood  $\mathcal{N}(\mathbf{x}, R)$  around  $\mathbf{x}$  of radius  $R$  where  $\mathcal{N}(\mathbf{x}, R) = \{\mathbf{x}' | d_{\mathcal{X}}(\mathbf{x}, \mathbf{x}') \leq R, \forall \mathbf{x}' \in \mathcal{X}\}$ . Therefore, the local sensitivity of query  $f$  on  $\mathbf{x}$  in the  $R$ -neighborhood is defined  $\forall \mathbf{x}' \in \mathcal{N}(\mathbf{x}, R)$  such that
$$
d _ {\mathcal {Y}} \left(f (\mathbf {x}), f \left(\mathbf {x} ^ {\prime}\right)\right) \leq \Delta_ {L S} (\mathbf {x}, R) d _ {\mathcal {X}} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right). \tag {3}
$$
We note that if  $d_{\mathcal{X}}$  is hamming distance and  $R$  is 1 then this formulation is exactly same as local sensitivity in  $\epsilon$ -DP[12]. The equation above can be re-written as:
$$
\Delta_ {L S} (\mathbf {x}, R) = \sup  _ {\mathbf {x} ^ {\prime} \in \mathcal {N} (\mathbf {x}, R)} \frac {d _ {\mathcal {Y}} (f (\mathbf {x}) , f \left(\mathbf {x} ^ {\prime}\right))}{d _ {\mathcal {X}} (\mathbf {x} , \mathbf {x} ^ {\prime})}. \tag {4}
$$
This formulation of local sensitivity is similar to the definition of the local Lipschitz constant. The local Lipschitz constant  $\mathcal{L}$  of  $f$  for a given open neighborhood  $\mathcal{N} \subseteq \mathcal{X}$  is defined as follows:
$$
\mathcal {L} ^ {\alpha , \beta} (f, \mathcal {N}) = \sup  _ {\mathbf {x} ^ {\prime}, \mathbf {x} ^ {\prime \prime} \in \mathcal {N}} \frac {\left| \left| f \left(\mathbf {x} ^ {\prime}\right) - f \left(\mathbf {x} ^ {\prime \prime}\right) \right| \right| _ {\alpha}}{\left| \left| \mathbf {x} ^ {\prime} - \mathbf {x} ^ {\prime \prime} \right| \right| _ {\beta}} \quad \left(\mathbf {x} ^ {\prime} \neq \mathbf {x} ^ {\prime \prime}\right) \tag {5}
$$
We note that while the local sensitivity of  $\mathbf{x}$  is described around the neighborhood of  $\mathbf{x}$ , the Lipschitz constant is defined for every possible pair of points in a given neighborhood. Therefore, in Lemma 4.1 we show that the local Lipschitz in the neighborhood of  $\mathbf{x}$  is an upper bound on the local sensitivity.