SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bounding the Invertibility of Privacy-Preserving Instance Encoding Using Fisher Information
Kiwan Maeng*
Penn State University
kvm6242@psu.edu
Chuan Guo*
chuanguo@meta.com
Sanjay Kariyappa
sanjaykariyappa@gatech.edu
G. Edward Suh
FAIR, Meta & Cornell University
edsuh@meta.com
Privacy-preserving instance encoding aims to encode raw data into feature vectors without revealing their privacy-sensitive information. When designed properly, these encodings can be used for downstream ML applications such as training and inference with limited privacy risk. However, the vast majority of existing schemes do not theoretically justify that their encoding is non-invertible, and their privacy-enhancing properties are only validated empirically against a limited set of attacks. In this paper, we propose a theoretically-principled measure for the invertibility of instance encoding based on Fisher information that is broadly applicable to a wide range of popular encoders. We show that dFIL can be used to bound the invertibility of encodings both theoretically and empirically, providing an intuitive interpretation of the privacy of instance encoding.
Machine learning (ML) applications often require access to privacy-sensitive data. A model that predicts diseases with x-ray scans requires access to patient's x-ray images. Next-word prediction models require access to user's text input that can contain sensitive information [23]. To enable ML applications on privacy-sensitive data, instance encoding ([6]; Figure 1, left) aims to encode data in a way such that it is possible to run useful ML tasks—such as model training and inference—on the encoded data while the privacy of the raw data is preserved. The concept is widespread under many different names: private data sharing [16, 17, 48, 5, 34, 32, 37], learnable encryption [35, 73, 71, 70], split learning [63, 55], split inference [40, 11], and vertical federated learning (vFL; [74, 59, 45]) are all collaborative training/inference schemes that operate on (hopefully) privately-encoded user data.
Unfortunately, the vast majority of existing instance encoding schemes rely on heuristics rather than rigorous theoretical arguments to justify their privacy-enhancing properties. Most existing works [35, 73, 64, 65, 45] claimed that their schemes are non-invertible by simply testing them against certain input reconstruction attacks. However, these schemes may not be private under more carefully designed attacks; in fact, many encoding schemes that were initially thought to be private have been shown to be vulnerable over time [6, 7]. While there exist instance encoding schemes that are based on a rigorous theory, e.g., metric differential privacy (metric-DP; [8, 16, 17, 44]), their theoretical analysis is only applicable to very simple encoders and cannot be applied to more powerful encoders (e.g., DNN-based encoders) that are common in practice [73, 64, 65, 45, 5, 34, 32].
Figure 1: Overview of instance encoding (left), and our bound against different attackers (Attack-ub, Attack-b) for a synthetic dataset (right). The bound adapted from prior work [20] (Bound-ub) only works against a certain adversary (Attack-ub). Our newly proposed bound (Bound-ours) works for any adversary. See Section 3.3.1 for evaluation details.
To fill the gap, we propose a theoretically-principled and easily applicable framework for quantifying the privacy of instance encoding by bounding its invertibility. Our framework uses (diagonal) Fisher information leakage (dFIL; [18, 22, 20])—an information-theoretic measure of privacy with similar properties to differential privacy (DP; [14, 15]). dFIL can be computed for common privacy-enhancing mechanisms and can lower-bound the expected mean squared error (MSE) of an input reconstruction attack. We apply this reasoning to instance encoding and show that dFIL can serve as a good measure for the invertibility of instance encoding, as it can bound the reconstruction MSE against any attacker that tries to reconstruct the private input from the encoding. dFIL can be easily calculated for any encoders that are differentiable and randomized—being applicable to most of the popular encoders with minimal modifications. To the best of our knowledge, our work is the first to theoretically lower-bound the invertibility of instance encoding and use it to design private training/inference systems.
Contributions Our main contributions are as follows:
1. We adapt the result of prior works [18, 20] for instance encoding to show how dFIL can lower bound the MSE of particular input reconstruction attacks (i.e., unbiased attacks) that aim to reconstruct the raw data given the encoding. We show how popular encoders can be modified minimally for dFIL to be applied (Section 3.1).
2. We extend the result of prior works [18, 20] and show that dFIL can lower bound the MSE of any input reconstruction attack (e.g., strong attacks leveraging knowledge of the input prior; Section 3.2). Our extension involves a novel application of the classical van Trees inequality [62] and connecting it to the problem of score matching [57] in distribution estimation.
3. We evaluate the lower bound using different attacks and encoders, and show that dFIL can be used to interpret the invertibility of instance encoding both theoretically and empirically (Section 3.3).
4. We show how dFIL can be used as a practical privacy metric and guide the design of privacy-enhancing training/inference systems with instance encoding (Section 4-5). We show that it is possible to achieve both high (theoretically-justified) privacy and satisfactory utility.
2 Motivation and background
2.1 Instance encoding
Instance encoding refers to the general concept of encoding raw input  $\mathbf{x}$  using an encoding function Enc, such that private information contained in  $\mathbf{x}$  cannot be inferred from its encoding  $\mathbf{e} = \operatorname{Enc}(\mathbf{x})$ , while e maintains enough utility for the downstream ML tasks. The principle behind the privacy-enhancing property of instance encoding is that the function Enc is hard to invert. However, prior works generally justify this claim of non-invertibility based on heuristics rather than rigorous theoretical analysis [64, 65, 45], except for a few cases where the encoder is very simple [16, 17, 44].
Attacks against instance encoding Given an instance encoder Enc, the goal of a reconstruction attack is to recover its input. Formally, given  $\mathbf{e} = \operatorname{Enc}(\mathbf{x})$ , an attack Att aims to reconstruct  $\mathbf{x}$  from  $\mathbf{e}$ :  $\hat{\mathbf{x}} = \operatorname{Att}(\mathbf{e})$ . Such an attack can be carried out in several ways. If Enc is known,  $\hat{\mathbf{x}}$  can be obtained by solving the following optimization [27]:  $\hat{\mathbf{x}} = \arg \min_{\mathbf{x}_0} ||\mathbf{e} - \operatorname{Enc}(\mathbf{x}_0)||_2^2$ . This attack can be further
improved when some prior of the input is known [49, 61]. For instance, images can be regularized with total variation (TV) prior to reduce high-frequency components [49]. Alternatively, if samples from the input distribution can be obtained, a DNN that generates  $\hat{\mathbf{x}}$  from  $\mathbf{e}$  can be trained [53, 27, 12].
Privacy metrics for instance encoding Measuring how much information an observation leaks about a secret quantity it depends on is a classical question that has long been studied in information theory [38, 3, 13, 33], and instance encoding privacy is directly relevant to the stream of works. Prior works proposed metrics providing a strong notion of privacy (e.g., Arimoto's mutual information [3] or Sibson's mutual information [38]). However, it is challenging to apply these metrics to ML and instance encoding which use high-dimensional data [33], because these metrics have to be calculated and optimized over the prior of the dataset, the encodings, and/or their joint distribution, all of which are unknown and hard to accurately model [33].
Differential privacy (DP) [14, 15], one of the most popular frameworks to quantify ML privacy [1], is not suitable for instance encoding [6]. DP guarantees the worst-case indistinguishability of the encoding from two different inputs, which significantly damages the utility [6] (Appendix 7.5). Metric-DP [8]—a weaker notion of DP—is applicable to instance encoding and has been used by several prior works [16, 17, 44, 48, 37]. However, metric-DP is applicable only when the encoder is extremely simple. For example, prior works used pixel-level perturbation (e.g., pixelization, blurring [16, 17]) or a single-layer convolution [44] as an encoder. These extremely-simple encoders are in stark contrast with the complex encoders most practitioners use (e.g., split learning [64], split inference [65], and vFL [45, 59] all uses DNN-based encoders), and results in a significant utility loss (Section 4.2). Applying metric-DP to an arbitrary DNN-based encoder is NP-hard [66]. Unlike these works that focus on indistinguishability, our work directly quantifies non-invertibility, a weaker but still relevant definition of privacy (Section 3).
Due to the limitations in existing theoretical metrics, the vast majority of works in the field are relying on empirical measures to demonstrate their encoder's privacy-enhancing properties. Some [46, 45] simply test their encoders against a limited set of attacks, while others [64, 65, 51] use heuristical privacy metrics without rigorous theoretical arguments. Using heuristical metrics that are not backed by a theoretical interpretation is dangerous, because a seemingly useful metric can actually severely mis-characterize the privacy of a system [38]. It is of both interest and practical importance to develop a privacy metric that is both theoretically-meaningful and widely applicable.
A concurrent work [72] proposed a notion of PAC privacy that can bound the adversary's attack success rate in theory for instance encoding. PAC privacy bound has not yet been evaluated empirically under a realistic attack and use cases, which remains an interesting future work.
2.2 Fisher information leakage
Fisher information leakage (FIL; [18, 22, 20]) is a measure of leakage through a privacy-enhancing mechanism. Let  $\mathcal{M}$  be a randomized mechanism on data sample  $\mathbf{x}$ , and let  $\mathbf{o} \sim \mathcal{M}(\mathbf{x})$  be its output. Suppose the log density function  $\log p(\mathbf{o};\mathbf{x})$  is differentiable w.r.t.  $\mathbf{x}$  and satisfies the following regularity condition:  $\mathbb{E}_{\mathbf{o}}\left[\nabla_{\mathbf{x}}\log p(\mathbf{o};\mathbf{x})|\mathbf{x}\right] = 0$ . The Fisher information matrix (FIM)  $\mathcal{I}_{\mathbf{o}}(\mathbf{x})$  is:
$$
\mathcal {I} _ {\mathbf {o}} (\mathbf {x}) = \mathbb {E} _ {\mathbf {o}} \left[ \nabla_ {\mathbf {x}} \log p (\mathbf {o}; \mathbf {x}) \nabla_ {\mathbf {x}} \log p (\mathbf {o}; \mathbf {x}) ^ {\top} \right]. \tag {1}
$$
Cramér-Rao bound Fisher information is a compelling privacy metric as it directly relates to the mean squared error (MSE) of a reconstruction adversary through the Cramér-Rao bound [41]. In detail, suppose that  $\hat{\mathbf{x}} (\mathbf{o})$  is an unbiased estimate (or reconstruction) of  $\mathbf{x}$  given the output of the randomized private mechanism  $\mathbf{o}\sim \mathcal{M}(\mathbf{x})$ . Then:
$$
\mathbb {E} _ {\mathbf {o}} \left[ \left\| \hat {\mathbf {x}} (\mathbf {o}) - \mathbf {x} \right\| _ {2} ^ {2} / d \right] \geq \frac {d}{\operatorname {T r} \left(\mathcal {I} _ {\mathbf {o}} (\mathbf {x})\right)}, \tag {2}
$$
where  $d$  is the dimension of  $\mathbf{x}$  and  $\mathrm{Tr}$  is the trace of a matrix. Guo et al. [20] defined a scalar summary of the FIM called diagonal Fisher information leakage (dFIL):
$$
\mathrm {d F I L} (\mathbf {x}) = \operatorname {T r} \left(\mathcal {I} _ {\mathbf {o}} (\mathbf {x})\right) / d, \tag {3}
$$
hence the MSE of an unbiased reconstruction is lower bounded by the reciprocal of dFIL. dFIL varies with  $\mathbf{x}$ , allowing it to reflect the fact that certain samples may be more vulnerable to reconstruction.
Limitations Although the Cramér-Rao bound gives a mathematically rigorous interpretation of dFIL, it depends crucially on the unbiasedness assumption, i.e.,  $\mathbb{E}_{\mathbf{o}}[\hat{\mathbf{x}} (\mathbf{o})] = \mathbf{x}$ . In practice, most real-world reconstruction attacks use either implicit or explicit priors about the data distribution and are biased (e.g., attacks using TV prior or a DNN). It is unclear how dFIL should be interpreted in these more realistic settings. In Section 3.2, we give an alternative theoretical interpretation based on the van Trees inequality [62], which lower-bounds the MSE of any adversary, biased or unbiased.
3 Quantifying the invertibility of an encoding
Motivated by the lack of theoretically-principled privacy metrics, we propose to adapt the Fisher information leakage framework to quantify the invertibility of instance encoding. We show that many existing encoders can be modified minimally to be interpreted with dFIL and the Cramér-Rao bound. Subsequently, we extend the framework and derive a novel bound for the reconstruction error of arbitrary attacks, by establishing a connection to van Trees inequality and score matching.
Threat model We focus on reconstruction attacks that aim to invert an encoding, i.e., reconstruct the input  $\mathbf{x}$  given its encoding  $\mathbf{e}$ . We assume that the attacker has full knowledge of the encoder Enc except for the source of randomness. We consider both unbiased attacks and biased attacks that can use arbitrary prior knowledge about the data distribution.
Privacy definition We consider Enc to be private if  $\mathbf{x}$  cannot be reconstructed from the encoding  $\mathbf{e}$ . We measure the reconstruction error with the mean squared error (MSE), defined as  $||\hat{\mathbf{x}} - \mathbf{x}||_2^2 / d$ . Although MSE does not exactly indicate semantic similarity, it is widely applicable and used as a proxy for semantic similarity [69, 43]. Preventing low reconstruction MSE does not necessarily protect against other attacks (e.g., property inference [50]), which we leave as future work.
3.1 Fisher information leakage for instance encoding
To adapt the framework of Fisher information to the setting of instance encoding, we consider the encoding function Enc as a privacy-enhancing mechanism (cf.  $\mathcal{M}$  in Section 2.2) and use dFIL to measure the invertibility of the encoding  $\mathbf{e} = \mathrm{Enc}(\mathbf{x})$ . However, many instance encoders do not meet the regularity conditions in Section 2.2, making dFIL ill-defined. For example, popular DNN-based encoders do not produce randomized output, and their log density function  $\log p(\mathbf{o};\mathbf{x})$  may not be differentiable when operators like ReLU or max pooling are present.
Fortunately, many popular encoders can meet the required conditions with small changes. For example, DNN-based encoders can be modified by (1) replacing any non-smooth functions with smooth functions (e.g., tanh or GELU [29] instead of ReLU, average pooling instead of max pooling), and (2) adding noise at the end of the encoder for randomness. Alternative modifications are also possible. In particular, if we add random Gaussian noise to the output of a differentiable, deterministic encoder  $\mathrm{Enc}_D$  (e.g., DNN):  $\mathrm{Enc}(\mathbf{x}) = \mathrm{Enc}_D(\mathbf{x}) + \mathcal{N}(0,\sigma^2)$ , the FIM of the encoder becomes [22]:
$$
\mathcal {I} _ {\mathbf {e}} (\mathbf {x}) = \frac {1}{\sigma^ {2}} \mathbf {J} _ {\text {E n c} _ {D}} ^ {\top} (\mathbf {x}) \mathbf {J} _ {\text {E n c} _ {D}} (\mathbf {x}), \tag {4}
$$
where  $\mathbf{J}_{\mathrm{Enc}_D}$  is the Jacobian of  $\mathrm{Enc}_D$  with respect to the input  $\mathbf{x}$  and can be easily computed using a single backward pass. Then, Equation 2 can be used to bound the reconstruction error, provided the attack is unbiased. Other popular encoders can be modified similarly.
3.2 Bounding the reconstruction of arbitrary attacks
Most realistic reconstruction attacks are biased, and their reconstruction MSE is not lower bounded by Equation 2. Consider an (biased) attacker who knows the mean  $\mu$  of the input data distribution. If the attacker simply outputs  $\mu$  as the reconstruction of any input  $\mathbf{x}$ , the expected MSE will be the variance of the data distribution regardless of dFIL. The above example shows a crucial limitation of the Cramér-Rao bound interpretation of dFIL: it does not take into account any prior information the adversary has about the data distribution, which is abundant in the real world (Section 2.1).
Bayesian interpretation of Fisher information We adopt a Bayesian interpretation of dFIL as the difference between an attacker's prior and posterior estimate of the input  $\mathbf{x}$ . This is achieved through the classical van Trees inequality [62]. We state the van Trees inequality in Appendix 7.3, and use it to derive our MSE bound for arbitrary attacks below as a corollary; proof is in Appendix 7.4.
Corollary 1. Let  $\pi$  be the input data distribution and let  $f_{\pi}(\mathbf{x})$  denote its density function with respect to Lebesgue measure. Suppose that  $\pi$  satisfies the regularity conditions of van Trees inequality (Theorem 2), and let  $\mathcal{J}(f_{\pi}) = \mathbb{E}_{\pi}[\nabla_{\mathbf{x}}\log f_{\pi}(\mathbf{x})\nabla_{\mathbf{x}}\log f_{\pi}(\mathbf{x})^{\top}]$  denote the information theorist's Fisher information [2] of  $\pi$ . For a private mechanism  $\mathcal{M}$  and any reconstruction attack  $\hat{\mathbf{x}} (\mathbf{o})$
Figure 2:  $1 / \mathrm{dFIL}$  vs. reconstruction MSE (left) and sample reconstructions (right) for MNIST. The values before each row (right) indicate  $1 / \mathrm{dFIL}$  of the encoder used.
operating on  $\mathbf{o}\sim \mathcal{M}(\mathbf{x})$
$$
\mathbb {E} _ {\pi} \mathbb {E} [ \| \hat {\mathbf {x}} - \mathbf {x} \| _ {2} ^ {2} / d ] \geq \frac {1}{\mathbb {E} _ {\pi} \left[ \mathrm {d F I L} (\mathbf {x}) \right] + \operatorname {T r} \left(\mathcal {J} \left(f _ {\pi}\right)\right) / d}. \tag {5}
$$
Implications of Corollary 1 We can readily apply Corollary 1 to instance encoding by replacing  $\mathcal{M}$  with Enc and  $\mathbf{o}$  with  $\mathbf{e}$  as in Section 3.1. Doing so leads to several interesting implications:
1. Corollary 1 is a population-level bound that takes expectation over  $\mathbf{x} \sim \pi$ . This is necessary because given any fixed sample  $\mathbf{x}$ , there is always an attack  $\hat{\mathbf{x}}(\mathbf{e}) = \mathbf{x}$  that perfectly reconstructs  $\mathbf{x}$  without observing the encoding  $\mathbf{e}$ . Such an attack would fail in expectation over  $\mathbf{x} \sim \pi$ .
2. The term  $\mathcal{J}(f_{\pi})$  captures the prior knowledge about the input. When  $\mathcal{J}(f_{\pi}) = 0$ , the attacker has no prior information about  $\mathbf{x}$ , and Corollary 1 reduces to the unbiased bound in Equation 2. When  $\mathcal{J}(f_{\pi})$  is large, the bound becomes small regardless of  $\mathbb{E}_{\pi}[\mathrm{dFIL}(\mathbf{x})]$ , indicating that the attacker can simply guess with the input prior and achieve a low MSE.
3. dFIL can be interpreted as capturing how much easier reconstructing the input becomes after observing the encoding  $(\mathbb{E}_{\pi}[\mathrm{dFIL}(\mathbf{x})]$  term) as opposed to only having knowledge of the input distribution  $(\mathrm{Tr}(\mathcal{J}(f_{\pi})) / d$  term). Using such a change in the belief after an observation as a privacy metric is common in information theory [38, 13].
Estimating  $\mathcal{J}(f_{\pi})$  The term  $\mathcal{J}(f_{\pi})$  captures the prior knowledge of the input and plays a crucial role in Corollary 1. In simple cases where  $\pi$  is a known distribution whose density function follows a tractable form (e.g., when the input follows a Gaussian distribution),  $\mathcal{J}(f_{\pi})$  can be directly calculated. In such settings, Corollary 1 gives a meaningful theoretical lower bound for the reconstruction MSE. However, most real-world data distributions do not have a tractable form and  $\mathcal{J}(f_{\pi})$  must be estimated from data. Fortunately, the  $\nabla_{\mathbf{x}}\log f_{\pi}(\mathbf{x})$  term in  $\mathcal{J}(f_{\pi})$  is a well-known quantity called the score function in generative AI literature, and there exists a class of algorithms known as score matching [36, 47, 57] that aim to estimate the score function from the data. We leverage these techniques to estimate  $\mathcal{J}(f_{\pi})$  when it cannot be calculated; details are in Appendix 7.1. As  $\mathcal{J}(f_{\pi})$  only depends on the dataset and not the encoder, its value can be estimated once for each dataset and shared across the community. The estimation of  $\mathcal{J}(f_{\pi})$  is expected to improve with the advancement of generative AI.
Using Corollary 1 in practice When  $\mathcal{J}(f_{\pi})$  is known (e.g., Gaussian), the bound from Corollary 1 always hold. However, when estimating  $\mathcal{J}(f_{\pi})$  from data, the bound can be incorrect due to several reasons, including improper modeling of the score function, not having enough representative samples, or the regularity conditions of the van Trees inequality not being met. The bound can also be loose when tightness conditions of the van Trees inequality do not hold. Even when the bound is not exact, however, Equations 2 and 5 can be interpreted to suggest that increasing  $1/\mathrm{dFIL}$  makes reconstruction harder. Thus, we argue that dFIL still serves as a useful privacy metric that in theory bounds the invertibility of an instance encoding. When not exact, the bound should be viewed more as a guideline for interpreting and setting dFIL in a data-dependent manner.
3.3 Evaluation of the bound
We show that Corollary 1 accurately reflects the reconstruction MSE on both (1) synthetic data with known  $\mathcal{J}(f_{\pi})$ , and (2) real world data with estimated  $\mathcal{J}(f_{\pi})$ .
Figure 3:  $1/$  dFIL vs. reconstruction MSE (left) and sample reconstructions (right) for CIFAR-10. The values before each row (right) indicate  $1/$  dFIL of the encoder used.
Figure 4:  $1/$  dFIL vs. reconstruction MSE (left) and sample reconstructions (right) for CIFAR-10, with a randomized smoothing noise of  $\mathcal{N}(0,0.01^2)$ . Corollary 1 breaks when the smoothing noise is too small, but dFIL still shows a strong correlation with the reconstruction quality. The values before each row (right) indicate  $1/$  dFIL of the encoder used.
3.3.1 Synthetic data with known  $\mathcal{J}(f_{\pi})$
Evaluation setup We consider a synthetic Gaussian input distribution:  $\mathbf{x} \sim \mathcal{N}(0, \tau^2 \mathbf{I}_d)$  with  $d = 784$  and  $\tau = 0.05$ . It can be shown that  $\mathrm{Tr}(\mathcal{J}(f_{\pi})) / d = 1 / \tau^2$ , hence a larger  $\tau$  forces the data to spread out more and reduces the input prior. We use a simple encoder which randomly projects the data to a 10,000-dimensional spaces and then adds Gaussian noise, i.e.,  $\mathbf{e} = \mathbf{M}\mathbf{x} + \mathcal{N}(0, \sigma^2)$ , where  $\mathbf{M} \in \mathbb{R}^{10,000 \times 784}$ . We evaluate with a simple encoder to make the attack easier, but note that our bound should hold for more complex encoders (e.g., DNN-based) as well.
Attacks We evaluate our bound against two different attacks. An unbiased attack (Attack-ub) solves the following optimization:  $\hat{\mathbf{x}} (\mathbf{e}) = \underset {\mathbf{x}_0}{\arg \min}||\mathbf{e} - \operatorname {Enc}(\mathbf{x}_0)||_2^2$  . The attack is unbiased as the objective is convex, and  $\mathbf{x}$  is recovered in expectation. A more powerful biased attack (Attack-  $b$  ) adds a regularizer term  $\lambda \log p_{\tau}(\mathbf{x}_0)$  to the above objective, where  $p_{\tau}$  is the density function of  $\mathcal{N}(0,\tau^2\mathbf{I}_d)$  One can show that with a suitable choice of  $\lambda$  , this attack returns the maximum a posteriori estimate of  $\mathbf{x}$  , which leverages knowledge of the input distribution. Details are in Appendix 7.2.