Bounding the Invertibility of Privacy-preserving Instance Encoding using Fisher Information
Authors: Kiwan Maeng
Chuan Guo
Sanjay Kariyappa
G. Edward Suh
Venue: NeurIPS 2023 poster (2023)
Keywords: privacy
instance encoding
split learning
TL;DR: We propose a theoretically-principled framework to bound the invertibility of instance encoding using Fisher information leakage. Our bound works for arbitrary (biased or unbiased) attackers and real-world datasets with an intractible prior.

Abstract: Privacy-preserving instance encoding aims to encode raw data into feature vectors without revealing their privacy-sensitive information. When designed properly, these encodings can be used for downstream ML applications such as training and inference with limited privacy risk. However, the vast majority of existing schemes do not theoretically justify that their encoding is non-invertible, and their privacy-enhancing properties are only validated empirically against a limited set of attacks. In this paper, we propose a theoretically-principled measure for the invertibility of instance encoding based on Fisher information that is broadly applicable to a wide range of popular encoders. We show that dFIL can be used to bound the invertibility of encodings both theoretically and empirically, providing an intuitive interpretation of the privacy of instance encoding.

Bounding the Invertibility of Privacy-Preserving Instance Encoding Using Fisher Information
Kiwan Maeng" Penn State University
Chuan Guo" FAIR, Meta chuanguo@meta.. com
kvm62420psu.
Sanjay Kariyappa Georgia Institute of Technology san janjjaaariyappa@aateee edu
G. Edward Suh
FAIR, Meta & Cornell University edsuh@meta. com
Abstract
Privacy-preservicg instance encoding aims to encode raw data into feature vectors without revealing their privacy- information. When designed properly, these encodings can be used for downstream ML applications such training and inference with limited privacy risk. However, the vast majority of existing schemes do not theoretically justify that their encoding is non-invertible, and their privacy- enhancing properties are only validated empirically against limited set of attacks. In this paper, propose theoretically- prinncilee measure for the invertibility instance encoding based on Fisher information that is broadly applicable to a wide range of popular encoders. We show that dFIL be used to bound the invertibility of encodings both theoretically and empirically, providing an intuitive interpretation of the privacy of instance encoding.
1 Introduction
Machine learning (ML) applications often require access to privacy-sensitive data. A model that predicts diseases with x-ray scans requires access to patient's x-ray images. Next-word prediction models require access to user's text input that can contain sensitive information [23]. To enable MIL applications on privacy-sensitive data, instance encoding (66): Figure left) aims to encode data in way such that it is possible run useful ML tasks-such
model training and inference-on the encoded data while the privacy of the raw data is preserved. The concept is widespread under many
different names: private data sharing [16] 17 48
34
32 37), learnable encryption (35] 73 71, 70),
split learning 663 (55, split inference 44 11, and vertical federated learning (vFL; 74
59 45 are
all collaborative training/inference schemes that operate (hopefully) privately
user data.
Unfortunately, the vast. majority of existing instance encoding schemes rely on heuristics rather than rigorous theoretical arguments to justify their privacy-enhancing properties. Most existing 45 claimed that their schemes are non-invertible by simply testing them against certain input reconstruction attacks. However, these schemes may not be private under more carefully designed attacks; in fact, many encoding schemes that were initially thought to be private have been shown to be vulnerable over time [6] 77. While there exist instance encoding schemes that are based on rigorous theory, e,g., metric differential privacy (metric-DP: [8] [6]
works 35
[7 (4) ), their
theoretical analysis is only applicable very simple encoders and camnot be applied to more powerful encoders (e.g., ONN-based encoders) that are common in practice [73 64
32
Equul contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Rec MSE
Attack-ub
Bound-ub
101
Attack-b
Enc
ML training inference
10
Bound--ours
Private x
Encoding
10
10
10"
10
1/dFIL
Figure 1: Overview of instance encoding (left), and our bound against different attackers (Attack-ub, Attack-b) for synthetic dataset (right). The bound adapted from prior work (20) (Bound-uu -bb) only works against a certain adversary (Attack-ub). Our newly proposed bound (Bound-ours) works for any adversary. See Section 3.3.1 for evaluation details.
To fill the gap, we propose iiteoreticclllypriiiiiii and easily applicable framework for quantifying the privacy of instance encoding by bounding its invertibility. Our framework uses (diagonal) Fisher information leakage (dFIL; [18 22 20 an information-theorrtic measure of privacy with similar properties to differential privacy (DP; [1 (.). dFIL can be computed for common privacy- enhancing mechanisms and can lower-bound the expected mean squared error (MSE) of an input reconstruction attack. We apply this reasoning to instance encoding and show that dFIL can serve as a good measure for the invertibility of instance encoding, as it can bound the reconstruction MSE against any attacker that tries to reconstruct the private input from the encoding. dFIL can be easily calculated for any encoders that are differentiable and randomized beiiizedd-eiinggg applicable to most of the popular encoders with minimal modifications. To the best of our knowledge, work is the first to theoretically lower-bound the invertibility of instance encoding and use it to design private training/inference systems.
Contributions
Our main contributions are as follows:
1. We adapt the result of prior works [18 20 for instance encoding to show how dFIL can lower bound the MSE of particular input reconstruction attacks (i.e., unbiased attacks) that aim to reconstruct the raw data given the encoding. We show how popular encoders can be modified minimally for dFIL to be applicd (Section
2. We extend the result of prior works [18 [0] and show that dFIL lower bound the MSE of any input reconstruction attack (e.g., strong attacks leveraging knowledge of the input prior; Section
3.2). Our extension involves novel application of the classical van Trees inequality [62 and connecting it to the problem of score matching [57] in distribution estimation.
3. We evaluate the lower bound using different attacks and encoders, and show that dFIL can be used to interpret the invertibility of instance encoding both theoretically and empirically (Section 3.3)
4. We show how dFIL can be used as practical privacy metric and guide the design of privacy- enhancing training/inference systems with instance encoding (Section 4 We show that it is possible to achieve both high pooooettcally-uustffiee privacy and satisfactory utility.
Motivation and background
2.1 Instance encoding
Instance encoding refers to the general concept of encoding raw input x using an encoding function Enc, such that private information contained in x cannot be inferred from its encoding e Enc(x). while e maintains enough utility for the downstream ML tasks. The principle behind the privacy- enhancing property of instance encoding is that the function Enc is hard to invert. However, prior works generally justify this claim of non-invertibility based on heuristics rather than rigorous theoret- ical analysis 664 65 45), except for few cases where the encoder is very simple [T6
Attacks against instance encoding Given an instance encoder Enc, the goal of a reconstruction attack is to recover its input. Formally, given Enc(x), attack Att aims reconstruct from e: x = Att(e). Such an attack can be carried out in several ways. If Enc is known, & can be obtained by solving the following optimization [27] X arg min Ene(xo))1 This attack can be further
For instance, images can be regularized with total variation (TV) prior to reduce high- gigh-freueecc components [49]. Alternatively, if samples
improved when some prior of the input is known 49
61
from the input distribution can be obtained, DNN that generates & from can be trained [53 27 12

---

Privacy metrics for instance encoding Measuring how much information an observation leaks about a secret quantity it depends on is a classical question that has long been studied in information
3 13 33 and instance encoding privacy is directly relevant to the stream of works. Prior works proposed metrics providing strong notion of privacy (e.g., Arimoto's s mutual information A or Sibson's ss mutual information (38). However, it is challenging to apply these metrics to MI. and instance encoding which use high-dimensional data [33], because these metrics have to be calculated and optimized over the prior of the dataset, the encodings, and/or their joint distribution, all of which are unknown and hard to accurately model [3)]
theory [38
Differential privacy (DP) [4 [5], one of the most popular frameworks to quantify ML privacy M. not suitable for instance encoding [6]. DP guaramtees the worst-case indistingusshability of the encoding from two different inputs, which significantly damages the utility [6] (Appendix (7.5). Metric-DP g a weaker notion of DP-is applicable to instance encoding and has been used by several prior works [16] 17
48 37. However, metric-DP is applicable only when the encoder extremely simple. For example, prior works used pixel-level perturbation (e-g.. pixelization, blurring [66] [ or single-layer convolution 44 as encoder. These extremely-simple encoders are in stark contrast with the complex encoders most practitioners use (e.g. split learning [64], split inference [65. and vFL 45 59]
44
all uses DNN-based encoders), and results in significant utility loss (Section[422 4.2). Applying metric-DD DP to an arbitrary DNN-based encoder is NP-hard [66]. Unlike these works that focus on indistinguishabiiity our work directly quantifies non-invertibilltyy, weaker but still relevant definition of privacy (Section 3
Due to the limitations in existing theoretical metrics, the vast majority of works in the field are relying on empirical measures to demonstrate their encoder's privacy-enhancing properties. Some 46 45 simply test their encoders against a limited set of attacks, while others [64 65 [1]] use heuristical privacy metries without rigorous theoretical arguments. Using heuristical metrics that are not backed by theoretical interpretation is dangerous, because seemingly useful metric can actually severely mis-characterize the privacy of a system [38] is of both interest and practical importance to develop and widely applicable.
privacy metric that is both eeporetically-mmannngge
concurrent work (72) proposed a notion of PAC privacy that can bound the adversary's 's attack success rate in theory for instance encoding. PAC privacy bound has not yet been evaluated empirically under realistic attack and use cases, which remains an interesting future work.
2.2 Fisher information leakage
is a measure of leakage through a privacy-enhancing mechanism. Let M be a randomized mechanism on data sample and let o ~ MMx) be its output. Suppose the log density function p(o; x) is differentiable w.r.t. x and satisfies the following regularity condition: E. Vx p(;; x] O. The Fisher information matrix (FIM) Io (x) is:
Fisher information leakage (FIL: I8 22 20
(x) = E., V log p(o; x) Vx lo
Cramér-Rao bound Fisher information a compelling privacy metric as it directly relates to the mean squared error (MSE) of a reconstruction adversary through the Cramér-Rao bound [41]. In detail, suppose that (() is an unbiased estimate (or reconstruction) of x given the output of the randomized private mechanism M(x).. Then:
d
E. [11)x(o)- - x112/42 Z
(2)
(x))
where d is the dimension of x and Tr is the trace of a matrix. Guo et al. 20 defined a scalar summary of the FIM called diagonal Fisher information leakage (dFIL):
dFIL(x) = Tr(Zo(x))/d,
hence the MSE of an unbiased reconstruction is lower bounded by the reciprocal of dFII. dFIL varies with x, allowing it to reflect the fact that certain samples may be more vulnerable to reconstruction.
Limitations Although the Cramér-Rao bound gives a mathematically rigorous interpretation of dFIL, it depends crucially on the unbiasedness assumption, i.e., E. [x(o)] = x. In practice, most real-world reconstruction attacks use either implicit or explicit priors about the data distribution and are biased (e.g., attacks using TV prior or DNN). It is unclear how dFIL should be interpreted in these more realistic settings. In Section 3.2] we give an alternative theoretical interpretation based on the van Trees incquality [62]. which fower-bounds the MSE of any adversary, biased unbiased.
3

---

3 Quantifying the invertibility of an encoding
Motivated by the lack of parniiiplllly privacy metrics, we propose to adapt the Fisher information leakage framework to quantify the invertibilityy of instance encoding. We show that many existing encoders can be modified minimally to be interpreted with dFIL and the Cramér- Rao bound. Subsequently, extend the framework and derive novel bound for the reconstruction error of arbitrary attacks, by establishing a connection to van Trees inequality and score matching.
Threat model We focus on reconstruction attacks that aim to invert an encoding, i.e., reconstruct the input given its encoding We assume that the attacker has full knowledge of the encoder Enc except for the source randomness. We consider both unbiased attacks and biased attacks that can use arbitrary prior knowledge about the data distribution.
Privacy definition We consider Enc to be private if x cannot be reconstructed from the encoding We measure the reconstruction eгror with the mean squared error (MSE), defined as ||* x x122/ d. Although MSE does not exactly indicate semantic similarity, it is widely applicable and used as a proxy for semantic similarity [69 43.. Preventing low reconstruction MSE does not necessarily protect against other attacks (e.g. property inference 50 ) which we leave as future work.
3.1 Fisher information leakage for instance encoding
To adapt the framework of Fisher information to the setting of instance encoding, we consider the encoding function Enc privacy-enhancing mechanism (cf. M in Section 2.2 and use dFIL to measure the invertibility of the encoding e = Enc(x). However, many instance encoders do not meet the regularity conditions in Section 2.2 making dFIL ill-defined. For example, popular DNN-based encoders do not produce randomized output, and their log density function log p(o:p(o x) may not be differentiable when operators like ReLU or max pooling are present.
Fortunately, many popular encoders can meet the required conditions with small changes. For example, DNN-based encoders can be modified by (1) replacing any non-smooth functions with smooth functions (e.g., lanh or GELU (29] instead of ReLU, average pooling instead of max pooling), and (2) adding noise the end of the encoder for randomness. Alternative modifications are also possible. In particular, if we add random Gaussian noise to the output of a differentiable, deterministic encoder Enep (e.g., DNN): Enc(x) Encp(x) + (0 o²). the FIM of the encoder becomes [22):
1 J g2 Enco (x)J (x),
Ia (x)
where Jenep is the Jacobian of Encp with respect to the input x and can be casily computed using a single backward pass. Then, Equation 2 can be used to bound the reconstruction error, provided the attack is unbiased. Other popular encoders can be modified similarly.
3.2 Bounding the reconstruction of arbitrary attacks
Most realistic reconstruction attacks are biased, and their reconstruction MSE not lower bounded by Equation 2 Consider an (biased) attacker who knows the mean μa of the input data distribution. the attacker simply outputs μ as the reconstruction of any input x, the expected MSE will be the variance of the data distribution regardless of dFIL. The above example shows crucial limitation of the Cramér-Ra. Rao bound interpretation dFIL: does not take into account any prior information the adversary has about the data distribution, which is abundant in the real world (Section 2.1
Bayesian interpretation of Fisher information We adopt Bayesian interpretation of dFIL as the difference between an attacker's prior and posterior estimate of the input x. This is achieved through the classical van Trees inequality [62]. We state the van Trees inequality in Appendix (7.3 and use it to derive our MSE bound for arbitrary attacks below as a corollary: proof
Appendix 7.4
Corollary Let x be the input data distribution and let [x(x) denote its density function with respect to Lebesgue measure. Suppose that πt satisfies the regularity conditions of van Trees inequality (Theorem 2 and let T(f=) [V log (x)xxxxxxx x log J [xx) Ti denote the information theorist's Fisher information 12] of π. For private mechanism M and any reconstruction attack x(o)

---

Recon. MSE
10a
Atback-ub Attack-b
Bound-ub
7
9
7
100
7
2
0
4
/
4
9
7
Bound-ours
10-1
10
7
2
0
4
9
3
<
10
10
10-3
10-2
10-1
10a
102
25
0
0
3
1/dFIL
Figure 2: 1/dFIL vs. reconstruction MSE (left) and sample reconstructions (right) for MNIST. The values before each (right) indicate 1/ dFLL of the encoder used.
operating M(x):
1
B||
x 12/42
(5)
B [F) + (T(()=)
Implications of Corollary !
We can readily apply Corollary to instance encoding by replacing
M with Enc and o with e as in Section 3.1 Doing so leads to several interesting implications:
1. Corollary is population-level bound that takes expectation
т. This is necessary because given any fixed sample x, there is always an attack x(e) = x that perfectly reconstructs x without observing the encoding c. Such an attack would fail in expectation
n.
The term J(fx) captures the prior knowledge about the input. When [(fx) = O. the attacker has no prior information about and Corollary reduces to the unbiased bound in Equation 2 When J(xx) is large, the bound becomes small regardless of E, [d)] indicating that the attacker can simply guess with the input prior and achieve low MSE.
dFIL can be interpreted as capturing how much easier reconstructing the input becomes after observing the encoding (E,, (dFIT(()) term) as opposed to only having knowledge of the input distribution (Tr(I((=)) / term). Using such a change the belief after an observation as privacy metric is common in information theory [38 13
Estimating [(z)) The term (Jx) captures the prior knowledge of the input and plays a crucial role in Corollary [ In simple cases where π is known distribution whose density function follows tractable form (e.g., when the input follows a Gaussian distribution),
(fx) can be directly calculated. In such settings, Corollary [ gives meaningful theoretical lower bound for the reconstruction MSE. However, most real-world data distributions do not have a tractable form and ((f=) must be estimated from data. Fortunately, the Vx log fx)x term in J(fx) is well-known quantity called the score function in generative AI literature, and there exists class algorithms known as score matching [6 47 57 that aim to estimate the score function from the data. We leverage these techniques to estimate J ((f)) when it cannot be calculated; details are in Appendix 7.1 As J (fx) only depends the dataset and not the encoder, its value be estimated once for each dataset and shared across the community. The estimation of [(xx) is expected to improve with the advancement of generative AI.
Using Corollary in practice When J(fa) is known (e.g., Gaussian), the bound from Corollary I always hold. However, when estimating J(f=) from data, the bound can be incorrect due several reasons, including improper modeling of the score function, not having enough representative samples, or the regularity conditions of the van Trees inequality not being met. The bound can also be loose when tightness conditions of the van Trees inequality do not hold. Even when the bound is not exact, however, Equations 2 and 5 can be interpreted to suggest that increasing 1/ /FFTI makes reconstruction harder. Thus, we argue that dFII. still serves as useful privacy metric that in theory bounds the invertibility of an instance encoding. When not exact, the bound should viewed more as guideline for interpreting and setting dFIL in data-dependent manner.
3.3 Evaluation of the bound
We show that Corollary aaccuratlyy reflects the reconstruction MSE on both (1) synthetic data with known J (fx), ) and (2) real world data with estimated J (fx).
5

---

Recon. MSE
Attack-ub Attack-b
Bound-ub
ref
100
0.1
Bound-ours
10-1_
10-5
10
10-3
10-2
10-1
10a
102
10
1/dFIL
Figure 3: 1/dFII vs. reconstruction MSE (left) and sample reconstructions (right) for CIFAR-10.
The values before each row (right) indicate // dFIL of the encoder used.
Recon. MSE
10a
Aktuck-ub
Bound-ub
ref
100
Attack-b
0.1
10-3
Bound-ours
10-5
10-4
10-3
10-2
10-1
10"
10
10a
10
1/dFIL
Figure 4: 1/ dFIL vs. reconstruction MSE (left) and sample reconstructions (right) for CIFAR- 10, with randomized smoothing noise of (0, 0.012). Corollary breaks when the smoothing noise is too small, but dFIL still shows strong correlation with the reconstruction quality. The values before each row (right) indicate // dFIL of the encoder used.
3.3.1 Synthetic data with known J(fx)
Evaluation setup We consider a synthetic Gaussian input distribution: = 784 and 0.05. It can be shown that (r([(=)))d to spread out more and reduces the input prior. We use a simple encoder which randomly projects the data to 10, 000-dimensional spaces and then adds Gaussian noise, i.e., e Mx + (000, o²), where M E R 12000000784 We evaluate with simple encoder to make the attack easier, but note that our bound should hold for more complex encoders (e.g., basddb as well.
rL)) with
hence a larger T forces the data
Attacks We evaluate our bound against different attacks. An unbiased attack (Attack-ub) solves the following optimization: x(e) arg min|le le Enc(x, xo) The attack is unbiased as the X
objective is convex, and x is recovered in expectation. A more powerful biased attack (Attack-b) adds regularizer term À log p, (Xo) to the above objective, where p, the density function of (0, One can show that with a suitable choice of X, this attack returns the maximum a posteriori estimate of which leverages knowledge of the input distribution. Details are in Appendix
Result Figure D (right) plots the MSE of the two attacks, and the bounds for unbiased (Equation 2 and arbitrary attack (Equation 5). The MSE Attack-ub (red circle) matches the unbiased attack lower bound (Bound-ub; red dashed line), showing the predictive power of Equation 2]against this restricted class of attacks. Under Attack-b (blue triangle), however, Bound-ub breaks. Our new bound from Equation 5 (Bound-ours; blue dotted line) reliably holds for both attacks, initially being close to the unbiased bound and converging to guessing only with the input prior (attaining 7²).
3.3.2 Real world data with estimated J(fx)
We also evaluated Corollary I on MNIST [10] and CIFAR-10 442 Here, estimated I (J=) using sliced score matching [57] As discussed in Appendix 7.1 a moderate amount of randomized smoothing (adding Gaussian noise the input [9] is necessary ensure that the score estimation is stable and that regularity conditions of the van Trees incquality are satisfied. We used a simple CNN-based encoder: Conv(x) (O,00²)
Evaluation setup
Attacks We evaluated Attuck-ub, which is the same in Section 3.3.1 and Atlack-b, which is trained DNN that outputs the reconstruction given an encoding [45). We also evaluated regularization- based attacks 44 61] and obtained similar results; omit those results for brevity.

---

Result Figures 2 and plot the result with randomized smoothing noise of (0, 0.25²). Again, Bound-ub correctly bounds the MSE achieved by Artack-ub. While Attack-b is not as effective for very low 1/ dFII. it outperforms Attack-ub for high 1/ dFIT,, breaking Bound-ub. In comparison, Corollary 1 estimated using score matching (Bound-ours) gives a valid lower bound for both attacks.
Figures 2 and also highlights some of the reconstructions visually. Here, the left-hand side number indicates the target 1/ dF'LL and the images are reconstructed using Attack-b. In both figures, it can be seen that dFIL correlates well with the visual quality of reconstructed images, with higher values of 1/ dFIL indicating less faithful reconstructions. See Appendix: Figure 10 11
for more results.
Figure 4 additionally shows the result with a much smaller randomized smoothing noise of (0,00.013) Unlike previous results, Bound-oury breaks around 1/ dFIL=1003 We suspect it is due to score matching failing when the data lie on a low-dimensional manifold and the likelihood changes rapidly near the manifold boundary, which can be the when the smoothing noise is small. Nonetheless, the bound still correlates well with MSE and the visual reconstruction quality. We claim that dFIL still serves useful privacy metric, with a eeeorrtically-prneiiie interpretation and strong empirical correlation to invertibility. More reconstructions are in Appendix: Figure 12
4 Case study 1: split inference with dFIL
4.1 Private split inference with dFIL
Split inference 40 4 65 is method to run inference of large DNN that is hosted on the server, without the client disclosing raw input. is done by running the first few layers of a large DNN on the client device and sending the intermediate activation, instead of raw data, to the server to complete the inference. The client computation can be viewed as instance encoding, where the first few layers the client device act as encoder. However, without additional intervention, split inference by itself not private because the encoding can be inverted [27].
We design private split inference system by measuring and controlling the invertibility of the encoder with dFIL. Because the encoder of split inference is DNN, dFIL. can be calculated using Equation 4 with minor modifications to the network (Section 3.1
Optimizations There are several optimizations to improve the model accuracy for the same dFIL: (1) We calculate the amount of noise that needs to be added to the encoding to achieve a target dFIL, and add a similar amount of noise during training. (2) For CNNs, we add a compression layer- a convolution layer that reduces the channel dimension significantly-att at the end of the encoder and a corresponding decompression layer at the beginning of the server-side model. Similar heuristics were explored in prior works [1] 455 (3) We add an SNR regularizer that is designed maximize the noise that needs to be added to
the signal-to- signal-to-noise ratio of the encoding. From Equations 3
achieve a certain dFIL is = TreJenep Enep (x]]ese,((x))
Thus, maximizing the signal-to- to-so ratio
d=dFIL
(SNR) of the encoding (cT c/a²) is equivalent minimizing Tr(JEne, (x)JEmep (x)) which we add to the optimizer during training. These optimizations were selected from comparing multiple heuristics
e
from prior work [60] 28 46
65
45 [I] and result in notable improvement of test accuracy.
4.2 Evaluation of dFse- split inference
4.2.1 Evaluation setup
Models and datasets We used three different models and four different datasets to cover a wide range of applications: ResNet-18 (25] with CIFAR-10/100 [42 for image classification, MLP-based neural collaborative filtering (NCF-MLP) (26) with MovieLens 200 20M [24) for recommendation, and DistilBert [56] with GLUE-SST2 68 for sentiment analysis. See Appendix 7.2 for details.
Detailed setups and attacks For ResNet-18, 18, we explored three split inference configurations: splitting early (after the first convolution layer), in the middle (after block 4), and late (after block 6). We evaluated the empirical privacy with DNN attacker 45 and measured the reconstruction quality with structural similarity index measure (SSIM) BI. Other popular attacks showed similar trends (see Appendix: Figure 9. NCF-MLP translates user id (uid) and a movie id (mid) into embeddings
7

---

late middle early
early
middle
late
1.0
SSIM
10-5
10"4
10
10 2
10-1
10"
10a
10a
10-5
10
10
10 2 1/dFIL
10
100
10
10a
1/dFIL
Figure 5: 1/d/// /FIFIL SSIM (left) and sample reconstructions (right) for split inference with ResNet- 18 and CIFAR-10. Higher SSIM indicates more successful reconstruction.
1.0
1.0
Attack acc.
mid top
early
Attack
0.5
0.5
middle
mid top
late
0.0
10
10
10
10
10 1
10
10a
10
10
10
10
10
10a
1/dFIL
1/dFIL
Figure 6: Attack accuracy vs. 1/ I/dFIT, for split inference with NCF-MLP (left) and DistilBert (right).
with an embedding table and sends them through DNN to make a prediction. We split the NCF- MLP model after the first linear layer of the MLP and tried reconstructing the original uid and mid from the encoding. This is done by first reconstructing the embeddings from the encoding using direct optimization (emb(e, e) = arg min|/c Enc(embo)ll)) and finding the original uid and mid emby
by choosing the closest embedding value in the embedding table: id = arg min||emd Emb[j]l..
where Emb[i] is the i-th entry of the embedding table. For DistilBert, different splitting configurations: splitting early (right after block 0), in the middle (after block 2), and late (after block 4). We similar attack to NCF-NLP to retrieve each word token.
again explored three
4.2.2 Evaluation results
Privacy Figure 5 shows the attack result for ResNet-18 and CIFAR-IO, Setups with lower dFIL lead lower SSIM (left) and less identifiable images (right), indicating that dFIL strongly correlates with the attack success rate. The figures also show that the privacy leakage estimated by dFIL can be conservative. Some setups show empirically-hige privacy even when dFIL indicates otherwise, especially when splitting late. See Appendix: Figure [3 for more results. Figures show the attack result for NCF-MLP (left) and DistilBert (right). Setps with lower dFIL again consistently showed worse attack success rate. A sample reconstruction for DistilBert is shown in Appendix: Table 5)
Utility Table I summarizes the test accuracy of the split inference models, where 1/ dFIL is chosen so that the attacker's reconstruction error is relatively high. For the same 1/ dFLL, our proposed optimizations (Ours) improve the accuracy significantly compared to simply adding noise (No opt.). In general, reasonable accuracy be achieved with encoders with relatively low dFIL. Accuracy degrades more when splitting earlier, indicating that using a DNN encoder with many layers can benefit utility. Prior works empirically observed similar trend where splitting earlier degrades the privacy-utility tradeoff, because the encoding becomes leakier [49]. Our framework supports complex DNN encoders, which is distinguishing benefit compared prior theoretical works [16] 17
Case study 2: training with dFIL
5.1 Training on encoded data with dFIL.
We consider scenario where users publish their encoded private data, and downstream model trained on the encoded data. We use the first few layers of a pretrained model as the encoder by

---

Table Test accuracy for different split inference setups with different dFIL. Base accuracy in parenthesis.
Table Accuracy from training with different encoders.
Setup
Split
No opt.
Ours
Pretrain dataset
Acc.
dFIL
aPIL
10.70%
74.44%
10
75.46%
early
TinylmageNet
100
10.14%
57.97%
100
42.57%
CIFAR-10 + ResNet-18
22.11%
91.35%
10
80.16%
middle
CIFAR-100
(ace: 92.70%)
100
12.94%
84.27%
100
70.27%
78.48%
92.35%
CIFAR-10
10
81.99%
late
100
33.54%
87.58%
(held-out 20%)
100
78.65%
10
1.00%
44.10%
early
100
1.00%
30.69 %
CIFAR-100 + ResNet-18 18
10
1.98%
59.51 %
middle
(ace: 72.13%)
100
1.29%
41.29 %
10
8.63%
65.77%
late
100
2.01%
39.18%
MovieLens- -CF-MMP
0.8172
0.8286
early
10
0.7459
0.8251
(AUC: 0.8228)
0.6120
0,8081
50.80%
82.80
early
100
49.08%
81.88 %
GLUE-SST2
DistilBert
76.61%
83,03
middle
(ace: 91.04%)
100
61.93%
82.22%
90.25
83. 03.03%
late 100
82.68%
82.82%
freezing the weights and applying the necessary changes in Section 3.1 model with its last layer modified for the downstream task and finetune it with the encoded data. We found that similar optimizations from split inference benefit this use case as well.
Then, we use the rest of the
5.2 Evaluation of dFII.-based training
We evaluate the model utility and show that it reach reasonable accuracy when trained encoded data. We omit the privacy evaluation as is similar to Section 4.2.2
Evaluation setup We train a ResNet-18 18 model for CIFAR-10 classification. First, we train the model on one of three different datasets: (1) TinylmageN 20% of CIFAR-10. Then, layers up to block 4 are frozen and used the encoder. The CIFAR-10 training set encoded using the encoder and used to finetune the rest of the model. The setup mimics scenario where some publicly-availad CIFAR-100) the same (held-out CIFAR- 10) with the target data is available and is used for encoder training. Detailed hyperparameters are in Appendix 7.2 7.2
58 (2) CIFAR-100 [2]. and (3) held-out
data whose distribution similar (TinyImageNer,,
Evaluation result Table summarizes the result. Our design was able to achieve a decent accuracy using encoded data with relatively safe 1/ dFIL values (10-100). The result indicates that model training with privately encoded data is possible. The achieved accuracy was higher when the encoder was trained with data whose distribution is more similar to the downstream task (CIFAR-10). We believe more studies in hyperparameteraacchitteuuur search will improve the result.
Limitations
We showed that dFIL can theoretically quantify the invertibility of instance encoding by providing a reconstruction enoor bound against arbitrary attackers. We subsequently showed that dFIL can be used to guide private training/inference systems that uses instance encoding- dFII has several potential limitations, which needs to be carefully considered before being used:

---

1. Corollary 1 Jonly bounds the MSE, which might not always correlate well with the semantic quality. To address this, van Trees inequality can be extended an absolutely continuous function (xx) to bound [+ (
/
19 which may be used to extend to metrics other than MSE
2. Equation provides average case bound, some data may experience lower reconstruction MSE than the bound. Similar average case bounds have also been used in prior works [3] 21 When bounding invertibilityy, it is unclear how define the privacy attack game target worst-case reconstruction bound: for any sample, there exists a trivial attacker that always outputs that fixed sample from the distribution, which gives a worst-case MSE of zero for that sample. Leakier data that might experience lower reconstruction MSE than the bound can be potentially detected and handled by dynamically calculating dFIL for each sample. See Appendix: Figure 14
3. For data types where MSE is not directly meaningful or the bound is inaccurate, it may not be straightforward to interpret the privacy of encoding given its dFIL. In such cases, acceptable values of dFIL. should be determined for each application through further research. The situation is similar DP, where it often not straightforward what privacy parameters (e.g., e, 6) need to be used (39).
Systems with the same dFIL may actually have different invertibility, as the bound from dFIL may be conservative. Comparing the privacy of two different systems using dFIL should be done with caution because dFIL is a lower bound rather than an accurate measure of invertibility.
Acknowledgments and Disclosure of Funding
We thank Yang Song for the invaluable discussion regarding the application of sliced score matching to our setup. We also thank Minjae Park and Jisu Kim for the discussion of our bound's 'ss implications. Kiwan Maeng was partially funded by the Charles K. Etner Early Career Professorship from Penn State University.
References
[1] Martín Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Edgar R. Weippl, Stefan Katzenbeisser, Christopher Kruegel, Andrew C. Myers, and Shai Halevi, editors, Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016, pages 308-318. ACM, 2016
[2] Efe Aras, Kuan- Yun Lee, Ashwin Pananjady, and Thomas A. Courtade. A family of bayesian cramér-rao bounds, and consequences for log-concave priors. In IEEE International Symposium on Information Theory, ISIT 2019, Paris, France, July 7-12, 2019, pages 2699-2703. IEEE,
2019.
[3] Shahab Asoodeh, Mario Díaz, Fady Alajaji, and Tamás Linder. Estimation efficiency under privacy constraints. IEEE Trans. Inf. Theory, 65(3):1512-1544 2019.
[4] Amin Sonitalebii-ehhkon Naveen Vedula, Jian Pei, Fei Xia, Lanjun Wang, and Yong Zhang. Auto-split: general framework of collaborative edge-cloud ai. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2543-224222222 2553,25533220 2021.
[5] Martín Bertrán, Natalia Martínez, Afroditi Papadaki. Qiang Qiu, Miguel R. D. Rodrigues, Galen Reeves, and Guillermo Sapiro. Adversarially learned representations for information obfuscation and inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA. volume 97 of Proceedings of Machine Learning Research, pages 614-623. PMLR, 2019.
[6] Nicholas Carlini, Samuel Deng, Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody, Shuang Song. Abhradeep Thakurta, and Florian Tramèr. An attack on instahide: Is private learning possible with instance encoding? CoRR, abs/2011.055 2020.
[7] Nicholas Carlini, Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody, and Florian Tramèr. Neuracrypt is not private. CoRR, abs/2108.0722 2021.
10

---

[8] Konstantinos Chatzikokolakis, Miguel E. Andrés, Nicolás Emilio Bordenabe, and Catuscia Palamidessi. Broadening the scope of differential privacy using metrics. In Emiliano De Cristofaro and Matthew K. Wright,, editors, Privacy Enhancing Technologies 13th International Symposium, PETS 2013, Bloomington, IN, USA, July 10-12, 2013. Proceedings, volume 7981 of Lecture Notes in Computer Science, pages 82-102. Springer, 2013.
[9] Jeremy M. Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1310-1320. PMLR, 2019.
[10] Li Deng. The mnist datahase of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142 2012.
[11] Xin Dong, Barbara De Salvo, Meng Li, Chiao Liu, Zhongnan Qu HT Kung. and Ziyun Designing neural architectures for efficient distributed computing on head- mounted systems. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 12559-12569, 2022.
Li. Splitnets:
[12] Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 4829-4837. IEEE Computer Society, 2016.
[13] Flávio du Pin Calmon and Nadia Fawaz. Privacy against statistical inference. In 50th Annual Allerton Conference on Communication, Control, and Computing. Allerton 2012, Allerton Park & Retreat Center, Monticello, IL. USA, October 1-5, 2012, pages 1401-1408. IEEE, 2012.
[14] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.
[15] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations differential privacy. Founda- lions and Trends@ in Theoretical Computer Science, 9(3-4):21 1-7777 2014.
[16] Liyue Fan. Image pixelization with differential privacy. In Florian Kerschbaum and Stefano Paraboschi, editors, Data and Applications Security and Privacy XXXII 32nd Annual IFIP WG 11.3 Conference, DBSec 2018, Bergamo, Italy, July 16-18, 2018, Proceedings, volume 10980 of Lecture Notes in Computer Science, pages 148-162. Springer, 2018.
[17] Liyue Fan. Differential privacy for image publication. In Theory and Practice of Differential Privacy (TPDP) Workshop, volume 1, page 6, 2019.
[18] Farhad Farokhi and Henrik Sandberg. Fisher information as a measure of privacy: Preserving privacy of houscholds with smart meters using batteries. IEEE Trans. Smart Grid, 95):4726 4734, 2018.
[19] Richard D Gill and Boris Y Levit. Applications of the van trees inequality: bayesian cramér-rao bound. Bernoulli, pages 59-79, 1995.
[20] Chuan Guo, Brian Karrer, Kamalika Chaudhuri, and Laurens van der Maaten. Bounding training data reconstruction in private (deep) learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA. volume 162 of Proceedings of Machine Learning Research, pages 8056-8071. PMLR, 2022.
[21] Chuan Guo, Alexandre Sablayrolles, and Maziar Sanjabi. Analyzing privacy leakage in machine learning via multiple hypothesis testing: lesson from fano. arXiv preprint arX1v:2210.13662. 2022.
[22] Awni Hannun, Chuan Guo, and Laurens van der Maaten. Measuring data leakage in machine- learning models with fisher information. In Uncertainty in Artificial Intelligence, pages 760-770.
PMLR, 2021.
11

---

[23] Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv: 181 I. 1.0600,, 2018.
[24] F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. 5(4):19:1 2016.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016.
[26] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Rick Barrett, Rick Cummings. Eugene Agichtein, and Evgeniy Gabrilovich, editors, Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, pages 173-182. ACM, 2017.
[27] Zecheng He, Tianwei Zhang, and Ruby B. Lee. Model inversion attacks against collaborative inference. In David Balenson, editor, Proceedings of the 35th Annual Computer Security Applications Conference, ACSAC 2019, San Juan, PR, USA, December 09-13, 2019, pages 148-162. ACM, 2019.
[28] Zecheng He, Tianwei Zhang, and Ruby B Lee. Attacking and protecting data privacy in edge- cloud collaborative inference systems. IEEE Internet of Things Journal, 8(12):9706-9716 2020.
[29] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 08415, 2016.
[30] Jonathan Ho. Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[31] Alain Horé and Djemel Ziou. Image quality metries: PSNR SSIM. In 20th International Conference on Pattern Recognition, ICPR 2010, Istanbul, Turkey, 23-26 August 2010, pages 3366-2369, IEEE Computer Society, 2010.
[32] Hsiang Hsu, Shahab Asoodeh, and Flávio P. Calmon. Obfuscation via information density estimation. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pages 906-917. PMLR, 2020.
[33] Hsiang Hsu, Natalia Martinez, Martin Bertran, Guillermo Sapiro, and Flavio P Calmon. A survey on statistical, information, and estimation theoretic views on privacy. IEEE BITS the Information Theory Magazine, (1))45-566, 2021.
[34] Chong Huang, Peter Kairouz, Xiao Chen, Lalitha Sankar, and Ram Rajagopal. Generative adversarial privacy. CoRR,
2018
[35] Yangsibo Huang, Zhao Song, Kai Li, and Sanjeev Arora. Instahide: Instance-hiding schemes for private distributed learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4507-4518. PMLR, 2020.
[36] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4). 2005.
[37] Jacob Imola, Shiva Prasad Kasiviswanathan, Stephen White, Abhinav Aggarwal, and Nathanael Teissier. Balancing utility and scalability in metric differential privacy. In James Cussens and Kun Zhang, editors, Uncertainty in A Artificial Intelligence, Proceedings of the Chirty-Eighhl Conference on Uncertuinty in Artificial Intelligence, UAI 2022, 1-5 August 2022, Eindhoven, The Netherlands, volume 180 of Proceedings of Machine Learning Research, pages 885-894.
PMLR, 2022.
12

---

[38] Ibrahim Issa, Aaron B. Wagner, and Sudeep Kamath. An operational approach to information leakage, IEEE Trans. Inf. Theory, 6663):165--1657, 2020.
[39] Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In 28th USENIX Security Symposium (USENIX Security 19), pages 1895-1912, 2019.
[40] Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingja Tang. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. ACM SIGARCH Computer Architecture News, 45(1):615-622 2017.
[41] Steven M Kay. Fundamentals of statistical signal processing: estimation theory. Prentice- Hall, Inc., 1993.
[42] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[43] Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In International conference on machine learning, pages 957-966. PMLR, 2015.
[44] Mathias Lécuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 JEEE Symposium on Security and Privacy, SP 2019, San Francisco, CA, USA, May 19-23, 2019, pages 656-672. IEEE, 2019.
[45] Jingtao Li, Adnan Siraj Rakin, Xing Chen, Zhezhi He, Deliang Fan, and Chaitali Chakrabarti. Ressfl: A resistance transfer framework for defending model inversion attack in split federated learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10184-10192. IEEE, 2022.
[46] Meng Li, Liangzhen Lai, Naveen Suda, Vikas Chandra, and David Z Pan. Privynet: A flexible framework for privacy-preserving deep neural network training. arXiv preprint 170.v:170,.0616,, 2017.
[47] Yingzhen Li and Richard E Turner. Gradient estimators for implicit models. arXiv preprint arXiv:1705.07107 2017.
[48] Bo Liu, Ming Ding, Hanyu Xue, Tianqing Zhu. Dayong Ye, Li Song, and Wanlei Zhou. Dp-image: Differential privacy for image data in feature space. CoRR, abs/2103.07077 2021.
[49] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 5188-5196. IEEE Computer Society, 2015.
[50] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage collaborative learning. In 2019 IEEE Symposium on Security and Privacy, SP 2019, San Francisco, CA, USA, May 19.23, 2019, pages 691-706. IEEE, 2019.
[51] Fatemehsadat Mireshghallah, Mohammadkazem Taram, Prakash Ramrakhyani, Ali Jalali, Dean Tullsen, and Hadi Esmaeilzadeh. Shredder: Learning noise distributions to protect inference privacy. In Proceedings of the Twenty-Fifth International Conference Architectural Support for Programming Languages and Operating Systems, pages 3-18, 2020.
[52] Ilya Mironov. Rényi differential privacy. In 30th IEEE Computer Security Foundations Symposium, CSF 2017, Santa Barbara, CA, USA, August 21-25, 2017, pages 263-275, IEEE Computer Society, 2017.
[53] Dario Pasquini, Giuseppe Ateniese, and Massimo Bernaschi. Unleashing the tiger: Inference attacks on split learning. In Yongdae Kim, Jong Kim, Giovanni Vigna, and Elaine Shi, editors, CCS '21: 2021 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, Republic of Korea, November 15- 19, 2021, pages 2113-2129, ACM, 2021.
[54] Huy Phan. Pytorch models trained on cifar- 10 dataset. https:: /tt::://itttttt wtt.coo///c/aeeee com/huyvnphan/ PyTorch_CIFARIO 2013.
13

---

[55] Maarten G Poirot, Praneeth Vepakomma, Ken Chang. Jayashree Kalpathy-Cramer, Rajiv Gupta, and Ramesh Raskar. Split learning for collaborative deep learning in healtheare. arXiv preprint arXiv: 19/2. /2/15, 2019.
[56] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01 108, 2019.
[57] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, volume 115 of Proceedings of Machine Learning Research, pages 5744584.44 AUAI Press, 2019.
[58]
Stanford. [TinyImageNet download link]. tiny- imagemet-200.020 zip 2023.
http://cs23in stanford. edu/
[59] Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, Seyit Camtepe, and Lichao Sun. Splitfed: When federated learning meets split learning. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty- Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22- - March 1, 2022, pages 8485-8493, AAAI Press, 2022.
[60] Tom Titcombe, Adam J Hall, Pavlos Papadopoulos, and Daniele Romanini. Practical defences against model inversion attacks for split neural networks. arXiv preprint arXiv:2104.05743, 2021.
[61] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Deep image prior. In 2018 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 9446-9454 Computer Vision Foundation IEEE Computer Society, 2018.
[62] Harry L Van Trees. Detection, estimation, and modulation theory, part I: detection, estimation, and linear modulation theory. John Wiley & Sons, 2004.
[63] Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learn- ing for health: Distributed deep learning without sharing raw patient data. arXiv preprint 1812:: 2018
[64] Praneeth Vepakomma, Abhishek Singh, Otkrist Gupta, and Ramesh Raskar. Nopeck: Informa- tion leakage reduction to share activations in distributed deep learning. In 2020 International Conference on Data Mining Workshops (ICDMW), pages 933-942. IEEE, 2020.
[65] Praneeth Vepakomma, Abhishek Singh, Emily Zhang, Otkrist Gupta, and Ramesh Raskar. Nopeek- -nfper--iffer: Preventing face reconstruction attacks in distributed inference after on-premise training. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recogni- tion (FG 2021), pages 1-8. IEEE, 2021.
[66] Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Dasa-Bianchi,co and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 3839-3848, 2018
[67] Patrick von Platen. google/ddpm--cffr11-233 10-32. Selpm-cifar10 2022.
[68] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Oner Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[69] Zhou Wang and Alan C Bovik, A universal image quality index. IEEE signal processing letters, 9(3):81-86 2002.
14

---

[70] Liyao Xiang, Hao Zhang. Haotian Ma, Yifan Zhang, Jie Ren, and Quanshi Zhang. Interpretable complex-valuee neural networks for privacy protection. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020.
[71] Hanshen Xiao and Srinivas Devadas. Dauntless: Data augmentation and uniform transformation for learning with scalability and security. IACR Cryptol. ePrint Arch., page 201, 2021.
[72] Hanshen Xiao and Srinivas Devadas. PAC privacy: Automatic privacy measurement and control of data processing. In Helena Handschuh and Anna Lysyanskaya, editors, Advances in Cryptology CRYPTO 2023 43rd Annual International Cryptology Conference, CRYPTO 2023, Santa Barbara, CA, USA, August 20-24, 2023, Proceedings, Part II, volume 14082 of Lecture Notes in Computer Science, pages 61 1-444. Springer, 2023.
[73] Adam Yala, Homa Esfahanizadeh, Rafael G. L. D'Oliveira, Ken R. Duffy, Manya Ghobadi, Tommi S. Jaakkola, Vinod Vaikuntanathan, Regina Barzilay, and Muriel Médard. Neu- racrypt: Hiding private health data via random neural networks for public training. CoRR, abs/2106.02484, 2021.
[74] Qiang Yang. Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
15

---

7 Appendix
7.1 Score matching details
We found that using score matching 7 does not work reliably when the data's structure lies low-dimensional manifold (e.g,, natural images). We found that applying randomized smoothing [2], which adds Gaussian noise to the image for robust training, helps stabilize score matching as smoothens the density function. Randomized smoothing also makes the bound tighter. We observed that adding reasonable amount of noise (e.g,, standard deviation of 0.25, which was originally used by Cohen et al. [9]) works well in general. but adding only small noise (standard deviation of 0.01)
does not. We show both results in Section 3.3.T
7.2 Hyperparameters
Attacks For attacks in Section 3.3.1 3.3.2 and 4.2, we used the following hyperparameters. For the optimizer-based attack for Gaussian synthetic input, we used Adam with Ir=10 3, and 0=0.1-100 for the regularizer. For the optimizer-based attack for NCF-MLP and DistilBert, we used Adam with 1r=0.1. For the DNN-based attack for MNIST and CIFAR-10 (Figure 2 DNN from Li et al. [45]. which uses a series of convolution (Conv) and convolution transpose (ConvT) T) layers interspersed with leaky ReLU of slope 0.2. All the models were trained for 100 epochs using Adam with 1r=10-3 Below summarizes the architecture parameters. For DNN-based attacks in Section 3.3.2 we put a sigmoid at the end. For the attack in Section
we used modified
we do not.
Table 3: DNN attacker architectures used in the paper. Output channel dimension (Cout), kernel size (k), stride (s), and output padding (op) are specified. Input padding was I for all layers.
Dataset encoder
Architecture
3 3xConv(cour 6, k=3, s= Conno( - k=3, s=1, op=0)
MNIST + Conv
+ k
xx Conv 64, k=3, 501) ConTT( k=3, s=1, op=
CIFAR-10 + split-early
+
k=33,
3 x Con
28, k=3, s=1) ConvT(cou= 1323 k=3, s=2, op=l)
CIFAR- 10 + split-middle
+ Cony k=3, s=2, op=l)
3 Conv(Com = 256, k=3, 2x
k=3, s=2, op=l)
CIFAR- 10 + satit-late
+ ConvT(C.uu k=3, s=2, op=l)
Split inference Below are the hyperparameters for the models used in Section 4.2 For ResNet-18. we used an implementation tuned for CIFAR- 10 dataset from [54]. with ReLU replaced with GELU and max pooling replaced with average pooling. We used the default hyperparameters from the repository except for the following: bs= 128, 1r=0.1, and weight 5 x 10 For NCF-MLP, we used an embedding dimension of 32 and MLP layers of output size [64, 32, 16, 1]. We trained NCF-MLP with Nesterov SGD with momentum=0.9, 1r=0.1, and batch size of 128 for single epoch. We assumed 5-star ratings as click and others as non-click. For DistilBert, we used Adam optimizer with batch size of 16, lr= x 10-5, 5 =,=0,, =090,9 and e = 10-8, We swept the compression layer channel dimension among 2, 4, 8, 16, and the SNR regularizer between 10-3 - and 100.
Training Below the hyperparameters for the models evaluated in Section 5.2 We used the same model and hyperparameters with split inference for training the encoder with the pretraining dataset. Then, freeze the layers up block 4 and trained the rest for 10 epochs with 10. with 1r=10-3 3 and keeping other hyperparameters the same.
Execution Environment All the evaluation was done on single A100 GPU, The training and evaluation of each model ranged roughly from less than an hour (ResNet- 18 with split-early, NCF. MLP) to 3-7 hours (ResNet- 18 with split-late, DistilBert).
7.3 van Trees inequality
Below, we restate the van Trees Inequality
19 which we use to prove Theorem
16

---

Theorem 2 (Multivariate van Trees inequality), Let (X: 8 )) be family of distributions on a sample space X' dominated by p. Let p(x(0) denote the density of X ~ Po and [ denotes its FIM. Let 0 E @ follows a probability distrihution with density X- (0) with respect to Lebesgue measure. Suppose that X and p(x are absolutely y-almosst surely continuous and x converges to 0 and the endpoints of O. Let và be an absolutely continuous function of 0 with the same output dimension of 0, and Un an arbitrary estimator of (0). Assume regularity conditions from Section 2.2
is met. If we make observations {x1, X2. Xny. then:
1 E Eolli - (((0)) 2
div ((0))(0)))0))
n (0)2, TAm))))))
where div (0) D,
(1) 30;
7.4 Proof of Corollary !
Proof. Let &b be an identity transformation (0) 0. For the setup in Corollary
! 1 and
(i)( = d, so the multivariate van Trees inequality from Theorem
2
reduces to:
E Ro/ x12/42
E, [dFIL(x) + Tr(3(fa))/d
7.5 Comparison with differential privacy
Differential privacy [ is not well-suited for instance encoding, as we discuss in Section 2.1 We formulate and compare a DP-based instance encoding and compare it with our dFIL-based instance encoding in a split inference setup (Section to show that DP-based instance encoding indeed does not work well.
To formulate DP for instance encoding, we define an adjacent set 'D and D' as two differing inputs. randomized method A (a, e)-Rényi differentially private (RDP) D. ((DD)]
for P(w) )a] As DP provides a different privacy guarantce with dFIL, we [[ Q(m)
(PII)) = ب NoggE>-1 use the theorem from Guo et al. [20] to derive an MSE lower bound using DP's privacy metric for an unbiased attacker. Assuming reconstruction attack
Att(c) that reconstructs x from the encoding e = Enc(x), repurposing the theorem Guo et al. 20 gives:
Ed diam, (
E|| 185-xppp x112/d]
(6)
e
for a (2, R)-D Enc, where A' is the input data space. We can construct a (2, <)-RDP encoder ENCADP from a deterministic encoder Encp by scaling and clipping the encoding adding Gaussian noise, or EnCEDP = Encn(x)/ max(1, [[xen(x)]la + (00 o²), similarly to [T]. The noise to be
added is a (2C)²
52 Equation 6 for DP is comparable to Equation for dFIL, and we use the
two equations to compare DP and diIL parameters. We Equation because 20 does not discuss
the bound against biased attackers.
We evaluate both encoders for split inference using CIFAR-10 dataset and ResNet-18. We split the model after block 4 (split-middle from Section I and did not add any optimizations discussed in Section 4 for simplicity. For the DP-based encoder, we retrain the encoder with scaling and clipping so that the baseline accuracy without noise does not degrade. We ran both models without standardizing the input, which makes diam,
for all i.
Table f compares the test accuracy achieved when targeting the same MSE bound for an unbiased attacker using dFIL and DP, respectively. The result clearly shows that DP degrades the accuracy much more for similar privacy levels (same unbiased MSE bound), becoming impractical very quickly. DP suffers from low utility because DP is agnostic with the input and the model, assuming
17

---

Table Test accuracy when targeting the same MSE bound.
Unbiased MSE bound
le-5
le-4
le-3
le-2
dFIL-based
93.09%
93.11 %
92.52% 46.46%
87.52% %
DP-based
64.64%
56.68%
33%
worst-case input and model weights. Our dFII.-based bound uses the information of the input and model weights in its calculation of the bound and can get tighter bound.
7.6 Attack based on diffusion model
We additionally designed a powerful, diffusion model-based reconstruction attacker to study the privacy of dFIL against the best-effort attacker, motivated from the fact that recently developed diffusion models (30) are excellent denoisers. During training of diffusion model, (1) a particularly- designed noise is added to an image, and (2) DNN is trained to predict and remove the noise (30). The first part can be thought of as an instance encoder (that is purposely made easy to invert), and we can calculate its dFIL. The second part can be thought of as a reconstruction attacker. As the noising and denoising are specifically designed for the denoising to work well, we expect mature, pretrained diffusion model to give very good attack quality. We used DDPM [30] pretrained with CIFAR-10 10 from Google [67].
Figure 7 shows the result. The first column of each row shows the original image, and other columns show the reconstruction our DDPM-based attacker with different dFIL. We scale and show dFIL with the same scale as Figure as DDPM works with a different normalized image that produces dFIL different scale. Our new attack provided an interesting result: the attack was able to reconstruct conceptually similar images with the original image even when pixel-by-pixe reconstruction was prohibited by high 1/ aFIL. For example, 7th row at 1/ dFIL. = 49.5 (6th column) successfully reconstructed white car with red background, although the reconstruction MSE was high and the design of the reconstructed car was nothing like the original image. The result shows that high-level information of the image (e.g., the color of the car/background the orientation of the car, etc.) can still be preserved after encoding with relatively high 1/ dFIL, which is why is possible to perform downstream traiiing/nfference with privately-encoder data without revealing the original data.
7.7 Per-pixel FIL
Instead of dFIL which the average of the Fisher information leakage (FIL) for all the features (pix- els), can directly inspect the per-feature FIL to see which features (pixels) leak more information and can reconstructed more easily. Figure
8 plots the per-pixel FIL from the split-middle setup of ResNet-18 8 and CIFAR-10 from Section 4.2.2 The result shows that identifying features of an object, such as the contour of a frog (8th column) or the tire of the car (7th column), are more easily leaked, following our intuition.
7.8 Additional figures and tables
Table The reconstruction quality of an input is highly correlated with dFIL. Correct parts in bold. Reconstructed text (from split-early)
1/FFIL
10-5
it's charming and often affecting journey.
it's cones charmingound often affecting journey closure
grounds yuki cum sign recklessound fanuche pm stunt
10
18

---

1/dFIL 0.494
L/dFIL 2.19
1/dFIL 6.42
1/dFIL 17.4
1/dFIL 49.5
1/dFIL 159
1/dFIL 606
1/dFIL 2780
1/dFIL 15600
ref
Figure 7: Results from DDPM 30 based reconstruction attack. High 1/ dFIL prevents exact pixel- to-pixel reconstruction, but images that share some high-level features with the original image can be generated unless 1/ dFTL is not too high.
Driig
FIL
Overaldd
Figure 8: Per-pixel FIL from the ResNet-18 CIFAR-10 split-middle setup in Section 4.2.2 best viewed with colors.
Image
19

---

1.00
split early middle split-laee
0.75
early
SSIM
middle
0.25
0.00
late
1c:05
0.0001
0.001
0.01
0.1
0.0001
0.1
100.0
1/dFIL
1/FFIL SSIM
(b) 1/dFIL vs. reconstructed image quality
Figure 9: Optimizer-basee based attack with total variation (TV) prior 49 against our split inference
system Section The trend very similar to Figure
7
2
9
1e-05
7
2
7
4
9
0.0001
7
2
7
O
4
9
0.001
7
2
/
O
4
4
9
0.01
7
2
7
4
4
9
0.1
7
2
7
4
4
9
1.0
7
2
/
O
4
4
9
10.0
7
2
Q
9
25.12
Figure 10: More reconstruction result of Figure
2
20

---

1o.05
0.0003
0.001
0,01
0.1
1.0
10.0
100.0
Figure 11: More reconstruction result of Figure
3
1o:05
0.0001
0.001
0.01
0.1
1.0
10.0
100.0
Figure 12: More reconstruction result of Figure
4
21

---

split-early, 1/dFiL=1.0
split-early. 1/dFIL=1.0
ref
ref
recon
split- early, 10.0
split-early, 1/dFiL=
recon ref
ref recon
split-middle, 1/dFiL=1.0
split-middle, 1/dFIL=1.0
recon ref
split-middie, 1/dFIL- 10.00
split-middie, 1/dFIL- 10.0
recon ref
recon
split-late, 1/dFIL=1.0
split-late, 1/dFIL=1.0
ref
recon
split-late, 1/dFIL-10.0
split-late, 1/dFiL AdFIL-10.0
recon ref
recon
(a) Reconstructions with the worst SSIM values.
(b) Reconstructions with the best SSIM values.
Figure 13: Ten reconstructions with the best and worst SSIM values for various setups. The result is an extension of Figure Images with simple shape and high color contrast tend to be reconstructed more easily, which matches our intuition. Omitting results from 1/ dFIL 100, as no meaningful reconstruction was possible.
Lowest dfil
Highest dfil
Ten images with the lowest and highest dFIL values, for split-middle setup in Figure 5 Images with high dFIL. tend to have a simpler shape and high color contrast, potentially being easier reconstruct. Individual dFIL. value of each samples potentially be used to detect data that more leaky.
Figure 14:
22


Reviews:
[{"review_id": "UzD18s7j3d", "reviewer": "Reviewer_KGoq", "summary": "This paper introduces a new theoretical measure, the \"diagonal Fisher Information Leakage\" (dFIL), for quantifying the privacy leakage in instance encoding mechanisms in machine learning models. The authors construct a framework that balances the trade-off between data privacy and utility in instance encoding.\n\nThe authors start by developing the mathematical foundation of dFIL, leveraging the principle of Cramér-Rao Lower Bound (CRLB). They present a theoretical analysis which demonstrates that dFIL serves as an upper bound on the Mean Squared Error (MSE) of any estimator trying to reconstruct the original input from the encoded instances.\n\nThe paper presents two case studies to showcase the practical application of dFIL. The first one focuses on private inference, where instance encoding is applied to an existing pretrained model. The second case study explores the possibility of training a model on instance-encoded data. The results from these studies confirm that models guided by dFIL can achieve reasonable performance while ensuring a high degree of privacy.\n\nDespite these advancements, the authors acknowledge the limitations of dFIL, such as the MSE bound not always correlating well with the semantic quality of the reconstruction, the average case bound provided by dFIL, and the difficulties of interpreting dFIL values in certain data types where MSE might not be directly meaningful or accurate. Recognizing these limitations, the paper calls for further research to explore and optimize the use of dFIL in designing and analyzing systems that use instance encoding for private inference and training.", "questions": "1. The paper uses mean squared error (MSE) as a metric to quantify the quality of reconstructions. However, as the authors acknowledge, this may not always correlate with the semantic quality of reconstructions. Could you please elaborate more on the choice of this metric and how it might affect the overall performance of the system? And the paper mentions that for certain data types where MSE is not directly meaningful, interpreting the privacy of an encoding given its dFIL might be challenging. Could you provide examples of such data types and explain how dFIL might be adapted to handle them?\n", "limitations": "The authors have done a commendable job addressing the potential limitations of their work. They detail four specific limitations in section 6:\n\n1. dFIL only bounds the mean squared error (MSE), which may not always correspond well with the semantic quality of reconstructions.\n2. The derived bounds are average-case, meaning that individual data instances could experience lower reconstruction MSE than the bound.\n3. For data types where MSE is not directly meaningful or the bound is inaccurate, interpreting the privacy of an encoding given its dFIL might be difficult.\n4. Systems with the same dFIL may have different invertibility, as the bound from dFIL could be conservative.\n\nThese limitations are articulated clearly, and the authors also suggest possible directions to address these limitations, such as exploring metrics other than MSE and calculating dFIL dynamically for each sample.\n\nIn terms of the broader societal impact, the paper does not directly address this aspect. However, it's understandable, given the theoretical nature of the work. Future research could consider potential misuse of this technology, as well as possible regulatory or ethical issues that might arise from its application, particularly in scenarios where private or sensitive data are involved. Nonetheless, the paper's focus on improving privacy in machine learning is itself an important societal contribution, as it addresses a growing need in an increasingly data-driven world.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Strengths:\n\n1. **Originality**: The concept of Diagonal Fisher Information Leakage (dFIL) is novel, offering a fresh perspective on privacy leakage quantification in instance encoding for machine learning models. The authors have done a commendable job in providing a new metric that has potential broad applications in designing and analyzing private machine learning systems.\n\n2. **Quality**: The theoretical foundation of dFIL is well-crafted and robust, using the Cramér-Rao Lower Bound to derive the maximum leakage. This gives dFIL a solid grounding in statistical theory. The authors carefully reason about the limitations of their approach and provide a roadmap for further exploration.\n\n3. **Clarity**: The paper is well-written and structured, making the complex concept of dFIL comprehensible. The authors articulate the theoretical underpinnings and practical implications of dFIL with clarity and precision. The use of case studies adds to the understanding of how dFIL can be applied in practical scenarios.\n\n4. **Significance**: The presented work is of significant value as it paves a way for protecting data privacy in instance encoding mechanisms, a critical issue in today's data-centric machine learning landscape. The demonstrated applications in private inference and training suggest the potential of dFIL to guide the design of machine learning systems that respect privacy constraints while maintaining utility.", "weaknesses": "1. **Limited empirical validation:** Although the theoretical development of dFIL is thorough and robust, the empirical validation seems somewhat limited. The authors conduct case studies on split inference and training with dFIL, which are commendable. However, additional experiments, especially on more diverse datasets and models, could strengthen the validity and generalizability of the dFIL measure. \n\n2. **Absence of comparative analysis:** While the paper introduces dFIL as a novel concept, it doesn't provide a comparative analysis with other similar measures, if any exist. This makes it difficult for readers to evaluate the practical advantages or disadvantages of dFIL in contrast to other measures or approaches. \n\n3. **Limited exploration of potential use-cases:** The authors present two case studies, however, the range of potential applications for dFIL could be broader. Additional use-cases illustrating how dFIL can guide the design of private machine learning systems would provide a more comprehensive view of the practical utility of dFIL. \n\n4. **Interpretability of dFIL:** As the authors acknowledge in the Limitations section, for data types where MSE is not directly meaningful or the bound is inaccurate, interpreting the privacy of an encoding given its dFIL may not be straightforward. More investigation is needed to establish a clear and intuitive interpretation of dFIL values, especially in the context of real-world applications.\n   \n5. **Assumptions and constraints:** The assumptions underlying the derivation of dFIL and its applications need further exploration. For example, how sensitive is dFIL to the assumptions of model linearity, Gaussian noise, and others? The impact of violating these assumptions, and how they can be mitigated, would be valuable areas to explore.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the insightful review and feedback. We provide our answers to the questions and comments below. For the new evaluation data and their brief explanation, please check the global response.\n\n**Comparison with similar measures**. Despite the popularity, there is *very little work on theoretical privacy analysis for instance encoding*, leaving us with no good alternative measures to compare. We discuss the lack of related works under the “Privacy metrics for instance encoding” paragraph in p. 2. The lack of theoretical measures in this popular field makes our work unique and important. We summarize the limitations of some of the related works below:\n- **Metrics without theory**. Some proposed *empirical* measures, such as mutual information (MI) [45] or distance correlation (dCorr) [21, 22]. However, these works failed to theoretically show how these measures are related to any form of privacy, if related at all (i.e., they lack what is often called an *operational interpretation*, which is considered essential in privacy literature [38]). As we explain in Line 95--98, prior works [38] warned against using such seemingly-valid but unjustified measures, because they can give a false sense of privacy. For example, [38] showed that MI sometimes mischaracterizes the severity of the information leakage (a system with a lower MI can actually be less private) and is not a good measure of privacy. As these measures (MI, dCorr) are not theoretically justified and can be misleading, we do not quantitatively compare them with dFIL. We will enrich Section 2 with the above qualitative discussion in the final draft.\n- **Metrics with theory**. There are metrics with operational interpretations; however, dFIL is much more practical than the others. Metric-DP [24] is not applicable to encoders used in Section 4 and 5 (Line 89). Differential privacy [30] degrades the utility too much (Line 82, Appendix: Section 7.5). A recent concurrent work proposed a new PAC privacy [a] that can bound the adversary’s success rate. However, the bound from [a] gets very loose on instance encoding setups we studied. For 1/dFIL=10 which was empirically hard to invert (Figure 5), the adversary attack success rate upper bound from [a] becomes 1 --- a trivial bound that does not give any practical information. We will add the discussion regarding [a] in the final draft. \n\n**Assumptions and constraints need further exploration**. dFIL is generally applicable to any differentiable and randomized methods, using the general Equation 1. The noise does not have to be Gaussian, and using different sources of randomization can lead to different privacy-accuracy tradeoffs. We leave such an exploration to future works. The differentiability (e.g., absence of ReLU or MaxPool) assumption is necessary for the van Trees inequality and the Cramer-Rao bound to hold. Although the bound may empirically work similarly without the differentiability assumption, we confined the scope of the paper to encoders where our Corollary can be mathematically precise. Please let us know of any other assumptions/constraints that the reviewer wants to discuss, if there are any.\n\n**Interpretability of dFIL**. We agree with the reviewer that interpreting dFIL might not be straightforward in some applications. As mentioned in Limitation 3 (p. 9), in such cases, acceptable values of dFIL should be determined for each application through further research. The well-adopted DP has similar issues: in many real-world scenarios, what $\\epsilon$, $\\delta$ means for a particular setting is not straightforward, and the acceptable values are often chosen empirically [69].\n\n**MSE as a metric**. For some data types (e.g., user’s geographical location), MSE can directly provide an intuitive notion of privacy. For other data types such as image or word embeddings, MSE might not exactly capture the semantic similarity of the reconstruction. In this work, we use MSE as a proxy to similarity as in prior works [47, 48], but the approach has limitations: two images with relatively high MSE may look semantically similar to humans. For example, see the images containing a white car on a red background in Appendix: Figure 7. Two images that are very different pixel-wise may still convey similar high-level information (e.g., the fact that there is a white car facing left in front of a red background), and MSE cannot capture the fact that such high-level information is still leaking, while it disallows an exact pixel-by-pixel reconstruction. The users of dFIL must understand these implications of dFIL and use it only when the privacy it provides makes sense. We believe our extensive discussions and evaluations will help future researchers to adopt dFIL in a meaningful way.\nAs discussed in Limitation 1, van Trees inequality can be extended to potentially support metrics other than MSE (see Appendix: Theorem 2), which may be useful in some cases. We leave such an extension to future work. \n\n**Limited empirical validation**. We want to emphasize that our case study spans *three distinct applications (image classification, recommendation, and sentiment analysis)* that use *three different input data types (image, user behavioral features, and natural language)*, showing the generality of dFIL. Following the reviewers’ suggestion, we additionally add the CIFAR-100 dataset to the split inference evaluation in **Supplementary PDF: Table 1**. The trend is similar to Table 1 in the original paper.\n\n[a ] Hanshen Xiao, and Srinivas Devadas. \"PAC Security: Automatic Privacy Measurement and Control of Data Processing.\" Crypto 2023 (to be appeared this August).\n"}, {"review_id": "cudFvx17lY", "reviewer": "Reviewer_k1TB", "summary": "Instance encoding (and some closely related lines of research) aim at finding ways to encode data examples (in a training set) as $E=Enc(e)$, in such a way that one can train models on the encoded examples $E_1,...E_n$, while $E_i$ does not leak much about $e_i$ to an adversary who inspects it.\n\nPrevious efforts for proposing \"private\" instance encodings have failed due to (1) subsequent attacks, and (2) lack of formal definitions of what privacy means here. In fact, it is known that such mechanisms cannot be DP in a strong sense, while they are also useful for training.\n\nThis paper aims to put forward a new way of arguing about privacy of instance encoding, based on Fisher Information (FI). FI is a useful measure of information revealed in random variables about a related random variable. In particular, knowing FI allows one to lower bound the Mean Squared Error (MSE) of the best reconstruction attack, assuming that the adversary has no prior knowledge about the distribution of the original instance.\n\nMore specifically, using FI, the paper shows how to give an *hardness of inversion* type privacy. The paper shows that both in the (more elementary) case of unbiased attacks (who see the original instances uniformly distributed) as well as more powerful attackers who have some prior information (e.g., know the distribution of the instances), one can lower bound the MSE of the best possible (information theoretic) attack, based on two parameters: FI of the encoding and a second parameter that depends on how flat/concentrated the distribution of the instance x is. Note that the second parameter also inherently shows up as one can approximate x well if it is too concentrated already. \n\nThe paper also discusses how to compute the two parameters of interest, or at least approximate them, in practical settings. \nIn particular, when the encoder is a Neural Net, the paper discusses how to use tools from previous work to approximate them using smooth functions that will be suitable for computing their FI.\n\nThe final product is an \"average case\" notion of privacy that is tailored for inputs coming from (flat enough) distributions.\n\nThe paper also presents experiments to validate its theory.", "questions": "see the questions from the previous box.\n\nis there any hope that your approach can say something about the expected $\\ell_\\infty$ distance between adversary's guess and the true instance (rather than MSE) ? ", "limitations": "See the \"weakness\" section for some comments related to limitations as well.\n\nOn the positive side, the paper itself has a a good \"limitations\" section at the end.\n\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Differential privacy is the golden standard for privacy at the moment. But it is also a good idea to look for new notions privacy that could be meaningful on their own and useful for certain settings. Hence, I find the overall goal of the paper positive and useful for putting new perspectives on privacy on the table.\n\nI also liked the fact that the paper made an effort to algorithmically compute their notion of privacy by approximating the two parameters of interest using tools from previous work.", "weaknesses": "(also a limitation) Bounding MSE sometimes might not say anything about privacy, when all an adversary wants to do is finding a specific sensitive feature (and not approximating the instance in $\\ell_2$ norm). So, more (context dependent) work is needed for finding whether this notion is actually a good notion of privacy or not.\n\n(also a question) The figures are not clear. For example, in Figure 2, what is the column numbers (Ref, 25, 10, 1) next to MNIST images?\n\n(also a question and limitation) The paper does experiments in which the MSE of certain encoders are evaluated. However, it is not clear to me how *useful* these encodings are *for training models* over the encoded instances. Note that without knowing how useful the encodings are, there is always a trivial fully private encoding: output $\\bot$ all the time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the insightful review and feedback. We provide our answers to the questions and comments below. For the new evaluation data and their brief explanation, please check the global response.\n\n**dFIL against sensitive attribute inference**. The initial motivation for dFIL was to protect against input reconstruction, which significantly hampers user privacy directly if allowed [28]. Still, dFIL can be adopted to protect sensitive attributes in some cases, if the attribute is contained within a subset of the input features. Currently, dFIL is an aggregate value across all features (e.g., pixels). Alternatively, we can calculate a per-feature FIL by looking at each entry in the diagonal of FIM instead of calculating the trace (Eq. 3) [27]. The per-feature FIL then gives us the reconstructability of each feature (pixel) individually. **Supplementary PDF: Figure 2** shows an example of this per-feature FIL for different images, for the split-middle architecture in Section 4. The figure shows that the key points of an image (e.g., contours of an object, tire of a car, tag of a cat, etc.) leak more through the encodings and can be easily reconstructed. If the sensitive attribute is contained in a subset of the image (e.g., eye color, background, …), we can measure its reconstructability directly using this per-feature FIL and understand if the attribute can be reconstructed by an adversary. If the sensitive feature is not contained within a few pixels (e.g., ethnicity), it would be harder to measure its leakage. A potential future direction is to disentangle the sensitive feature into a certain dimension in a latent space [a, b] and use dFIL in the latent space.\n\n**Figures are not clear (what are the column numbers in Fig 2?)**. The values indicate the 1/dFIL values (Line 231--232), which means more private if the value is higher. We will refine the figures in the final draft to make them more clear.\n\n**Utility of the encoders in Sec 3**. We perform two separate evaluations. First, in Section 3, we evaluate whether our MSE bound holds well against attacks. Second, in Section 4--5, we evaluate whether an encoder with a reasonably-low dFIL (high MSE bound) can still have good utility. We separate out the evaluations because we wanted to evaluate our bound in Section 3 against the most adversarial setup to see if it always holds. \n\nEncoders used in Section 3 were deliberately chosen to be simple and relatively easy to invert (e.g., single-layer convolution with a large output channel), and as they are designed solely for the ease of attack and not for utility, it is almost certain that they will not be very useful in training models. The goal of encoders in Section 3 was to show that the attackers cannot achieve an MSE below our bound *even for the encoders that are deliberately designed solely to make the attack easy (and do not care about utility)*. The results from Section 3 show that our MSE bound is reliable even in these adversarial cases.\n\nThe second set of evaluations in Section 4--5 studies whether an encoder with a reasonably-low dFIL can still be useful in training/inference. For these encodings, we did not try inverting and comparing their MSE with the bound, as these practical encoders are already empirically harder to invert than the encodings that are deliberately made easy to invert (from Section 3), and we already showed that even the easier ones cannot be inverted to break the bound.\n\n**Can we bound L-inf?** While we cannot directly bound the expected L-inf, we can adopt the dFIL to bound something similar. Instead of calculating dFIL which is the average quantity of all features, we can calculate the expected squared L2 distance for each feature (the per-feature FIL discussed in the first bullet). If we take the max of these instead of an average, we can get the expected “squared” L-inf, which is similar (but not the same) with L-inf.\n\n[a] Ricky TQ Chen, et al. \"Isolating sources of disentanglement in variational autoencoders.\" NeurIPS ‘18.\n\n[b] Zheng Ding, et al. \"Guided variational autoencoder for disentanglement learning.\" CVPR ‘20. "}, {"review_id": "Yo6RDAr9Os", "reviewer": "Reviewer_yPPy", "summary": "This paper discusses the privacy concerns surrounding data encoding methods employed in machine learning (ML) operations. Its primary objective is to present a theoretical framework that quantifies the extent of privacy leakage in an encoding scheme and facilitates the calculation of its invertibility. The authors propose a novel technique that leverages Fisher Information Leakage to compute the lower bound of Mean Squared Error (MSE) for an encoding scheme, considering both unbiased and biased attackers. Finally, The authors evaluate their technique against some simple reconstruction attacks both for training and inference tasks.\n", "questions": "See weakness.", "limitations": "No", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper takes a theoretical approach to explain reconstruction attacks. The problem is interesting and the presentation and flow of the paper are good. It was easy to follow and understand.\n2. The authors' approach of initially focusing on unbiased attacks and subsequently extending their scheme to encompass attackers with prior knowledge is a good approach and beneficial in comprehending their contributions. The paper also provides evaluation of their technique considering both unbiased and biased attackers.\n3. The paper is upfront about many of its limitations. This is commendable and helps understand the solution better and paves the way for identifying future research avenues.\n", "weaknesses": "1. The core contribution of the paper is adopting Fisher Information Leakage and van Trees inequality to come up with a lower bound for MSE. While this is a new idea, it would benefit from more comprehensive construction. Furthermore, the reconstruction attacks examined by the authors to showcase the efficacy of their technique are relatively simplistic. It would be advantageous for the authors to consider incorporating relevant literature from the privacy and security domain and evaluate their approach against more sophisticated and state-of-the-art reconstruction attacks.\n\n2. The paper should provide more rigorous analysis of the privacy and utility trade-off. The paper readily discards Differential Privacy on utility ground, but later adopts noise addition to introduce privacy. Regardless, the privacy guarantee by noise addition and the corresponding utility trade-offs should be analytically explained. While the paper includes an empirical demonstration of the utility trade-off, it falls short of its intended goal of providing a theoretical explanation. \n\n3. The authors should clarify how this lower bound contributes to preventing attacks, developing privacy-preserving encoding techniques, or addressing other important aspects related to data privacy.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the insightful review and feedback. We provide our answers to the questions and comments below. For the new evaluation data and their brief explanation, please check the global response.\n\n**Reconstruction attacks are simplistic**. We want to first highlight that the attacks we studied are based on *state-of-the-art works from top-tier conferences*. We tried both well-cited classics (e.g., CVPR ‘15 [34], CVPR ‘18 [35], ACSAC ‘19 [33]) and recent works (e.g., CCS ‘21 [36], CVPR ‘22 [20]), and chose the best-performing attacks. Especially, our attack-b in Figure 3--4 is directly adopted from CVPR ‘22 [20], which worked the best among the attacks we studied. The field does not have many sophisticated attacks because, unlike model inversion or gradient inversion which is much harder and still requires a lot of research, inverting instance encoding is easy, and existing simple attacks already work well in the absence of a defense like ours. Also, prior instance encoding works did not have theoretical justifications (unlike our work) and were considered not strong in terms of privacy [2, 23], so researchers had less motivation to design sophisticated attacks against them.\n\nNonetheless, we agree with the reviewer that the work can benefit from a more sophisticated attack. Thus, we additionally designed an attack based on a DDPM diffusion model pretrained by Google [73]. Please refer to Appendix: Section 7.6 for more details. Appendix: Figure. 7 shows the reconstructed images with different dFIL. Similar to Figure 3-4, it can be seen that 1/dFIL >= 10 is relatively safe: pixel-to-pixel reconstruction becomes hard for 1/dFIL >= 6.42, and even the high-level semantic information becomes hard to reconstruct when 1/dFIL >= 49.5. At the same time, the figure shows that sometimes the high-level information of the image can be reconstructed (e.g., the fact that there is a white car facing left), even when our dFIL bound disallows exact pixel-by-pixel reconstruction. The result gives us an insight into what kind of information can still leak, even when a pixel-by-pixel reconstruction is prohibited by Corollary 1.\n\nWe additionally plot what is equivalent to Figure 3 (left) for the diffusion model attacker in **Supplementary PDF: Figure 1**. The figure shows that the bound again works well even for the diffusion model attacker. Here, we used inputs with randomized smoothing noise as in Figure 3 to calculate the bound, and because the pretrained model is not trained with the randomized smoothing noise, we trained the diffusion model for 5000 epochs with the noisy inputs, following the hyperparameters from [73].\n\n**Why discard DP, but later adopts noise addition?** We want to clarify that adding noise is a privacy mechanism that is more general than DP, and our approach, although it adds noise, is not differentially-private. Prior work [2] theoretically proved that indistinguishability, which differential privacy aims to provide, is *fundamentally incompatible* with utility in instance encoding. As the privacy notion of indistinguishability is impossible to achieve, our approach is designed to achieve an alternative (weaker) privacy, non-invertibility. While it is also possible to achieve non-invertibility with DP methods, as DP was originally designed for indistinguishability, the privacy-utility trade-off becomes much worse. We show the empirical comparison with DP in Appendix: Section 7.5.\n\n**Lack of more rigorous theoretical analysis of the privacy and utility trade-off**. The main contribution of this paper is to (1) theoretically show that dFIL is a useful privacy metric that can bound the invertibility of instance encoding, and (2) empirically show that an instance encoding with a low dFIL can still have reasonable utility. Providing a theoretical analysis of the privacy-utility trade-off was not part of the scope of this paper, and we leave it as a future work.\n\nWe believe our theoretical contribution is still meaningful, as we are the **first** work to provide a meaningful privacy metric that can bound the invertibility of an instance encoding. The result will help future researchers to design instance encoders using theoretical principles, rather than relying on empirical claims. Also, we do so by connecting the van Trees inequality with score matching in generative AI, a novel connection first made by our work. It allows bounding the posterior success rate of arbitrary adversaries for real-world data (e.g., CIFAR10) with intractable priors, which is known to be challenging in general. Prior works on similar fields usually assume data with simple priors like Gaussian or uniform [a, b]. We believe our idea of connecting score matching to capture real-world data priors can be extended to similar fields.\n\n**How does the lower bound contribute to developing privacy-preserving encoding?** dFIL allows us to quantitatively measure the invertibility of an instance encoding for the first time. System/model designers can now measure and compare their encoding techniques’ invertibility and select encodings that are harder to invert. This is exactly what we did in Section 4--5, where we tried out several empirically-proposed techniques from prior works  [17, 20, 22, 44, 58, 59] and chose techniques that provided the lowest dFIL (better privacy) with the same accuracy (Line 262--264). Before our work, there were no known ways to theoretically measure how private or not private each technique is, other than fully empirical methods.\n\n**The idea would benefit from more comprehensive construction**. We could not understand what this sentence meant exactly. Can you please elaborate, so that we can respond and/or add more results if necessary?\n\n[a] Arpita Ghosh et al., \"Inferential privacy guarantees for differentially private mechanisms.\" Arxiv 2016.\n\n[b] Borja Balle et al., \"Reconstructing training data with informed adversaries.\" S&P 2022."}, {"review_id": "bsnLONiCM7", "reviewer": "Reviewer_AWfY", "summary": "In this paper the authors propose (diagonal) Fisher information leakage (dFIL) as a theoretically-grounded and practical framework for assessing the privacy guarantees of instance encoding. At its core, dFIL quantifies the potential invertibility of the encoding mapping of an instance encoding scheme. The authors establish, under mild regularity conditions, that the reciprocal of dFIL, plus the information theorist's Fisher information, serves as a lower bound for the mean squared error (MSE) of any input reconstruction attack (Corollary 1). This lower bound is derived from the so-called van Trees' inequality. Furthermore, the authors present extensive numerical experiments that highlight the key features of dFIL as a privacy metric for instance encoding.", "questions": "One of the key contributions of this paper is the introduction of dFIL as an intuitive privacy metric for instance encoding. However, it is important to note that, for (general) biased attacks, the information theorist's Fisher information plays an equally, if not more significant role than dFIL. Regrettably, the paper lacks a thorough discussion of the empirical and theoretical properties of the latter quantity, apart from its estimation using score matching. Consequently, it becomes challenging to evaluate the relative importance of dFIL compared to the information theorist's Fisher information. Clarifying these aspects would enhance the overall assessment of dFIL's significance.\n\nAddendum: In their rebuttal, the authors clarified the relative importance of dFIL and the information theorist's Fisher information.", "limitations": "The authors addressed the limitations of the proposed method very well.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. dFIL is a very intuitive privacy metric under unbiased reconstruction attacks. In such cases, the Cramér-Rao bound establishes that the reciprocal of dFIL is a lower bound for the MSE of an adversary attempting to reconstruct the raw data based on the encoding.\n\n2. The limitations of dFIL are discussed thoroughly.\n\n3. The paper is well-written.", "weaknesses": "Although dFIL serves as an intuitive privacy metric under unbiased reconstruction attacks, these attacks are uncommon in ML applications. In the case of biased attacks, which are more representative of practical scenarios, the main result (Corollary 1) highlights the need to consider not only dFIL but also the information theorist's Fisher information. Consequently, certain practical and theoretical concerns are shifted to the latter quantity, potentially diminishing the significance of dFIL itself. Furthermore, it is worth mentioning that the paper's technical contribution is minor since the main result is a direct consequence of van Trees' inequality. While this is not a problem in itself, it does not contribute to make the paper stand out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the insightful review and feedback. We provide our answers to the questions and comments below. For the new evaluation data and their brief explanation, please check the global response.\n\n**Relative importance of dFIL compared to the information theorist's Fisher information**. While it is true that the information theorist’s Fisher information is important in deriving the bound, it is a quantity only related to the dataset, not the encoder design. It can be calculated once and shared with the community for each dataset, and also it can be refined over time within the community; each encoder designer does not have to calculate the information theorist’s Fisher information individually.\n\nMoreover, when comparing several encoder designs’ privacy under the same dataset, the information theorist’s Fisher information is just a constant, so calculating dFIL is more important in comparing each encoder’s relative privacy. Our work focuses on calculating dFIL, as our goal was to develop a metric that can be used to compare and evaluate the privacy of different encoders.\n\n**Minor technical contribution**. While our Corollary is directly based on the van Trees inequality, we are the first to connect the van Trees inequality to score matching in generative AI and use it to provide a bound for a real-world data distribution. Deriving any inferential privacy guarantee (bounding the posterior success rate of arbitrary adversaries) is known to be very challenging for real-world datasets with complex priors, and most existing works (on similar fields) assume simple data priors, such as data following Gaussian or uniform distribution [a, b]. We believe our insight in connecting the concept of score matching in generative AI is novel, and can be extended to other fields on inferential privacy.\n\nWe also want to highlight that the theoretical result of our work, albeit simple, will have a high impact within the field. Currently, the field of instance encoding lacks **any** measure of privacy that is theoretically justified and empirically useful, despite the field’s high popularity. Our work is the **first** to bring theory to instance encoding privacy, which will inspire future researchers in the field to design encoders based on theoretical principles and deter bogus, empirical claims.\n\n[a] Arpita Ghosh et al., \"Inferential privacy guarantees for differentially private mechanisms.\" Arxiv 2016.\n\n[b] Borja Balle et al., \"Reconstructing training data with informed adversaries.\" S&P 2022.\n"}]

Author rebuttal:
We thank the reviewers for their insightful reviews and feedback. We respond to the concerns and questions individually in a separate rebuttal for each review. Here, we upload a supplementary PDF containing additional results that were asked by the reviewers. We also give a brief explanation:

**Table 1** holds an additional evaluation result for split inference, using the CIFAR-100 dataset. The result directly extends Table 1 in the paper, and all the model hyperparameters are the same except for the last fully-connected layer. The result shows that our optimizations again improve the accuracy significantly on a reasonably-low dFIL (e.g., 1.98% -> 59.51%).

**Figure 1** holds a 1/dFIL vs. reconstruction MSE plot similar to Figure 3 (left) in the paper, for a new diffusion model-based attacker that we designed. The details of the diffusion model attacker are in Appendix: Section 7.6. To calculate the bound, we need to add a randomized smoothing noise to the input, so we retrained the model for 5000 epochs with the noise with the hyperparameters from Google [73]. The figure shows that the bound holds reliably for the diffusion model attacker as well.

**Figure 2** plots the per-feature FIL that can bound the reconstruction MSE of individual features (pixels), instead of the average of the entire input. Per-feature FIL can be calculated by directly using the diagonal entries of the Fisher information matrix (FIM), instead of calculating the trace (Eq. 3) [27]. Plotting the per-feature FIL helps us understand which pixels, or attributes occupying that pixels, can be more easily reconstructed. Figure 2 shows that identifying features, such as the contour of a frog (8th column), the tire of a car (7th column), or the necklace of a cat (9th column) leak more and can be reconstructed more easily.

Meta-review: The paper proposes diagonal Fisher Information Leakage as a way to quantify the privacy leakage in instance encoding mechanisms. While  this notion has limitations, the paper does a reasonable job of discussing those. The reviewers found the work to be interesting and any concerns that were raised were largely addressed by the rebuttal. I would encourage the authors to update the paper to incorporate the reviewer feedback and the discussion. I am happy to recommend acceptance.
Decision: Accept (poster)