SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Result Figure 1 (right) plots the MSE of the two attacks, and the bounds for unbiased (Equation 2) and arbitrary attack (Equation 5). The MSE of Attack-ub (red circle) matches the unbiased attack lower bound (Bound-ub; red dashed line), showing the predictive power of Equation 2 against this restricted class of attacks. Under Attack-  $b$  (blue triangle), however, Bound-ub breaks. Our new bound from Equation 5 (Bound-ours; blue dotted line) reliably holds for both attacks, initially being close to the unbiased bound and converging to guessing only with the input prior (attaining  $\tau^2$ ).
3.3.2 Real world data with estimated  $\mathcal{J}(f_{\pi})$
Evaluation setup We also evaluated Corollary 1 on MNIST [10] and CIFAR-10 [42]. Here, we estimated  $\mathcal{J}(f_{\pi})$  using sliced score matching [57]. As discussed in Appendix 7.1, a moderate amount of randomized smoothing (adding Gaussian noise to the raw input [9]) is necessary to ensure that the score estimation is stable and that regularity conditions of the van Trees inequality are satisfied. We used a simple CNN-based encoder:  $\mathbf{e} = \mathrm{Conv}(\mathbf{x}) + \mathcal{N}(0,\sigma^2)$ .
Attacks We evaluated Attack-ub, which is the same as in Section 3.3.1, and Attack- $b$ , which is a trained DNN that outputs the reconstruction given an encoding [45]. We also evaluated regularization-based attacks [49, 61] and obtained similar results; we omit those results for brevity.
Result Figures 2 and 3 plot the result with a randomized smoothing noise of  $\mathcal{N}(0,0.25^2)$ . Again, Bound-ub correctly bounds the MSE achieved by Attack-ub. While Attack- $b$  is not as effective for very low  $1/$  dFIL, it outperforms Attack-ub for high  $1/$  dFIL, breaking Bound-ub. In comparison, Corollary 1 estimated using score matching (Bound-ours) gives a valid lower bound for both attacks.
Figures 2 and 3 also highlights some of the reconstructions visually. Here, the left-hand side number indicates the target  $1/$  dFIL and the images are reconstructed using Attack- $b$ . In both figures, it can be seen that dFIL correlates well with the visual quality of reconstructed images, with higher values of  $1/$  dFIL indicating less faithful reconstructions. See Appendix: Figure 10-11 for more results.
Figure 4 additionally shows the result with a much smaller randomized smoothing noise of  $\mathcal{N}(0,0.01^2)$ . Unlike previous results, Bound-ours breaks around  $1 / \mathrm{dFIL} = 10^{-3}$ . We suspect it is due to score matching failing when the data lie on a low-dimensional manifold and the likelihood changes rapidly near the manifold boundary, which can be the case when the smoothing noise is small. Nonetheless, the bound still correlates well with MSE and the visual reconstruction quality. We claim that dFIL still serves as a useful privacy metric, with a theoretically-principled interpretation and a strong empirical correlation to invertibility. More reconstructions are in Appendix: Figure 12.
4 Case study 1: split inference with dFIL
4.1 Private split inference with dFIL
Split inference [40, 4, 65] is a method to run inference of a large DNN that is hosted on the server, without the client disclosing raw input. It is done by running the first few layers of a large DNN on the client device and sending the intermediate activation, instead of raw data, to the server to complete the inference. The client computation can be viewed as instance encoding, where the first few layers on the client device act as an encoder. However, without additional intervention, split inference by itself is not private because the encoding can be inverted [27].
We design a private split inference system by measuring and controlling the invertibility of the encoder with dFIL. Because the encoder of split inference is a DNN, dFIL can be calculated using Equation 4 with minor modifications to the network (Section 3.1).
Optimizations There are several optimizations to improve the model accuracy for the same dFIL: (1) We calculate the amount of noise that needs to be added to the encoding to achieve a target dFIL, and add a similar amount of noise during training. (2) For CNNs, we add a compression layer—a convolution layer that reduces the channel dimension significantly—at the end of the encoder and a corresponding decompression layer at the beginning of the server-side model. Similar heuristics were explored in prior works [11, 45]. (3) We add an SNR regularizer that is designed to maximize the signal-to-noise ratio of the encoding. From Equations 3-4, the noise that needs to be added to achieve a certain dFIL is  $\sigma = \sqrt{\frac{\mathrm{Tr}(\mathbf{J}_{\mathrm{Enc}_D}^\top(\mathbf{x})\mathbf{J}_{\mathrm{Enc}_D}(\mathbf{x}))}{d*\mathrm{dFIL}}}$ . Thus, maximizing the signal-to-noise ratio (SNR) of the encoding  $(\mathbf{e}^{\top}\mathbf{e} / \sigma^{2})$  is equivalent to minimizing  $\frac{\mathrm{Tr}(\mathbf{J}_{\mathrm{Enc}_D}^\top(\mathbf{x})\mathbf{J}_{\mathrm{Enc}_D}(\mathbf{x}))}{\mathbf{e}^{\top}\mathbf{e}}$ , which we add to the optimizer during training. These optimizations were selected from comparing multiple heuristics from prior work [60, 28, 46, 65, 45, 11], and result in a notable improvement of test accuracy.
4.2 Evaluation of dFIL-based split inference
4.2.1 Evaluation setup
Models and datasets We used three different models and four different datasets to cover a wide range of applications: ResNet-18 [25] with CIFAR-10/100 [42] for image classification, MLP-based neural collaborative filtering (NCF-MLP) [26] with MovieLens-20M [24] for recommendation, and DistilBert [56] with GLUE-SST2 [68] for sentiment analysis. See Appendix 7.2 for details.
Detailed setups and attacks For ResNet-18, we explored three split inference configurations: splitting early (after the first convolution layer), in the middle (after block 4), and late (after block 6). We evaluated the empirical privacy with a DNN attacker [45] and measured the reconstruction quality with structural similarity index measure (SSIM) [31]. Other popular attacks showed similar trends (see Appendix: Figure 9). NCF-MLP translates a user id (uid) and a movie id (mid) into embeddings
Figure 5: 1/ dFIL vs. SSIM (left) and sample reconstructions (right) for split inference with ResNet-18 and CIFAR-10. Higher SSIM indicates more successful reconstruction.
Figure 6: Attack accuracy vs.  $1 / \mathrm{dFIL}$  for split inference with NCF-MLP (left) and DistilBert (right).
with an embedding table and sends them through a DNN to make a prediction. We split the NCF-MLP model after the first linear layer of the MLP and tried reconstructing the original uid and mid from the encoding. This is done by first reconstructing the embeddings from the encoding using direct optimization  $(\hat{emb}(\mathbf{e}) = \underset{emb_0}{\arg \min} ||\mathbf{e} - \mathrm{Enc}(emb_0)||_2^2)$ , and finding the original uid and mid
by choosing the closest embedding value in the embedding table:  $id = \arg \min_{i}||\hat{emb} - Emb[i]||_2^2$ , where  $Emb[i]$  is the  $i$ -th entry of the embedding table. For DistilBert, we again explored three different splitting configurations: splitting early (right after block 0), in the middle (after block 2), and late (after block 4). We use a similar attack to NCF-NLP to retrieve each word token.
4.2.2 Evaluation results
Privacy Figure 5 shows the attack result for ResNet-18 and CIFAR-10. Setups with lower dFIL lead to lower SSIM (left) and less identifiable images (right), indicating that dFIL strongly correlates with the attack success rate. The figures also show that the privacy leakage estimated by dFIL can be conservative. Some setups show empirically-high privacy even when dFIL indicates otherwise, especially when splitting late. See Appendix: Figure 13 for more results. Figures 6 show the attack result for NCF-MLP (left) and DistilBert (right). Setups with lower dFIL again consistently showed a worse attack success rate. A sample reconstruction for DistilBert is shown in Appendix: Table 5.
Utility Table 1 summarizes the test accuracy of the split inference models, where  $1/\mathrm{dFIL}$  is chosen so that the attacker's reconstruction error is relatively high. For the same  $1/\mathrm{dFIL}$ , our proposed optimizations (Ours) improve the accuracy significantly compared to simply adding noise (No opt.). In general, reasonable accuracy can be achieved with encoders with relatively low dFIL. Accuracy degrades more when splitting earlier, indicating that using a DNN encoder with many layers can benefit utility. Prior works empirically observed a similar trend where splitting earlier degrades the privacy-utility tradeoff, because the encoding becomes leakier [49]. Our framework supports complex DNN encoders, which is a distinguishing benefit compared to prior theoretical works [16, 17, 44].
5 Case study 2: training with dFIL
5.1 Training on encoded data with dFIL
We consider a scenario where users publish their encoded private data, and a downstream model is trained on the encoded data. We use the first few layers of a pretrained model as the encoder by
Table 1: Test accuracy for different split inference setups with different dFIL. Base accuracy in parenthesis.
Setup	Split	1 dFIL	No opt.	Ours
CIFAR-10 + ResNet-18 (acc: 92.70%)	early	10	10.70%	74.44%
100	10.14%	57.97%
middle	10	22.11%	91.35%
100	12.94%	84.27%
late	10	78.48%	92.35%
100	33.54%	87.58%
CIFAR-100 + ResNet-18 (acc: 72.13%)	early	10	1.00%	44.10%
100	1.00%	30.69%
middle	10	1.98%	59.51%
100	1.29%	41.29%
late	10	8.63%	65.77%
100	2.01%	39.18%
MovieLens-20M + NCF-MLP (AUC: 0.8228)	early	1	0.8172	0.8286
10	0.7459	0.8251
100	0.6120	0.8081
GLUE-SST2 + DistilBert (acc: 91.04%)	early	10	50.80%	82.80%
100	49.08%	81.88%
middle	10	76.61%	83.03%
100	61.93%	82.22%
late	10	90.25%	83.03%
100	82.68%	82.82%
Table 2: Accuracy from training with different encoders.
Pretrain dataset	1 dFIL	Acc.
TinyImageNet	10	75.46%
100	42.57%
CIFAR-100	10	80.16%
100	70.27%
CIFAR-10 (held-out 20%)	10	81.99%
100	78.65%
freezing the weights and applying the necessary changes in Section 3.1. Then, we use the rest of the model with its last layer modified for the downstream task and finetune it with the encoded data. We found that similar optimizations from split inference benefit this use case as well.
5.2 Evaluation of dFIL-based training
We evaluate the model utility and show that it can reach a reasonable accuracy when trained on encoded data. We omit the privacy evaluation as it is similar to Section 4.2.2.
Evaluation setup We train a ResNet-18 model for CIFAR-10 classification. First, we train the model on one of three different datasets: (1) TinyImageNet [58], (2) CIFAR-100 [42], and (3) held-out  $20\%$  of CIFAR-10. Then, layers up to block 4 are frozen and used as the encoder. The CIFAR-10 training set is encoded using the encoder and used to finetune the rest of the model. The setup mimics a scenario where some publicly-available data whose distribution is similar (TinyImageNet, CIFAR-100) or the same (held-out CIFAR-10) with the target data is available and is used for encoder training. Detailed hyperparameters are in Appendix 7.2.
Evaluation result Table 2 summarizes the result. Our design was able to achieve a decent accuracy using encoded data with relatively safe  $1 / \mathrm{dFIL}$  values (10-100). The result indicates that model training with privately encoded data is possible. The achieved accuracy was higher when the encoder was trained with data whose distribution is more similar to the downstream task (CIFAR-10). We believe more studies in hyperparameter/architecture search will improve the result.
We showed that dFIL can theoretically quantify the invertibility of instance encoding by providing a reconstruction error bound against arbitrary attackers. We subsequently showed that dFIL can be used to guide private training/inference systems that uses instance encoding. dFIL has several potential limitations, which needs to be carefully considered before being used:
1. Corollary 1 only bounds the MSE, which might not always correlate well with the semantic quality. To address this, van Trees inequality can be extended to an absolutely continuous function  $\psi (\mathbf{x})$  to bound  $\mathbb{E}[||\psi (\hat{\mathbf{x}}) - \psi (\mathbf{x})||_2^2 /d]$  [19], which may be used to extend to metrics other than MSE.
2. Equation 5 provides an average case bound, so some data may experience lower reconstruction MSE than the bound. Similar average case bounds have also been used in prior works [13, 21]. When bounding invertibility, it is unclear how to define the privacy attack game to target worst-case reconstruction bound: for any sample, there exists a trivial attacker that always outputs that fixed sample from the distribution, which gives a worst-case MSE of zero for that sample. Leakier data that might experience lower reconstruction MSE than the bound can be potentially detected and handled by dynamically calculating dFIL for each sample. See Appendix: Figure 14.
3. For data types where MSE is not directly meaningful or the bound is inaccurate, it may not be straightforward to interpret the privacy of an encoding given its dFIL. In such cases, acceptable values of dFIL should be determined for each application through further research. The situation is similar to DP, where it is often not straightforward what privacy parameters (e.g.,  $\epsilon$ ,  $\delta$ ) need to be used [39].
4. Systems with the same dFIL may actually have different invertibility, as the bound from dFIL may be conservative. Comparing the privacy of two different systems using dFIL should be done with caution because dFIL is a lower bound rather than an accurate measure of invertibility.
We thank Yang Song for the invaluable discussion regarding the application of sliced score matching to our setup. We also thank Minjae Park and Jisu Kim for the discussion of our bound's implications. Kiwan Maeng was partially funded by the Charles K. Etner Early Career Professorship from Penn State University.
[1] Martin Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Edgar R. Weippl, Stefan Katzenbeisser, Christopher Kruegel, Andrew C. Myers, and Shai Halevi, editors, Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016, pages 308-318. ACM, 2016.
[2] Efe Aras, Kuan-Yun Lee, Ashwin Pananjady, and Thomas A. Courtade. A family of bayesian cramér-rao bounds, and consequences for log-concave priors. In IEEE International Symposium on Information Theory, ISIT 2019, Paris, France, July 7-12, 2019, pages 2699-2703. IEEE, 2019.
[3] Shahab Asoodeh, Mario Diaz, Fady Alajaji, and Tamás Linder. Estimation efficiency under privacy constraints. IEEE Trans. Inf. Theory, 65(3):1512-1534, 2019.
[4] Amin Banitalebi-Dehkordi, Naveen Vedula, Jian Pei, Fei Xia, Lanjun Wang, and Yong Zhang. Auto-split: a general framework of collaborative edge-cloud ai. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2543–2553, 2021.
[5] Martin Bertran, Natalia Martínez, Afroditi Papadaki, Qiang Qiu, Miguel R. D. Rodrigues, Galen Reeves, and Guillermo Sapiro. Adversarily learned representations for information obfuscation and inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 614-623. PMLR, 2019.
[6] Nicholas Carlini, Samuel Deng, Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoozy, Shuang Song, Abhradeep Thakurta, and Florian Tramér. An attack on instahide: Is private learning possible with instance encoding? CoRR, abs/2011.05315, 2020.
[7] Nicholas Carlini, Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoyo, and Florian Tramèr. Neuracrypt is not private. CoRR, abs/2108.07256, 2021.
[70] Liyao Xiang, Hao Zhang, Haotian Ma, Yifan Zhang, Jie Ren, and Quanshi Zhang. Interpretable complex-valued neural networks for privacy protection. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[71] Hanshen Xiao and Srinivas Devadas. Dauntless: Data augmentation and uniform transformation for learning with scalability and security. IACR Cryptol. ePrint Arch., page 201, 2021.
[72] Hanshen Xiao and Srinivas Devadas. PAC privacy: Automatic privacy measurement and control of data processing. In Helena Handschuh and Anna Lysyanskaya, editors, Advances in Cryptology - CRYPTO 2023 - 43rd Annual International Cryptology Conference, CRYPTO 2023, Santa Barbara, CA, USA, August 20-24, 2023, Proceedings, Part II, volume 14082 of Lecture Notes in Computer Science, pages 611-644. Springer, 2023.
[73] Adam Yala, Homa Esfahanizadeh, Rafael G. L. D'Oliveira, Ken R. Duffy, Manya Ghobadi, Tommi S. Jaakkola, Vinod Vaikuntanathan, Regina Barzilay, and Muriel Medard. Neurocrypt: Hiding private health data via random neural networks for public training. CoRR, abs/2106.02484, 2021.
[74] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
7.1 Score matching details
We found that using score matching [57] does not work reliably when the data's structure lies on a low-dimensional manifold (e.g., natural images). We found that applying randomized smoothing [9], which adds Gaussian noise to the image for robust training, helps stabilize score matching as it smoothens the density function. Randomized smoothing also makes the bound tighter. We observed that adding a reasonable amount of noise (e.g., standard deviation of 0.25, which was originally used by Cohen et al. [9]) works well in general, but adding only small noise (standard deviation of 0.01) does not. We show both results in Section 3.3.1.
7.2 Hyperparameters
Attacks For attacks in Section 3.3.1, 3.3.2, and 4.2, we used the following hyperparameters. For the optimizer-based attack for Gaussian synthetic input, we used Adam with  $\mathrm{lr} = 10^{-3}$ , and  $\lambda = 0.1 - 100$  for the regularizer. For the optimizer-based attack for NCF-MLP and DistilBert, we used Adam with  $\mathrm{lr} = 0.1$ . For the DNN-based attack for MNIST and CIFAR-10 (Figure 2, 3, 5), we used a modified DNN from Li et al. [45], which uses a series of convolution (Conv) and convolution transpose (ConvT) layers interspersed with leaky ReLU of slope 0.2. All the models were trained for 100 epochs using Adam with  $\mathrm{lr} = 10^{-3}$ . Below summarizes the architecture parameters. For DNN-based attacks in Section 3.3.2, we put a sigmoid at the end. For the attack in Section 4.2, we do not.
Table 3: DNN attacker architectures used in the paper. Output channel dimension  $(c_{out})$ , kernel size (k), stride (s), and output padding (op) are specified. Input padding was 1 for all layers.
Dataset + encoder	Architecture
MNIST + Conv	3×Conv(cout=16, k=3, s=1) + ConvT(cout=32, k=3, s=1, op=0) + ConvT(cout=1, k=3, s=1, op=0)
CIFAR-10 + split-early	3×Conv(cout=64, k=3, s=1) + ConvT(cout=128, k=3, s=1, op=0) + ConvT(cout=3, k=3, s=1, op=0)
CIFAR-10 + split-middle	3×Conv(cout=128, k=3, s=1) + ConvT(cout=128, k=3, s=2, op=1) + ConvT(cout=3, k=3, s=2, op=1)
CIFAR-10 + split-late	3×Conv(cout=256, k=3, s=1) + 2×ConvT(cout=256, k=3, s=2, op=1) + ConvT(cout=3, k=3, s=2, op=1)
Split inference Below are the hyperparameters for the models used in Section 4.2. For ResNet-18, we used an implementation tuned for CIFAR-10 dataset from [54], with ReLU replaced with GELU and max pooling replaced with average pooling. We used the default hyperparameters from the repository except for the following:  $\mathrm{bs} = 128$ ,  $\mathrm{lr} = 0.1$ , and weight Decay  $= 5 \times 10^{-4}$ . For NCF-MLP, we used an embedding dimension of 32 and MLP layers of output size [64, 32, 16, 1]. We trained NCF-MLP with Nesterov SGD with momentum  $= 0.9$ ,  $\mathrm{lr} = 0.1$ , and batch size of 128 for a single epoch. We assumed 5-star ratings as click and others as non-click. For DistilBert, we used Adam optimizer with a batch size of 16,  $\mathrm{lr} = 2 \times 10^{-5}$ ,  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , and  $\epsilon = 10^{-8}$ . We swept the compression layer channel dimension among 2, 4, 8, 16, and the SNR regularizer  $\lambda$  between  $10^{-3}$  and 100.
**Training** Below are the hyperparameters for the models evaluated in Section 5.2. We used the same model and hyperparameters with split inference for training the encoder with the pretraining dataset. Then, we freeze the layers up to block 4 and trained the rest for 10 epochs with CIFAR-10, with  $\mathrm{lr} = 10^{-3}$  and keeping other hyperparameters the same.
Execution Environment All the evaluation was done on a single A100 GPU. The training and evaluation of each model ranged roughly from less than an hour (ResNet-18 with split-early, NCF-MLP) to 3-7 hours (ResNet-18 with split-late, DistilBert).
7.3 van Trees inequality
Below, we restate the van Trees Inequality [19], which we use to prove Theorem 5.
Theorem 2 (Multivariate van Trees inequality). Let  $(\mathcal{X},\mathcal{F},P_{\theta}:\theta \in \Theta)$  be a family of distributions on a sample space  $\mathcal{X}$  dominated by  $\mu$ . Let  $p(\mathbf{x}|\theta)$  denote the density of  $X\sim P_{\theta}$  and  $\mathcal{I}_{\mathbf{x}}(\theta)$  denotes its FIM. Let  $\theta \in \Theta$  follows a probability distribution  $\pi$  with a density  $\lambda_{\pi}(\theta)$  with respect to Lebesgue measure. Suppose that  $\lambda_{\pi}$  and  $p(\mathbf{x}|\theta)$  are absolutely  $\mu$ -almost surely continuous and  $\lambda_{\pi}$  converges to 0 and the endpoints of  $\Theta$ . Let  $\psi$  be an absolutely continuous function of  $\theta$  with the same output dimension of  $\theta$ , and  $\psi_{n}$  an arbitrary estimator of  $\psi (\theta)$ . Assume regularity conditions from Section 2.2 is met. If we make  $n$  observations  $\{\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_n\}$ , then:
$$
\int_ {\Theta} \mathbb {E} _ {\theta} [ | | \psi_ {n} - \psi (\theta) | | _ {2} ^ {2} ] \lambda_ {\pi} (\theta) d \theta \geq \frac {(\int \mathrm {d i v} \psi (\theta) \lambda_ {\pi} (\theta) d \theta) ^ {2}}{n \int \mathrm {T r} (\mathcal {I} _ {\mathbf {x}} (\theta)) \lambda_ {\pi} (\theta) d \theta + \mathrm {T r} (\mathcal {J} (\lambda_ {\pi}))},
$$
where  $\operatorname{div}\psi (\theta) = \Sigma_i\frac{\partial\psi_i(\theta)}{\partial\theta_i}.$
7.4 Proof of Corollary 1
Proof. Let  $\psi$  be an identity transformation  $\psi(\theta) = \theta$ . For the setup in Corollary 1,  $n = 1$  and  $\mathrm{div}(\psi(\theta)) = d$ , so the multivariate van Trees inequality from Theorem 2 reduces to:
$$
\mathbb {E} _ {\boldsymbol \pi} \mathbb {E} _ {\boldsymbol \theta} [ | | \hat {\mathbf {x}} - \mathbf {x} | | _ {2} ^ {2} / d ] \geq \frac {d}{\mathbb {E} _ {\boldsymbol \pi} [ \mathrm {T r} (\mathcal {I} _ {\mathbf {e}} (\mathbf {x})) ] + \mathrm {T r} (\mathcal {J} (f _ {\boldsymbol \pi}))} = \frac {1}{\mathbb {E} _ {\boldsymbol \pi} [ \mathrm {d F I L} (\mathbf {x}) ] + \mathrm {T r} (\mathcal {J} (f _ {\boldsymbol \pi})) / d}.
$$
7.5 Comparison with differential privacy
Differential privacy [1] is not well-suited for instance encoding, as we discuss in Section 2.1. We formulate and compare a DP-based instance encoding and compare it with our dFIL-based instance encoding in a split inference setup (Section 4) to show that DP-based instance encoding indeed does not work well.