Reimagining Mutual Information for Enhanced Defense against Data Leakage in Collaborative Inference

Authors: lin  duan
jingwei  sun
jinyuan  jia
yiran  chen
maria  gorlatova
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:42:13

Abstract: Edge-cloud collaborative inference empowers resource-limited IoT devices to support deep learning applications without disclosing their raw data to the cloud server, thus protecting user's data. Nevertheless, prior research has shown that collaborative inference still results in the exposure of input and predictions from edge devices. To defend against such data leakage in collaborative inference, we introduce InfoScissors, a defense strategy designed to reduce the mutual information between a model's intermediate outcomes and the device's input and predictions. We evaluate our defense on several datasets in the context of diverse attacks. Besides the empirical comparison, we provide a theoretical analysis of the inadequacies of recent defense strategies that also utilize mutual information, particularly focusing on those based on the Variational Information Bottleneck (VIB) approach. We illustrate the superiority of our method and offer a theoretical analysis of it.

Main paper: 
 introduction: 
 Edge devices are becoming smarter and more versatile. These devices are expected to efficiently perform a wide range of deep learning (DL) inference tasks with remarkable performance. However, implementing DL inference applications on such edge devices is challenging due to the constraints imposed by the on-device resource availability. As we see the rise of state-of-the-art (SOTA) DL models, such as Large Language Models [1, 2] , they are becoming increasingly complex, housing a colossal number of parameters. This escalation in complexity and size makes it difficult to store a DL model on an edge device, which typically has limited memory space. Furthermore, the restricted computational resources could lead to prolonged latency during inference. One potential solution to this predicament is to transmit the input data directly from the edge device to a cloud server. The server, which houses the DL model, then conducts inference and sends the prediction back to the device. However, this approach carries a risk of data leakage, particularly if the input data are sensitive in nature -such as facial images. In addition, the output data (i.e., predictions) can also contain confidential information, such as the patient's diagnostic results.Collaborative inference [3] [4] [5] [6] [7] [8] [9] has emerged as an approach to prevent data leakage when deploying DL inference applications on commodity edge devices with constrained computing resources. Fig. 1 shows a general collaborative inference system. Suppose an edge device and a cloud server conduct collaborative inference. The deep learning model can be divided into three parts 2 . The first and last few layers of the network are deployed on the edge device, while the remaining layers are offloaded to the cloud server. This division allows most of the computational tasks to be handled by the server, effectively mitigating the resource limitations on the device. The edge device and the cloud server communicate only the intermediate outputs of the model, ensuring that the raw input and predictions remain inaccessible to the server. However, recent works [10, 11] have revealed that sharing these intermediate outputs can still lead to data leakage from edge devices, including input data and predictions. A malicious server can, for instance, reconstruct input data from the representations (i.e., r in Fig. 1 ) uploaded by the device through Model Inversion (MI) attacks [12, 13, 11] . Furthermore, the high-level features (i.e., z in Fig. 1 ) contain rich information about the predictions, making it feasible for a malicious server to infer the device's predictions through these features [14] [15] [16] . While there have been considerable explorations into data protection in collaborative inference [10, 11, 17, 18] , existing defenses tend to significantly degrade model utility. This degradation is particularly evident in scenarios where attacks are relatively strong. For example, when the head model on the device (i.e., θ h in Fig. 1 ) is shallow, existing defenses [10, 11, 19, 20, 20, 18] cannot guarantee robustness against MI attacks without a significant drop in model accuracy as shown in our results.We propose InfoScissors, a defense method designed from a mutual information perspective to protect the edge device's data in collaborative inference. This approach works by protecting both the device's input data and its predictions. The goal of our method is to preserve user privacy in collaborative inference, and privacy preservation is achieved by manipulating the training phase (i.e., collaborative training). By applying our defense, the model is normalized to filter the private information when extracting features and representations, such that privacy is preserved during the inference phase. To protect the input data, we regularize the head model on the device to extract representations that contain less mutual information with the input. To protect the prediction, we regularize the features extracted by the server's encoder to minimize the mutual information they contain with the label. We derive a variational mutual information upper bound and develop an adversarial training method to minimize this bound on the device side. There are works [21, 17, 18] that prevent input leakage from the mutual information perspective, and most of them are based on Variational Information Bottleneck (VIB) [22] . Our work is not a simple replacement of the mutual information approximation compared with these works. We analyze the inadequacies of the VIB-based methods in protecting input data in the context of collaborative inference and illustrate the superiority of our method. We evaluate our method on CIFAR10 and CIFAR100 against input leakage using both black-box and white-box MI attacks. The results show that our method can effectively defend the attacks with less than a 3% drop in model accuracy even when the head model on the device has only one convolutional layer, where the attacks are extremely strong. We also evaluate our defense against prediction leakage using multiple Model Completion (MC) attacks [14, 15] . The results show that our defense achieves the best trade-off between the model accuracy and the defense effectiveness compared to the baselines. Our contributions are summarized as follows:• We propose InfoScissors, a defense method against data leakage in collaborative inference from the mutual information perspective, encompassing both input leakage and prediction leakage.• We offer a theoretical analysis of our defense against input recovery attacks and prediction inference attacks. We also analyze the superiority of our method compared with the VIB-based methods.• We empirically evaluate InfoScissors across multiple datasets and against multiple attacks. Our method effectively defends against MI and MC attacks, outperforming the baselines. 
related work: 
 Data Leakage in Collaborative Inference Data leakage is drawing more and more attention as the rapid growth of commercial deployment of DL, especially in collaborative learning scenarios, whose primary concern is data safety. In collaborative inference, we categorize data leakage into two types, i.e., input leakage [23, 10, 24, 25] and prediction leakage [14] [15] [16] . For input leakage, [23] proposes general attack methods for complex models, such as Neural Networks, by matching the correlation between adversary features and target features, which can be seen as a variant of model inversion [26, 27] . [10, 28, 25, 29, 30, 24] also propose variants of model inversion attack. While all these attacks are in the inference phase, [25] proposes a variant of DLG [12] , which can perform attacks in the training phase. For prediction leakage, [15] proposes an attack and defense method for two-party split learning on binary classification problems, a special collaborative inference setting. Additionally, [14] proposes three different label inference attack methods considering different settings in collaborative inference: direct label inference attack, passive label inference attack, and active label inference attack.Defense in Collaborative Inference Defensive methods have been proposed against data leakage in collaborative inference. To defend against input leakage, some works apply differential privacy (DP) [10, 11, 19, 31] and compression [10, 11, 17, 32] to the representations and models. While these methods can successfully defend against input leakage from the representations, they cause substantial model performance degradation because they weaken the knowledge/information in the representations. Some recent works also try to prevent input leakage by regularizing the representations from the mutual information perspective [17, 18, 21] . However, their methods only achieve decent results when the head model on the edge device is deep, which is not practical when the computation power is constrained on the edge device. Our paper analyzes that such a disadvantage comes from the inadequacies of VIB in the context of input protection. Some other works [33] [34] [35] apply mutual information on input space to protect input data, but their methods are only feasible with limited input dimension in the context of collaborative inference due to computational constrain on edge devices. One recent work [36] studies inference defense under a similar setting to our paper. But they only focus on prediction protection. They add noise to the training label by randomly sampling a class label, which is intuitively inspired by DP. Our training method is theoretically derived from the perspective of mutual information, and we provide a theoretical analysis of our defense performance. To defend against prediction leakage, [16] manipulates the labels following specific rules to defend the direct label inference attack, which can be seen as a variant of label differential privacy (label DP) [37, 38] in collaborative inference. Compression and quantization of the gradients [14, 18] are also applied to prevent prediction leakage. However, similar to the defense against input leakage, these defenses cause substantial model performance degradation to achieve decent defense performance.compromise the inference process conducted by the device. Nevertheless, the adversary (i.e., malicious server) is capable of training a surrogate classifier and generator to mimic the victim's data. With the received representation r i , the server can reconstruct the input x i on the device by conducting MI attacks [12, 13, 10] . Notably, the head model on the device is usually shallow due to the computation resource limitation, which aggravates the input leakage from the representation [11] . The encoder on the server extracts high-level features containing rich information about the prediction, which enables the server to infer predictions of the device. We conduct preliminary experiments to illustrate the data leakage in collaborative inference, which can be found in Appendix A. 
defense formulation: 
 To defend against data leakage, we propose InfoScissors, a learning algorithm that regularizes the model during the training phase. Following the setup of 3.1, suppose the edge device has sample pairs {(x i ,y i )} N i=1 drawn from a distribution p(x,y). The representation is calculated as r = f h θ h (x) by the edge device, and the cloud server computes features z = f e θ e (r). We apply x,y,r,z here to represent random variables, while x i ,y i ,r i ,z i are deterministic values. To defend against the leakage of the edge device's input data and inference results, InfoScissors is designed to achieve three goals:• Goal 1: To preserve the performance of collaborative inference, the main objective loss should be minimized. • Goal 2: To prevent the input leakage from the representations, θ h should not extract representations r containing much information about the input data x. • Goal 3: To reduce the leakage of the predictions on the edge device, θ e on the cloud server should not be able to extract features z containing much information about the true label y. where I(r;x) is the mutual information between the representation and the input, which indicates how much information r retains about the input data x. Similarly, I(z;y) is the mutual information between the feature and the label. We minimize these mutual information terms to prevent the cloud server from inferring the input x and label y from r and z, respectively.The prediction objective is usually easy to optimize (e.g., cross-entropy loss for classification). However, the mutual information terms are hard to calculate in practice for two reasons: 1. r and x are high-dimensional, and it is extremely computationally heavy to compute their joint distribution; 2.Calculating the mutual information requires knowing the distributions p(x|r) and p(y|z), which are both difficult to compute. We do not follow existing works [18, 21, 17] to employ VIB to derive tractable estimations of the mutual information objectives. We analyze the inadequacies of VIB in protecting input, which can be found in Sec. 4.4, and leverage CLUB [39] to formulate variational upper bounds of mutual information terms. We first formulate a variational upper bound of I(r;x):EQUATIONwhere q ψ (x|r) is a variational distribution with parameters ψ to approximate p(x|r). To guarantee the inequality of Eq. (3), q ψ (x|r) should satisfy:EQUATIONwhich can be achieved by minimizing KL(p(r,x)||q ψ (r,x)):EQUATIONWith sample pairs {(x i ,y i )} N i=1 , we apply the sampled vCLUB (vCLUB-S) mutual information estimator in [39] to reduce the computational overhead, which is an unbiased estimator of I vCLUB and is formulated as:EQUATIONwhere k ′ i is uniformly sampled from indices {1,...,N }. With Eq. ( 3), Eq. ( 5) and Eq. ( 6), the objective of input protection is formulated as:EQUATIONSimilarly, we can use a variational distribution q ϕ (y|z) with parameter ϕ to approximate p(y|z), and formulate the objective of label protection as:min θ h ,θ e I(z;y) ⇔ min θ h ,θ eÎvCLUB-S(z;y) = minEQUATIONSuppose we use g ψ , h ϕ to parameterize q ψ and q ϕ , respectively. By combining Eq. ( 7), Eq. ( 8) and the prediction objective with weight hyper-parameters λ d and λ l , the overall optimizing objective is:EQUATIONh ϕ can be easily constructed to estimate p (y|z) given the task of inference (e.g., classifier for classification task). To estimate p(x|r), we assume that x follows the Gaussian distribution of which the mean vector is determined by r and the variance is 1. Under this assumption, we apply a generator g ψ to estimate the mean vector of x given r. 𝑒𝑒 𝜃𝜃 𝑐𝑐 𝑟𝑟 𝑥𝑥 𝜃𝜃 ℎ Cloud Server Edge device 𝜓𝜓 𝜙𝜙 Goal 2: Input protection ℒ 𝑑𝑑_𝑎𝑎 = log 𝑔𝑔 𝜓𝜓 𝑥𝑥 𝑟𝑟 ℒ 𝑑𝑑_𝑟𝑟 = -log 𝑔𝑔 𝜓𝜓 𝑥𝑥 𝑘𝑘 ′ 𝑟𝑟 Goal 3: Prediction protection ℒ 𝑙𝑙_𝑎𝑎 = log ℎ 𝜙𝜙 𝑦𝑦 𝑧𝑧 ℒ 𝑙𝑙_𝑟𝑟 = -log ℎ 𝜙𝜙 𝑦𝑦 𝑛𝑛 ′ 𝑧𝑧 Goal 1: Prediction ℒ 𝑐𝑐 = ℒ(𝑓𝑓 𝜃𝜃 𝑐𝑐 𝑐𝑐 (𝑧𝑧), 𝑦𝑦)The overall objective has five terms. For simplicity, we denote these five objective terms as L c , L d_a , L d_r , L l_a and L l_r , respectively, as shown in Eq. ( 9). L c is the prediction objective. L d_a and L d_r comprise the input data protection objective. L d_a is an adversarial training objective where an auxiliary generator g ψ is trained to capture input information while the head layers f h θ h are trained to extract as little input information as possible. L d_r regularizes f h θ h to extract representations that can be used to generate randomly picked samples. L l_a and L l_r have similar effect with L d_a and L d_r , respectively. We can reorganize the overall training objective as:EQUATIONBased on Eq. ( 10), we develop a collaborative learning algorithm. For each batch of data, the device first optimizes the classifiers θ c and ϕ by minimizing L c and maximizing L l_a , respectively. Then, the device optimizes the generator ψ by maximizing L d_a . Finally, θ h and θ e are optimized by minimizing(1 -λ d -λ l )L c + λ l L l_a + λ l L l_r + λ d L d_a + λ d L d_r .The detailed algorithm can be found in Appendix B. Note that θ h ,θ c ,ψ, and ϕ are deployed on devices, and their training does not need additional information from the cloud server compared with training without our defense. The training procedure of θ e does not change, which makes our defense concealed from the cloud server.We provide a theoretical analysis of our defenses against prediction and input leakage. Following the notations in Sec. 4.1, we have the following theorem of defense performance for prediction leakage after applying InfoScissors. All the proofs can be found in Appendix C. Theorem 1. Let h ϕ parameterize q ϕ in Eq. (8) . Suppose the malicious server optimizes an auxiliary model h m (y|z) to estimate p(y|z). For any h m (y|z), we always have:EQUATIONwhereEQUATIONSpecifically, if the task of collaborative inference is classification, we have the following corollary: Corollary 1. Suppose the task of collaborative inference is classification. Following the notations in Theorem 1 and let epsilon be defined therein, we have:EQUATIONwhere CE denotes the cross-entropy loss, and CE random is the cross-entropy loss of random guessing.For input leakage, we have the following theorem. Theorem 2. Let the assumption of p(x|r) in Sec. 4.1 hold and g ψ parameterize the mean of q ψ in Eq. (7) . Q denotes the dimension of x. Suppose the malicious server optimizes an auxiliary model g m (x|r) to estimate the mean of p(x|r). For any g m (x|r), we always have:EQUATION)where MSE denotes the mean square error, andEQUATIONRecent works also try to prevent input leakage by regularizing the representations from the mutual information perspective [21, 17, 18] . The ultimate goal of these works is the same as our method, which is to minimize I(r;x). However, most of these works apply Variational Information Bottleneck (VIB) [22] to derive the upper bound of I(r;x). Even though VIB is commonly applied in DNN feature regularization, we analyze that it is not optimal in defending against input leakage in collaborative inference.The variational upper bound of I(r;x), which is also an objective to minimize, derived through VIB is formulated asEQUATIONwhere ϵ is the Gaussian random variable used to reparameterize the randomness of representation extractor θ h . r(r) is the variational approximation of the marginal distribution p(r). In practice, without any prior information, r(r) is set to be a fixed spherical Gaussian, r(r) = N (r|0, I). To minimize I(r;x), the VIB-based works [21, 17, 18] regularize the representation distribution to be close to a fixed Gaussian. This will cause the complementary loss of information contained by r, including the mutual information with y, which is crucial for the primary inference task. In contrast, the training objective derived in our method is to only filter out the information in r that is crucial for reconstructing x, which will cause less information loss and inference performance drop.We can also analyze the difference between VIB-based methods and our method from a high level, which also motivates us to choose CLUB approximation. The mutual information I(r;x) can be expanded asEQUATIONVIB follows the first equality to derive a tractable upper bound of I(r;x) based on a parameterized variational p(r|x). Subsequently, the training objective is focused on minimizing this variational p(r|x). We follow the second equality to parameterize p(x|r) and derive an upper bound based on CLUB. Our training goal is centered on minimizing the variational p(x|r). Importantly, in the context of defending against model inversion attacks, the primary aim is to attain a low p(x|r). This aim is more closely aligned with our training objective compared to VIB-based methods. This alignment, alongside the empirical results presented in the subsequent section, highlights the superiority of our method.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Please check Sec. 5. 
experiments: 
 We first evaluate our method against input leakage and prediction leakage, both separately and in an integrated manner. The experiments are conducted on a server with 4 RTX TITAN GPUs. Attack methods For input leakage, we evaluate InfoScissors against two Model Inversion (MI) attacks: (1) Knowledge Alignment (KA) [17] is a black-box MI attack, in which the malicious server trains an inversion model that swaps the input and output of the target model using an auxiliary dataset. The inversion model is then used to reconstruct the input given any representation. (2) Regularized Maximum Likelihood Estimation (rMLE) [11] is a white-box MI attack that the malicious server has access to the device's extractor model θ h . The server trains input to minimize the distance between the fake representations and the received ground-truth representations. It is an unrealistic assumption that the server can access the model on the device, and we apply this white-box attack to evaluate our defense against extremely strong attacks. For prediction leakage, we evaluate our defense against two Model Completion (MC) attacks: (1) Passive Model Completion (PMC) [14] attack assumes that the malicious server has access to an auxiliary labeled dataset and utilizes this auxiliary dataset to fine-tune a classifier that can be applied to its encoder. (2) Active Model Completion (AMC) [14] attack is included as an adaptive attack against our defense. The primary goal of our defense is to reduce the collaborative model's reliance on the server-side encoder for specific tasks, allowing data and prediction information to be filtered during inference. Under the adaptive attack setting, the adversary is allowed to modify the training profiling such that it can trick the collaborative model into relying more on its encoder, thereby extracting more private data information from the encoder's features. In this setting, the adversary is directly confronting the fundamental principles of our defense method, constituting a highly potent form of adaptive attack.Baselines We compare InfoScissors with five existing defense baselines: (1) Differential Privacy (DP) [10, 11, 19] , (2) Adding Noise (AN) [40] , (3) Data Compression (DC) [20] , (4) Privacy-preserving Deep Learning (PPDL) [41] , and (5) Mutual Information Regularization Defense (MID) [18] , which is the SOTA defense against data leakage in collaborative inference based on Variational Information Bottleneck (VIB). The details of the baselines can be found in Appendix D.Dataset & Hyperparameter configurations We evaluate on CIFAR10 and CIFAR100. For both datasets, we apply ResNet18 as the backbone model. The first convolutional layer and the last basic block are deployed on the device as the representation extractor and the classifier, respectively. We set batch size B as 32 for both datasets. We apply SGD as the optimizer with the learning rate η set to be 0.01. The server has 40 and 400 labeled samples to conduct KA and MC attacks for CIFAR10 and CIFAR100, respectively. For InfoScissors, we apply a 1-layer decoder and a 3-layer MLP to parameterize ψ and ϕ. For AN, we apply Laplacian noise with a mean of zero and a scale between 0.0001-0.01. For DC, we set the compression rate from 90% to 100%. For PPDL, we set the Laplacian noise with a scale of 0.0001-0.01, τ = 0.001 and θ between 0 and 0.01. For MID, we set the weight of mutual information regularization between 0-0.1.Evaluation metrics (1) Utility metric (Model accuracy): We use the test data accuracy of the classifier on the device to measure the performance of the collaborative model. (2) Robustness metric (SSIM): We use SSIM (structural similarity) between the reconstructed images and the raw images to evaluate the effectiveness of the defense against input leakage. The lower the SSIM, the better the defense performance. (3) Robustness metric (Attack accuracy): We use the test accuracy of the server's classifier after conducting MC attacks to evaluate the defense against prediction leakage. The lower the attack accuracy, the higher the robustness against prediction leakage.We conduct experiments on CIFAR10 and CIFAR100 to evaluate our defense against the KA attack and the rMLE attack. We set different defense levels for our methods (i.e., different λ d values in Eq. ( 9)) and baselines to conduct multiple experiments to show the trade-off between the model accuracy and SSIM of reconstruction. The results are shown in Fig. 4 . For defense against KA attack, our InfoScissors can reduce the SSIM of reconstruction to lower than 0.2 with a model accuracy drop of less than 2% for CIFAR10. In contrast, the other baselines reduce model accuracy by more than 10% and cannot achieve the same defense effect even with an accuracy drop of more than 10%. Notably, the malicious server has more auxiliary data on CIFAR100 than CIFAR10, making the defense harder on CIFAR100. However, InfoScissors can still achieve an SSIM of lower than 0.2 with a model accuracy drop of less than 2%. We also evaluate our defense against the KA attack with a larger auxiliary dataset on the malicious server, and the results, which can be found in Appendix D, show that our defense can effectively defend against the KA attack when the server has more auxiliary samples. For defense against rMLE attacks, InfoScissors achieves similar results of reducing the SSIM to lower than 0.2 with a model accuracy drop of less than 2% for CIFAR10 and 1% for CIFAR100, respectively, which outperforms the other baselines significantly.To perceptually demonstrate the effectiveness of our defense, we show the reconstructed images by the KA attack on CIFAR10 after applying baseline defenses and our defense in Fig. 3 . It is shown that by applying the baseline defenses, the reconstructed images still contain enough information to be recognizable with the model accuracy of lower than 70%. For our method, the reconstructed images do not contain much information about the raw images, with the model accuracy higher than 76%.We evaluate InfoScissors on two datasets against the PMC attack and the AMC attack. We set different defense levels for our methods (i.e., different λ l values in Eq. ( 9)) and baselines to conduct multiple experiments to show the trade-off between the model accuracy and attack accuracy. The defense results against PMC and AMC attacks are shown in Fig. 5 and Fig. 6 , respectively. To simulate the realistic settings where the malicious server uses different model architectures to conduct MC attacks, we apply different model architectures (MLP & MLP_sim) for MC attacks. The detailed model architectures can be found in Appendix D. For defense against PMC on CIFAR10, InfoScissors achieves 10% attack accuracy (equal to random guess) by sacrificing less than 0.5% model accuracy, while the other baselines suffer a model accuracy drop by more than 4% to achieve the same defense effect. Similarly, InfoScissors achieves 1% attack accuracy on CIFAR100 by sacrificing less than 1% model accuracy, while the other baselines achieve the same defense effect by sacrificing more than 6% model accuracy.InfoScissors also shows robustness against AMC attack. InfoScissors achieves attack accuracy of the rate of random guess by sacrificing less than 1% and 0.5% model accuracy on CIFAR10 and CIFAR100, respectively. The other baselines achieve the same defense performance by sacrificing more than 5% and 4% model accuracy, respectively.We evaluate the integration of input and prediction protection of InfoScissors. We set λ d and λ l between 0.05-0.4 and evaluate the defenses. The results of defense against the KA and PMC attacks on CIFAR10 and CIFAR100 are shown in Fig. 7 . It is shown that InfoScissors can effectively protect input data and predictions simultaneously with less than a 2% accuracy drop for both datasets. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion and limitation: 
 We propose a defense method (InfoScissors) to defend against data leakage in collaborative inference by reducing the mutual information between the model's intermediate outcomes and the device's input data and predictions. The experimental results show that our method can defend against input leakage and prediction leakage effectively. Our work can make the public aware of the risk of privacy leakage posed by collaborative inference, which is considered a positive societal impact. One limitation of this paper is that we only focus on the scenario where there is only one edge device, even though our defense can be easily applied to the collaborative inference scenario with multiple edge devices. With the received representation r i , the server can reconstruct the input data x i on the edge device by conducting model inversion (MI) attacks [17] . 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not release a model or scraped datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the corresponding papers when using the pre-trained models and existing datasets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
 
 Appendix: Notably, the head model on the edge device is usually shallow due to the computation resource limitation, which aggravates input leakage from the representation [11] . We conduct experiments on CIFAR10 with ResNet18 to demonstrate the input leakage problem. One convolutional layer is deployed on the device as the head model, and one basic block is deployed as the classifier. The malicious server conducts Knowledge Alignment (KA) attack [17] to recover the input image from the received representation through a generator. The detailed experimental settings can be found in Sec. 5. The reconstructed images are shown in Fig. 8 . The high-quality reconstructed images illustrate the collaborative inference's vulnerability to the device's input leakage from the representation.The training process enables the cloud server to extract high-level features useful for the collaborative inference task. These high-level features allow the malicious server to fine-tune a classifier head with very few labeled data and accurately conduct inference. The leakage of the prediction makes the edge device holder's privacy, including behavior and preference, exposed to the cloud server. For example, prediction leakage of a collaborative inference-based navigation mobile app allows the cloud server to infer the positions and destinations of the app users. To demonstrate the extent of prediction leakage by the features, we follow the experimental setup in Appendix A.1 and let the cloud server conduct model completion (MC) attack [14] to train a classifier using a small number of auxiliary labeled samples. We also let the cloud server train an entire model with the auxiliary dataset from scratch for comparison.The results are shown in Tab. 1. It is shown that by fine-tuning a classifier with the collaboratively trained encoder, the cloud server can achieve an accuracy of nearly 70% using an auxiliary dataset with only 40 labeled samples. However, training from scratch cannot achieve decent accuracy using the same auxiliary dataset, which shows that the high-level features extracted by the encoder on the server cause prediction leakage.Algorithm 1 Training algorithm of InfoScissors. ← means information is sent to the server; ← means information is sent to the device; red steps are conducted on the cloud server.Input: Dataset {(xi,yi)} N i=1 ; Learning rate η. Output: θ h ;θ e ;θ c ;ψ;ϕ.1: Initialize θ h ,θ e ,θ c ,ψ,ϕ; 2: for a batch of data {(xi,yi)} i∈B do 3:{ri} i∈B ←{f h θ h (xi)} i∈B ; 4: L d_a ← 1 |B| i∈B logg ψ (xi|ri); 5: ψ ← ψ+η∇ ψ L d_a ; 6:{zi} i∈B ←{f e θ e (ri)} i∈B ; 7: Lc ← 1 |B| i∈B L(f c θ c (zi),yi); 8: L l_a ← 1 |B| i∈B logh ϕ (yi|zi); 9: θ c ← θ c -η∇ θ c Lc; 10: ϕ ← ϕ+η∇ ϕ L l_a ; 11: {y n ′ i } i∈B ← randomly sample {y n ′ i } i∈B from {yi} i∈[N ] ; 12: {x k ′ i } i∈B ← randomly sample {x k ′ i } i∈B from {xi} i∈[N ] ; 13: L d_r ← 1 |B| i∈B -logg ψ x k ′ i |r 2 i ; 14: L l_r ← 1 |B| i∈B -logh ϕ y n ′ i |z 2 i ; 15: {∇z i L} i∈B ←{∇z i [(1-λ d -λ l )Lc +λ l L l_a +λ l L l_r ; +λ d L d_a +λ d L d_r ]} i∈B ; 16: ∇ θ e L ←We compare InfoScissors with five existing defense baselines: (1) Differential Privacy (DP) [10, 11, 19] protects the data with a theoretical guarantee by clipping the representation and gradients norm and injecting perturbations to the representations and gradients. (2) Adding Noise (AN) [40] is proven effective against data leakage in collaborative learning by adding Laplacian noise to the representations and gradients. (3) Data Compression (DC) [20] prunes representations and gradients that are below a threshold magnitude, such that only a part of the representations and gradients are sent to the server. (4) Privacy-preserving Deep Learning (PPDL) [41] is a comprehensive privacy-enhancing method including three defense strategies: differential privacy, data compression, and random selection. ( 5) Mutual Information Regularization Defense (MID) [18] is the SOTA defense against data leakage in split learning and collaborative inference. MID is also based on mutual information regularization by applying Variational Information Bottleneck (VIB).The malicious server uses models with different architectures (MLP and MLP_sim) to conduct MC attacks. MLP_sim has one FC layer. MLP has three FC layers with a hidden layer of size 512×256.Besides the experiments in Sec. 5, we also evaluate our defense against the KA attack with a larger auxiliary dataset on the malicious server. The server has 80 and 800 labeled samples to conduct KA and MC attacks for CIFAR10 and CIFAR100, respectively, and the results are shown in Fig. 9 . • The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Note that some applications might divide the model into two parts, and the edge devices might hold the first or the last few layers, which have different data leakage problems. This paper considers the general setting. 


Decision: **Summary**

This paper introduces InfoScissors, a defense method to reduce data leakage in collaborative inference by minimizing mutual information between a model’s intermediate states and the edge device's input/output. The authors propose using CLUB, a mutual information approximation, instead of the more commonly used VIB, providing both theoretical analysis and empirical evidence across various datasets.

**Decision**

I recommend accepting this paper. The approach addresses an important privacy issue and offers a novel defense mechanism that shows improved performance in preserving privacy while maintaining model accuracy. The authors have responded well to reviewer concerns, particularly by expanding their evaluations to larger datasets (e.g., mini-ImageNet) and different model architectures (ViT). Although more real-world dataset evaluations would strengthen the paper, the theoretical contributions and empirical results support the paper’s significance and relevance to the field.

For the final version, I encourage the authors to enhance clarity on adaptive attacks and ensure that additional experiments are included.

Reviews: review: paper_summary: paper_summary: This paper provides InfoScissors, a learning algorithm that regularizes the model during the training phase. This paper also compares their method with VIB-based methods and evaluates it with multiple attacks.
strength_weakness: strength_weakness: The author claims that LLMs cannot be handled on edge devices; however, in the evaluation part, they only use CIFAR10 and CIFAR100, both of which can be handled by edge devices. The paper does not measure the method with real large-size datasets.
questions: questions: No
limitations: limitations: The paper does not measure the method with real large-size datasets.
paper_summary: paper_summary: This paper considers a setting where two parties (Cloud Server and Edge Device) are collaboratively training a deep model. The threat model considers both inputs and outputs to be private data of the Edge Device that should be protected from the Cloud Server. The authors propose a learning algorithm that is theoretically motivated by a recent method called CLUB [38], and performs better than other baselines on two datasets CIFAR10 and CIFAR100.  The main contribution of this paper is that the authors show CLUB offers better optimization results compared to Variational Information Bottleneck (VIB).
strength_weakness: strength_weakness: The main weaknesses are motivation of the problem and the evaluations of the algorithm. The paper needs some improvements in this regard. Please see below.
questions: questions: (1) One important, but totally ignored, aspect of the proposed framework (Figure 1) is that the Cloud Server can run the classifier part (that is \theta^c) and easily discover y without needing any sophisticated attack. Unless, the authors argue that the classifier part is not available to Cloud Server! Which is a strong and unrealistic assumption because usually the server is the entity that trains the model and then splits it among the Edge devices.  I am struggling to find a real-world application in which the Edge Device has access to the classifier part but the Cloud Server does not have that access. Authors need to clarify and motivate this.

(2) When reading the paper, it is not clear to distinguish between training vs. inference stage. The title mentions “Collaborative Inference”, but the paper is actually more related to “Collaborative Training”. The fact that two parties (Cloud Server and Edge Device) are collaborating in training a model or they are collaborating in making a prediction (via a trained model in inference mode) makes a huge difference in terms of threat model and evaluation. The authors need to revise the text and make this as clear as possible from the beginning of the paper. It is also suggested to discuss relevance to a work published last year https://arxiv.org/abs/2310.13384 (Salted Inference: Enhancing Privacy while Maintaining Efficiency of Split Inference in Mobile Computing)

(3) The authors should compare and clarify the contribution of this paper compared to Club [38]. They should explain to what extent the theoretical analysis of this paper is novel compared to what has been already presented in [38].

(4) The paper should show the performance of the work on more complex data types. The two chosen datasets are of similar complexity. Similarly, beside ResNet 18, other model architectures should be examined. It is not clear if this method only works for ConvNets or whether it supports other types of layers.

(5) In Figure 3, it is not clear what is the difference between different rows. For each method there are three pairs of images and results, but how different are these and in what sense?
limitations: limitations: The main limitation is that the generalization of the method to other complex datasets and model architectures is not explored or discussed.
paper_summary: paper_summary: The collaborative inference enables resource-constrained IoT devices to support deep learning applications without sharing raw data with the cloud server, but previous research has revealed that this approach still leaks input and prediction information from edge devices. To address this vulnerability, the authors propose InfoScissors, a defense strategy designed to minimize the mutual information between a model's inner states and the device's input and outputs. The effectiveness of InfoScissors is evaluated on common datasets like CIFAR10/100 against various attacks, demonstrating its superiority over existing defense strategies that utilize mutual information. The paper also provides theoretical analysis to support the proposed method.
strength_weakness: strength_weakness: - The novelty of proposed method is somehow limited. The proposed defense InfoScissors relies on the Mutual Information (MI) regularization that has been explored in previous study like MID. The paper directly adopts a new formulation [38] for the upper bound of MI terms. As a result, the novelty and contribution of this paper appear to be limited. Compared to previous VIB-based approaches, even though the authors manage to find out several advantages  and provide a theoretical explanation, the challenges are not clear.


- Limited evaluation of adaptive attack, making the robustness of proposed defense InfoScissors doubtful. The only mention of adaptive attack is the AMC attack, but there is not many details about the adaptive attack. A strong adversary can follow the proposed method to train surrogates classifier and generator to mimic the victim's data. I suggest the authors to clarify the adversary capacity and thoroughly evaluate the potential privacy breach after the method is known by the adversary.

- Limited evaluation of benchmarks. The paper only evaluates on the simple benchmark like CIFAR10/100 but not more large-scale dataset like ImageNet. The model architecture evaluation is also important to demonstrate the scalability of proposed defense. Also, it seems that the method is only evaluated on the vision classification models, is it applicable to generative model or other tasks? If so, what is the cost of efficiency and accuracy?
questions: questions: See comments above
limitations: limitations: None

Comments: title: Follow-up on Rebuttal
Follow-up on Rebuttal
Response by Authors
Discussion period starts
comment: I hope our rebuttal addressed your concerns effectively. As we approach the end of the discussion period, we wanted to check if you have any remaining questions or points you'd like to clarify. Your insights are very important to us, and we would appreciate any further feedback you can provide before the discussion period ends.
May I kindly inquire if you have any further concerns or questions after reviewing our rebuttal? As the discussion period is nearing its conclusion, your feedback is incredibly valuable to us. I would greatly appreciate it if you could let us know if there are any additional points you'd like to discuss or if our rebuttal satisfactorily addresses your concerns.
Thanks for your response! We will follow your comments and revise the draft accordingly.
Dear reviewers,

Thank you for your valuable contributions to the NeurIPS review process! The author-reviewer discussion period has now begun. I’ve noticed that the ratings for this paper are dispersed, and opinions among the reviewers are not fully aligned. This makes our discussion even more crucial to reach a consensus and ensure a fair and thorough evaluation. Please engage actively with the authors during this period. If you have any questions or need further clarification on any points, this is the best time to address them directly with the authors.

best,

AC
I’m happy with your answers and would strongly recommend enhancing the presentation of your paper. Including results from additional datasets and different model architectures would be an excellent idea. In particular, including datasets from real-world applications, such as wearables or medical images (as mentioned in your answer on motivation), could show the generalizability of your methodology and its relevance to a wider audience. Overall, I’m satisfied with the work and look forward to seeing these improvements.