# Privacy Auditing in Differential Private Machine Learning:
## 4 Privacy Auditing Schemes
### Privacy Audits
#### 4.3.1 Differential Privacy Auditing Using Membership Inference
The auditing process utilizes a small hold-out dataset that has not been exposed during training. Their approach allows forthe use of an arbitrary hockey-stick divergence to measure the distance between the score distributions of audit training and test samples. This work fits well with FL scenarios. **White-box inference membership auditing**: White-box audits leverage full access to the internal parameters of a model, including gradients and weights. They are often used in corporate ML research, where the internal parameters of the model are available and allow a detailed analysis of the DP efficiency. [PERSON] and [PERSON] (2017) propose a calibrated white-box membership inference attack by evaluating the resulting privacy risk, which also leverages the intermediate representations of the target model. The work investigates how MIAs exploit the tendency of deep networks to memorize specific data points, leading to overfitting. They linearly approximated each layer, launched a separate attack on each layer, and trained the target model (DP-SGD) that combines the outputs of the layer-wise attacks. The high-precision calibration ensures that the attack can confidently identify whether a data point was part of the training set. [PERSON] et al. (2017) evaluate a differential private convolutional neural network (CNN) and Lasso regression model with and without sparsity using a MIA on high-dimensional training data, using genomic data as an example. They show that sparsity of the model in contrast to the non-private setting can improve the accuracy of the model in the non-private setting. By applying a regularization technique (e.g., Lasso), the study demonstrates that sparsity can complement DP efforts. There are seminal works that use both **white-box and black-box settings**: [PERSON] et al. (2017) extended the study of ([PERSON] et al., 2017) [PERSON] et al. (2020) on empirical privacy estimation techniques by analyzing DP-SGD through an increasing series of _black-box membership inference to white-box poisoning attacks_. They are the first to audit DP-SGD tightly. To do so, they use attacker-crafted datasets and active white-box attacks that insert canary gradients into the intermediate steps of DP-SGD. [PERSON] et al. (2019) propose a method for auditing backpropagation clipping algorithm (a modification of the DP-SGD algorithm), assuming that it works in black-box or white-box settings. The goal was to empirically evaluate how often the outputs of \(M(D)\) and \(M(D^{\prime})\) are distinguishable. The auditor's task with MIA is to maximize the FPR/TPR ratio to assess the strength of the privacy mechanism. [PERSON] et al. (2017) follow up on their earlier work ([PERSON] et al., 2021) and design an improved auditing scheme for testing DP implementations in black-box and white-box settings for DP-SGD with gradient canaries or input space canaries. This method provides a tight privacy estimation that significantly reduces the computational cost by leveraging tight composition theorems for DP. The authors check each individual step of the DP-SGD algorithm; that is, they do not convert each obtained lower bound obtained into a guarantee for \(\epsilon\), which is given after compiling over all training steps with understanding of the privacy understanding of the end-to-end algorithm. **Shadow model auditing**: Shadow model membership auditing is a technique used to assess the privacy of machine-learning models by replicating the target models that are trained on similar datasets. They allow the auditor to infer information about the target model's training data without direct access to it. In this method, multiple shadow models are created that mimic the behavior of the target model so that the auditor can infer the membership status based on the outputs of the shadow models. The primary purpose of using shadow models is to facilitate MIAs determining whether specific data points were part of the training dataset. The groundbreaking work [PERSON] et al.