# Privacy Auditing in Differential Private Machine Learning:
## 4 Privacy Auditing Schemes
### Privacy Audits
#### 4.3.1 Differential Privacy Auditing Using Membership Inference
The authors show that an effective defense against label-only MIAs involves DP and strong regularization, which significantly reduces the leakage of private information. **Single-run membership auditing:** It is a technique using a single execution of the audit process. [PERSON] et al. [43] propose a novel auditing scheme that uses only a single training of the model and can be evaluated using a one-time model output, making audits feasible in practical applications. The authors apply their auditing scheme specifically to the DP-SGD algorithm. After training, auditors select a set of _canary data points_ (auditing examples) and apply MIA thresholds and model parameter tuning to maximize audit assurance from a single model output. The attack estimates the sensitivity of the model to individual data, which is an indication of the privacy risk of the model. By adjusting the MIA parameters and interpreting the model's response to the canary points, the auditor approximates the empirical privacy loss, \(\epsilon_{emp}\). This empirical estimate provides information on how closely the model's practical privacy matches the theoretical DP guarantees. Their analysis uses _parallelism_ to add or remove multiple data points (training examples independently) in a single training run of the algorithm and _statistical generalization_. This auditing scheme requires minimal assumptions about the underlying algorithm, making it applicable in both black-box and white-box settings. [PERSON] et al. (2018) propose a novel one-shot auditing framework that enables efficient auditing during a single training run without a priori knowledge of the model architecture, tasks, or DP training algorithm. The method is proven to provide provably correct estimates for privacy loss under the Gaussian mechanism, demonstrating its performance on FL benchmark datasets. The method they propose is model and dataset agnostic, so it can be applied to any local DP task. [PERSON] et al. (2019) present a one-shot _nearly tight black-box auditing_ scheme for the privacy guarantees of the DP-SGD algorithm to investigate _empirical vs. theoretical aspects_. The main idea behind the auditing is to craft worst-case initial model parameters, since DP-SGD is agnostic to the choice of initial model parameters that can yield to tighter privacy audits. The model was initialized with the average-case initial parameters. The authors empirically estimate the privacy leakage from DP-SGD by using the gradient-based membership inference attacks approach. The authors' key finding is that by crafting worst-case initial model parameters, more realistic privacy estimates can be obtained to address the limitations of the theoretical privacy analysis of DP-SGD. **Loss-based membership inference auditing:** It is a technique that measures privacy leakage in differentially private models by evaluating the differences in the model's loss values generating during the training of ML models when predicting training data versus non-training data. [PERSON] et al. (2018) introduced a novel randomized approach to privacy accounting, which aims to improve on traditional deterministic methods by achieving tighter bounds on privacy loss. The method leverages the concept of Privacy Loss Distribution (PLD) to more accurately measure and track the cumulative privacy loss over a sequence of computations. This approach is particularly beneficial for large-scale data applications where the privacy budget is strict. **Confidence score membership auditing:** In this type of audit, the vulnerability of the model is assessed on the basis of the confidence scores of the predictions. Higher confidence in the predictions for training data points compared to non-training points often indicates a leak. By examining the confidence values for predictions in a large sample, auditors can determine whether training points have a higher confidence than non-training points and thus estimate membership leakage. [PERSON] et al.