# Privacy Auditing in Differential Private Machine Learning:
## 4 Privacy Auditing Schemes
### Privacy Audits
#### 4.3.1 Differential Privacy Auditing Using Membership Inference  
**Membership inference audits:** These audits test the resilience of the model against membership attacks, where an attacker tries to determine whether certain data points were included in the training set. The auditor performs MIAs to estimate how much information about individual records may have been leaked. This category is divided into the following subcategories:  
* **Black-box inference membership auditing:** This approach relies solely on assessing the privacy guarantees of machine-learning models by evaluating their vulnerability to membership inference attacks (MIAs) without accessing the internal workings of the model. [PERSON] et al. [175] examine how robust training, including a differential privacy mechanism, affects the vulnerability to black-box MIAs. The success of MIAs is measured using metrics such as attack accuracy and the relationship between model overfitting and privacy leakage. It also investigates MIAs under attacker robustness and differential privacy conditions and shows that DP models are also vulnerable under black-box conditions. [PERSON] et al. [103] present a DP audit method related to black-box threshold MIAs by proposing a first-principles approach. The authors introduce the likelihood ratio attack (LiRA). It analyzes the most vulnerable points in the model predictions. The authors question the use of existing methodologies that rely on average-case accuracy metrics to evaluate empirical privacy, which do not adequately capture an attacker's ability to identify the actual members of the training dataset. They propose to measure the attacker's ability to infer membership of a dataset using the true-positive rate (TPR) and the low false-positive rate (FPR) at very low positive rates (e.g., \(<\)0.1%). The authors offer a \(\epsilon\)--maximization strategy. The authors conclude that even a powerful DP mechanism can sometimes be vulnerable to carefully constructed black-box accesses. [PERSON] et al. [172] present Eureka, a novel method for estimating relative DP guarantees in black-box settings, which defines a mechanism's privacy concerning a specific input set. At its core, Eureka uses a hypothesis testing technique to empirically estimate privacy loss parameters. By computing outputs on adjacent datasets, the potential leakage and thus the degree of privacy guarantee is determined. The authors use MIAs based on classifiers to audit \((\epsilon,\delta)\)-DP algorithms. They demonstrate that Eureka achieves tight accuracy bounds in estimating privacy parameters with relatively low computational cost for large output space. [PERSON] et al. [174] present a black-box privacy auditing method for ML target models based on a MIA using both training data (i.e., \"members\" (true positives)) and generated data not included in the training dataset (i.e., non-members (true negatives)). This method leverages membership inference as primary method to audit datasets used in the training of ML models without retraining them (ensembled membership auditing (EMA)). EMA aggregates the membership scores on a data-by-data basis based on individual data, using statistical tests. The method, which authors call PANORAMIA, quantifies privacy leakage for large-scale ML models without controlling the training process or retraining of the model. [PERSON] et al. [176] use the total variation distance (TVD), a statistical measure that quantifies the difference between two probability distributions, between the output distributions of a model when trained on two neighboring datasets. The authors suggest that TV distance can serve as a robust indicator of privacy guarantee when examining the outputs of a DP mechanism. The auditing process compares two TVD how much they differ from each other across different outputs generated from adjacent datasets to approximate the privacy parameters. It is directly related to the privacy parameter, \(\epsilon\), and provides a tangible way to evaluate privacy loss.