# Privacy Auditing in Differential Private Machine Learning:
## 4 Privacy Auditing Schemes
### Privacy Audits
#### 4.3.1 Differential Privacy Auditing Using Membership Inference
Given the score of the victim model on \(x\), the attack then applies a likelihood ratio test to distinguish between the two hypotheses. To estimate the score distribution, multiple shadow models must be trained by repeatedly sampling a training set and training models. **Data augmentation-based auditing:** This form of auditing involves generating synthetic or modified versions of the data in order to assess and improve privacy guarantees. This approach is useful in evaluating models with overfitting tendencies, where small perturbations could reveal privacy weaknesses. [PERSON] et al. [184] present a notable connection between machine unlearning and MIA. Their method provides a mechanism for privacy auditing without modifying the model. By leveraging forgeability (creating new, synthetic data samples), data owners can construct Proof-of-Repudiation (PoR) concept that allows a model owner to refute claims made by MIAs to enhance privacy protection and mitigate privacy risks. Recently, there has been works on auditing lower bounds for differential **Renyi differential privacy (RDP)**. [PERSON] et al. [170] investigate the relationship between RDP and its interpretation in terms of a _statistical hypothesis testing_ interpretation. The authors investigate the conditions for a privacy definition based on statistical divergence, which allows for an improved conversion rules between divergence and differential privacy. They provide precise privacy loss bounds under RDP and interpret these in terms of type I and type II errors in hypothesis testing. [PERSON] et al. [185] develop a framework to estimate lower bounds for RDP parameters (especially for \(\epsilon-\text{RDP}\)) by investigating a mechanism in a black-box manner. Their framework allows auditors to derive minimal privacy guarantees without requiring internal access to the mechanism. Their goal was to observe how much the outputs deviate for small perturbations in the inputs. [PERSON] et al. [186] propose an auditing procedure of DP with the regularized kernel Renyi divergence (KRD) to define the regularized kernel Renyi differential privacy (KRDP). Their auditing procedure can estimate from samples even in high dimensions for \(\epsilon\)-DP, \((\epsilon,\delta)\)-DP, and \((\alpha,\epsilon)\)-Renyi DP. Their proposed auditing method does not suffer from curse of dimension and has parametric rates in high dimension. However this approach requires knowledge of the covariance matrix of the underlying mechanisms, which is impractical for most mechanisms other than Laplace and Gaussian mechanisms and inaccessible in black-box settings. [PERSON] et al. [40] introduce a family of function-based testers for Renyi DP (also for pure and approximate DP). The authors introduce DP-Auditorium, a DP auditing library implemented in Python that allows to test DP guarantees from the _black-box_ access to the mechanism. DP-Auditorium facilitates the development and execution of privacy audits, allowing researchers and practitioners to evaluate the robustness of DP implementations against various adversarial attacks, including _membership inference_ and _attribute inference_. The library also supports multiple privacy auditing protocols and integrates configurable privacy mechanisms, allowing for testing across different privacy budgets and settings. [PERSON] et al. [32] proposes a framework for _auditing private predictions_ with different _poisoning_ and _querying_ capabilities. They investigate privacy leakage in terms of Renyi DP of four private prediction algorithms PATE, CaPC, Prompt PATE and Private-KNN. The experiments have shown that there are algorithms that are easier to poison and lead to much higher privacy leakage. Moreover, the privacy leakage is significantly lower for attackers without query control than for attackers with full control. The authors have shown that privacy leakage is lower for attackers without query control.