# Privacy Auditing in Differential Private Machine Learning:
## 4 Privacy Auditing Schemes
### Privacy Audits
#### 4.3.2 Differential Privacy Auditing with Data Poisoning
The empirical privacy estimates are derived from Clopper-Pearson confidence intervals of the FP and FN rates of attacks. The authors provide a \(\epsilon\)-maximisation strategy to obtain a lower bound on the privacy leakage. **Empirical evaluation of privacy loss** evaluates how the privacy budgets are affected by poisoned data. Auditors measure the effective privacy loss, or empirical epsilon, by feeding in poisoned data and calculating whether the privacy budget remains within acceptable bounds. [PERSON] and [PERSON] [191] introduce auditing mechanisms that track the empirical privacy losses and provide insights into the impact of poisoned data on privacy guarantees in real-world applications. The authors clarify the relationship between pure and approximate DP by establishing quantitative bounds on privacy loss under different conditions and introducing adaptive data analysis. [PERSON] et al. [192] develop empirical privacy assessment methods applicable to DP-SGD in high-risk inversion settings. This method allows a detailed examination of how shuffling affects privacy guarantees. The authors evaluate the different parameters, such as batch size and privacy budget, in terms of privacy leakage. **Privacy violation:** The first work in the field of DP auditing, [PERSON] et al. [193] consider relaxing the DP notations to cover different types of privacy violations, such as unauthorized data collection, sharing and targeting. The authors outline key dimensions that influence privacy, such as individual factors (e.g., awareness and knowledge), technological factors (e.g., data processing), and contextual factors (e.g., legal framework). However, the data leakage is not assessed. [PERSON] et al. (1994) evaluate the existing DP implementations for correctness of implementation. The authors create a privacy evaluation framework, named DPBench. This framework is designed to evaluate, test and validate privacy guarantees. Recent work proposes efficient solutions for auditing simple privacy mechanisms for scalar or vector inputs to detect DP violations ([PERSON] et al., 2018; [PERSON] et al., 2021). For each neighboring input pair, the corresponding output is determined, and Monte Carlo probabilities are measured to determine privacy. [PERSON] et al. (2015) were the first to propose practical methods for testing privacy claims in black-box access to a mechanism. The authors designed StatDP, a hypothesis testing pipeline for checking DP violations in many classical DP algorithms, including noisy argmax and for identifying \(\epsilon\)-DP violations in sparse algorithms, such as the spare vector technique and local DP algorithms. Their work focuses on univariate testing of DP and evaluates the correctness of existing DP implementations. [PERSON] et al. (1995) offer a code analysis-based tool CheckDP to generate or prove counterexamples for a variety of algorithms, including spares variety vector algorithm. [PERSON] et al. (1996) investigate the decidability of DP. CheckDP and DiPC can not only detect violations of privacy claims, but can also be used for explicit verification. [PERSON] et al. (2016) present a privacy violation detection tool, DP-Sniper, which shows that a black-box setting can effectively identify privacy violations. It utilizes two strategies: (1) classifier training to train a classifier that predicts whether an observed output is likely to have been generated from one of two inputs; and (2) optimal attack transformation, where this classifier is then transformed into an approximately optimal attack on differential privacy. DP-Sniper is particularly effective at exploiting floating-point vulnerabilities in naively implemented algorithms and detecting significant privacy violations. [PERSON] et al. (1995) present DP-Opt(mizer), a disprover that attempts to find for counterexamples whose lower bounds on differential privacy exceed the claimed level of privacy guaranteed by the algorithm. The authors focus exclusively on \(\epsilon\)-DP.