# Privacy Auditing in Differential Private Machine Learning:
## 4 Privacy Auditing Schemes
### Privacy Auditing Techniques  
Privacy auditing techniques in differential privacy are essential to ensure that privacy guarantees are met in practical implementations. Empirical auditing techniques establish practical lower bounds on privacy leakage, complimenting the theoretical upper bounds provided by DP [32]. Before we address privacy auditing schemes, it is necessary to explain the main auditing techniques ([PERSON] et al., 2024) that have been used to evaluate the effectiveness of DP mechanisms and algorithms against privacy attacks in ML models. **Canary-based auditing:** Canary-based auditing is a technique for assessing the privacy guarantees of DPML algorithms by introducing specially designed examples, known as canaries, into the dataset [43; 160]. The auditor then tests whether these canaries are included in the outputs of the model and distinguishes between models trained with different numbers of canaries. An effective DP should limit the sensitivity of the model to the presence of these canaries, minimizing the privacy risk. Canaries must be carefully designed to ensure that they can detect potential privacy leaks without jeopardizing overall privacy guarantees. Canary-based auditing often requires dealing with randomized datasets, which enables the development of randomized canaries. The introduction of Lifted Differential Privacy (LiDP) [160] by distinguishing between models trained with different numbers of canaries can leverage statistical tests and novel confidence intervals to improve sample complexity. There are several canary strategies: (1) a random sample from a dataset distribution with a false label, (2) the use of an empty sample, (3) an adversarial sample, and (4) the canary crafting approach [42]. The disadvantage of canaries is that an attacker must have access to the underlying dataset and knowledge of the domain and model architecture. **Statistical auditing:** In this context, statistical methods are used to empirically evaluate privacy guarantees [167]. These include influence-based attacks and improved privacy search methods that can be used to detect privacy violations and understand information leakage in datasets, thus greatly improving the auditing performance of various models, such as logistic regression and random forest [62]. **Statistical hypothesis testing interpretation:** The aim of this approach is to find the optimal trade-off between type I and type II errors [168]. This means that no test can effectively determine whether a specific individual's data are included in a dataset, ensuring that both high power and high significance are simultaneously unattainable simultaneously [169; 170]. It is used to derive the theoretical upper bound and is very useful in deriving the tight compositions [171] and has even motivated a new relaxed notion of DP called f-DP [76]. **Single training run auditing** (also known as **one-shot auditing**): It enables privacy auditing during a single training run, eliminating the need for multiple retraining sessions. This technique utilizes the parallelism of independently adding or removing multiple training examples and enables meaningful empirical privacy reduction with only one training run of the model [43]. The technique is efficient and requires no prior knowledge of the model architecture or DP algorithm. This method is particularly useful in FL settings and provides accurate estimates of privacy loss under the Gaussian mechanism [118]. **Empirical privacy estimation:** In this technique, the actual privacy loss of an algorithm is evaluated by practical experiments rather than theoretical guarantees [172]. This technique is used to audit the implementations of DP mechanisms or claims about the models trained with DP [42]. They are also useful to estimate the privacy loss in cases where a tight analytical upper bound on \(\epsilon\) is unknown. **Post hoc privacy auditing:** This technique traditionally establishes a set of lower bounds for privacy loss (e.g., thresholds).