# Privacy Auditing in Differential Private Machine Learning:
## 4 Privacy Auditing Schemes
### Privacy Audits
#### 4.3.1 Differential Privacy Auditing Using Membership Inference
(2018) establish a direct link between overfitting and membership inference vulnerabilities by analyzing confidence scores. It is shown that high-confidence predictions are often associated with data memorization, which increases privacy risks, especially when attackers exploit confidence scores. **Metric-based membership inference auditing:** This refers to the use of various metrics and statistics calculated from a ML model's outputs to assess the privacy risks in DPML systems. This approach applies membership inference techniques and calculates metrics and statistics that allow a quantitative assessment of privacy leakage. It is often used to compare different models and _privacy parameters_. [PERSON] et al. [180] evaluates DP mechanisms against MIAs and uses _accuracy and F-score_ as a privacy leakage metrics to measure the privacy loss on models trained with DP algorithms. [PERSON] and [PERSON] [21] evaluate the private mechanism against both membership inference and attribute inference attacks. They used balanced prior data distribution probability. Note that if the prior probability is skewed, the above-mentioned methods are not applicable. [PERSON] et al. [169] evaluate DP mechanism using a hypothesis testing framework. They connect _precision, recall_, and F-score_ metrics to the DP parameters \((\epsilon,\delta)\). Based on the attacker's background knowledge, they give insight into choosing these parameter values. [PERSON] et al. [170] explain DP through a _statistical hypothesis testing_ interpretation, where conditions for a privacy definition based on statistical divergence are identified, allowing for an improved conversion rules between divergence and differential privacy. [PERSON] et al. [102] investigate how neural networks unintentionally memorize specific training data. The authors develop an attack methodology to quantify unintended memorization by evaluating how easy it is to reconstruct specific data points (e.g., training examples with private information) from the trained model. This study uses metric-based approaches to measure memorization and unintended data retention, which are both critical components in determining membership inference. The research identifies factors contributing to memorization, including model size, training duration, and dataset characteristics. [PERSON] et al. [181] investigate the effectiveness of DP in protecting against MIAs in ML. The authors perform an empirical evaluation by varying the \(\epsilon\) values in DP-SGD and observing the effect on the success rate of MIAs including back-box and white-box attacks. The authors suggest that DP needs to be complemented by other techniques that specifically target membership inference risk. [PERSON] et al. [41] evaluate the impact of _privacy parameters_ by adjusting \(\epsilon\) on the effectiveness of DP in mitigating gradient-based MIAs. The authors recommend specific DP parameter settings and training procedures to improve privacy without sacrificing model utility. [PERSON] et al. [182] explore statistical methods for quantifying and verifying differential privacy (DP) claims. This method includes estimators and confidence intervals for the optimal privacy parameter \(\epsilon\) of a randomized algorithm and avoids the complex process of event selection, which simplifies the implementation, provides estimators and confidence intervals for the optimal privacy parameter \(\epsilon\) of a randomized algorithm. [PERSON] and [PERSON] [158] report on extensive hypothesis testing using DPML using the Neyman-Pearson criterion. They give guidance on setting the privacy budget on assumption about the attacker's knowledge considering different types of _auxiliary information_ that an attacker can obtain (to strengthen the MIA such as probability distribution of data, record correlation, and temporal correlation). [PERSON] et al. [183] design adaptive membership inference attacks based on the LiRA framework [103], which frames membership inference as a hypothesis testing problem.