# Privacy Auditing in Differential Private Machine Learning:
## 4 Privacy Auditing Schemes
### Privacy Audits
#### 4.3.1 Differential Privacy Auditing Using Membership Inference
(2017) evaluates the membership inference attack in a black-box environment in which the attacker only has access to the target model via a query. The attack algorithm is based on the concept of _shadow models_. The attacker trains shadow models that are similar to the target model and uses these shadow models to train a membership inference model. MIA is modeled as a binary classification task for an attack model that is trained using the prediction of shadow models on the attacker dataset. [PERSON] et al. [112] utilize data augmentation to create shadow models and analyze privacy leakage. It provides insights into the impact of data transformations on inference accuracy in both black-box and white-box settings. [PERSON] et al. [10] investigate overfitting in ML model auditing using a threshold membership inference attack as a primary method and attribute inference attack based on distinction between training and testing per-instance losses. The authors provide an upper bound on the success of MIA as a function of the parameters in DP. By training shadow models, the authors demonstrate how models that memorize training data are more susceptible to MIAs, especially when DP techniques are not optimally applied. They conclude that overfitting is sufficient for an attacker to perform MIA. [PERSON] et al. [22] focus on the comparison of black-box attacks and white-box attacks by effectively estimating the model loss for a data point. The authors use the shadow model technique to demonstrate MIAs across architectures and training methods. They also investigate Bayes optimal strategies for MIAs that leverage knowledge of model parameters in white-box settings. Their findings suggest that white-box attacks do not require specific information about model weights and losses, but can still be performed effectively using probabilistic assumptions, and that optimal attacks depend on the loss function and thus black-box attacks are as good as white-box attacks. The authors introduce the Inverse Hessian attack (IHA), which utilizes model parameters to enhance the effectiveness of membership inference. By computing inverse-Hessian vector products, these attacks can exploit the sensitivity of model output to specific training examples. **Label-only membership auditing:** label-only membership inference auditing in differentially private machine learning is a privacy assessment method where an auditor attempts to deduce whether a particular data point was part of the training dataset based on the model's predicted labels (without access to probabilities or other model details). This form of auditing is particularly relevant for real-world scenarios. **[PERSON] et al. [178]** adapt a heuristic method to evaluate label differential privacy (Label DP) in different configurations where privacy of the labels associated with training examples is preserved, while features may be publicly accessible. The authors propose two primary approaches: Private Aggregation of Teacher Ensembles (PATE) and Additive Laplace with Iterative Bayesian Inference (ALBI). They apply noise exclusively to the labels in the training data, leading to the development of different label-DP mechanisms, and investigate model accuracy and estimate lower bounds for the privacy parameter \(\epsilon\) values. They trained several models with and without a training point, while the rest of the training set remained unchanged. [PERSON] et al. [179] focus on a black-box attack, where only the labels of the model, and not the full probability distribution, are available to the attackers. They investigate privacy leakage in four private prediction algorithms: PATE, CaPC, PromptPATE, and Private-KNN. The authors showed that DP provides the strongest protection against privacy violation in both the average-case and worst-case scenarios and when the model is trained with overconfidence. However, this may come at the expense of the model's test accuracy.