Privacy auditing experiments All experiments were conducted on a single workstation computer equipped with 64 GB of RAM and a 12-core Intel Xeon CPU with a Thermal Design Power of 105 W and implemented in the Python programming language. For the synthetic learning tasks, a dataset of 1025 samples with one feature each, split evenly across two classes, was generated using scikit-learn's make_classification method. Of these, 1024 samples were used for shadow model training and a single example  $(x^{*})$  was held out and its label flipped, similarly to the data poisoning technique of [12] to maximise the point's impact on the model's confidence. This process was repeated for 1024 iterations, and shadow models were trained. No fixed random seed was used. Shadow models were trained in pytorch using full-batch DP gradient descent for the experiments in the main manuscript and Poisson sampling for the experiment in the Appendix, with a sample-wise  $\ell_{2}$  norm bound of 1, a noise scale of 1 and implemented using a single linear layer with a single unit. Models were trained to convergence on the training database, at which point the training loss was  $< 0.1$ . In every case, we considered a single step of model training, i.e. disregarded all previous model updates. For the logit-space auditing, logits were transformed by applying the logistic sigmoid activation function, converted to confidence scores and a Gaussian likelihood fitted as described in the Likelihood Ratio Attack Section IV.C of [12]. For executing the offline MI attacks, we followed Algorithm 1 of [12] with the modifications described to implement offline MI. In brief, likelihoods were fitted to the so-called IN models, i.e. the ones the adversary is assumed to have access to in the RTM. The models' confidence was then measured on the unseen sample  $x^{*}$  to determine its membership status by implementing the one-sided hypothesis test described in [12] for offline MI. To compute the trade-off functions, the test statistic was evaluated at 50 critical values using scikit-learn's ROC functionality, as the ROC and the trade-off function are complements of each other. Results are reported as averages and standard deviations over the  $\beta$  at each critical value  $\alpha$ , whereby the critical values were chosen identically for each repetition. For the gradient space auditing, the gradients of the final training iteration, when norms were lowest, were used as a reference "database"  $(D)$ . To these, a gradient with a magnitude of  $\Delta_{2}$ , i.e. equal to the clipping threshold was added to simulate the maximal effect of  $x^{*}$ , similar to the Dirac canary technique described in [11]. Noise was then added to all gradient samples. Then, the magnitude of the gradients was used as a test statistic as discussed in Appendix A.B. of [12], and as proven to correspond to the optimal hypothesis test under the RTM in Lemma 4. Similar to above, the test statistic was evaluated at 50 critical value settings and the ROCs were computed. For the high-dimensional experiment, we utilised the diabetes dataset [30], a 10-dimensional regression dataset with 442 samples. The same strategy was followed as in the gradient space auditing section above, but this time, a model with a latent space dimensionality of 50 was implemented. The noise multiplier was set to 0.5. Once again, a single Dirac canary gradient was added to the gradient "database" and the ROCs were computed as above. The experiments did not require a graphics processing unit and required approximately 60 minutes of net computation time on a CPU. At an assumed carbon efficiency of  $0.432\mathrm{kg / kWh}$ , the computation of these experiments caused approximately  $0.04\mathrm{kg}$  of  $\mathrm{CO}_{2}$  equivalent emissions.