SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We can relate this decrease in pinball loss conditional on  $g(x) = 1$  to the decrease in pinball loss on the underlying distribution  $\mathcal{D}$ :
$$
\begin{array}{l} \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \left[ \mathrm {P B} _ {1 - \alpha} \left(q ^ {\prime} (x), s (x, y)\right) \right] \\ = \operatorname * {P r} _ {\mathcal {D}} [ g (x) = 1 ] \underset {(x, y) \sim \mathcal {D} _ {g}} {\mathbb {E}} \left[ \mathrm {P B} _ {1 - \alpha} \left(q ^ {\prime} (x), s (x, y)\right) \right] + \operatorname * {P r} _ {\mathcal {D}} [ g (x) = 0 ] \underset {(x, y) \sim \mathcal {D} _ {\bar {g}}} {\mathbb {E}} \left[ \mathrm {P B} _ {1 - \alpha} \left(q ^ {\prime} (x), s (x, y)\right) \right] \\ <   \operatorname * {P r} _ {\mathcal {D}} [ g (x) = 1 ] \underset {(x, y) \sim \mathcal {D} _ {g}} {\mathbb {E}} \left[ \mathrm {P B} _ {1 - \alpha} \left(q ^ {\prime} (x), s (x, y)\right) \right] + \operatorname * {P r} _ {\mathcal {D}} [ g (x) = 0 ] \underset {(x, y) \sim \mathcal {D} _ {\bar {g}}} {\mathbb {E}} \left[ \mathrm {P B} _ {1 - \alpha} \left(q (x), s (x, y)\right) \right] \\ = \operatorname * {P r} _ {\mathcal {D}} [ g (x) = 1 ] \underset {(x, y) \sim \mathcal {D} _ {g}} {\mathbb {E}} \left[ \mathrm {P B} _ {1 - \alpha} \left(q ^ {\prime} (x), s (x, y)\right) \right] + \operatorname * {P r} _ {\mathcal {D}} [ g (x) = 0 ] \underset {(x, y) \sim \mathcal {D} _ {\bar {g}}} {\mathbb {E}} \left[ \mathrm {P B} _ {1 - \alpha} \left(q ^ {\prime} (x), s (x, y)\right) \right] \\ \end{array}
$$
But because  $\mathcal{H}$  is closed under additive shifts by all  $g\in \mathcal{G}$ , we also have that  $q^{\prime}\in \mathcal{H}$ . Together, these contradict the optimality of  $q$  as measured by pinball loss, which completes the proof.
We use Ray Tune Liaw et al. [2018] for hyperparameter tuning on image datasets. All experiments use Async Hyperband Scheduler Li et al. [2020] and the Hyperopt search package Bergstra et al. [2013]. Table 4 summarizes the hyperparameters that were tuned and their configurations
Table 4: Summary of hyperparameters optimized for our quantile regressor model on all image experiments
Hyperparameter	Configuration	Description
lr	loguniform(10-6, 10-2)	Learning rate
Weight Decay	loguniform(2 * 10-6, 5 * 10-3)	12 weight regularization (excluding biases)
Hidden dims	choice(), [512,512])	size and number of hidden dimensions of MLP
Accumulate gradient batches	choice([1,2,4,8,16,32,64])	number of batches to accumulate (base batch size=32)
Epochs	randid( start=5, stop=50, step=5)	number of training epochs
We ran 32 trials to find the optimal configuration per experiment. After examining the results, we found that a hidden dim of [512, 512] and accumulate gradient batches of 2 (Effective batch size 64) were consistently chosen across all experiments. Similarly, epochs were mostly chosen on the  $10 - 20$  range, learning rates and weight decays were consistently chosen near the middle of the value range. This indicates that hyperparameter tuning may not be especially task sensitive and can be shared across attacks.
For tabular data, we use Catboost for model training, and Optuna Akiba et al. [2019] for hyperparameter tuning. Table 5 presents the hyperparameters that were tuned and their corresponding ranges. Each model was tuned with 300 trials with 5-fold cross-validation.
Table 5: Summary of hyperparameters optimized for our quantile regressor model on tabular data
Hyperparameter	Configuration	Description
depth	uniform(1,10)	Depth of a tree
l2_leaf_reg	loguniform(1e-2,1e+6)	Strength of L2 regularization
learning_rate	loguniform(1e-6,1)	Learning rate of gradient boosting
subsample	loguniform(1e-2,1)	Subsampling ratio at each leave node
iterations	loguniform(1,1000)	Number of boosting iterations
Here we show extended results on membership inference attacks on CIFAR-10 and CIFAR-100 for ResNet-10, -18, -34, and -50 architectures. Tables 6 and 7 show precision and pinball loss at  $1\%$  and  $0.1\%$  FPR for all target architectures on CIFAR-10 and CIFAR-100 respectively. We observe a strong correlation between lowest pinball loss on test samples and highest precision across the majority of experiments. Figure 5 additionally presents a visual comparison of the true positive rate and false positive rate trade-off for all tested methods on CIFAR-10 and CIFAR-100 for the ResNet-50 architecture.
Table 6: Precision and pinball loss of all membership inference attack at  $1\%$  and  $0.1\%$  false positive rates on CIFAR10 for ResNet-10, -18, -34, and -50 architectures. Pinball losses are computed on a held out test set. Lower pinball losses consistently predict better membership inference performance.
Method	PB1%	Precision @ 1% FPR	PB0.1%	Precision @ 0.1% FPR
CIFAR-10 ResNet-10
LIRA (n=2)	0.1454	80.43%	0.0194	89.20%
LIRA (n=4)	0.1439	83.45%	0.0193	93.15%
LIRA (n=6)	0.1441	84.46%	0.0193	94.47%
LIRA (n=8)	0.1442	84.79%	0.0193	94.67%
Marginal Baseline	0.2157	47.84%	0.0262	46.81%
Ours	0.1543	62.69%	0.0262	57.83%
CIFAR-10 ResNet-18
LIRA (n=2)	0.1637	82.17%	0.0222	94.09%
LIRA (n=4)	0.1607	84.91%	0.0216	93.90%
LIRA (n=6)	0.1603	85.33%	0.0215	94.68%
LIRA (n=8)	0.1603	85.26%	0.0215	95.04%
Marginal Baseline	0.1853	43.43%	0.0223	50.00%
Ours	0.1707	63.08%	0.0213	65.65%
CIFAR-10 ResNet-34
LIRA (n=2)	0.1674	82.75%	0.0225	92.45%
LIRA (n=4)	0.1672	84.16%	0.0223	94.45%
LIRA (n=6)	0.1669	85.65%	0.0222	95.40%
LIRA (n=8)	0.1668	85.79%	0.0222	95.30%
Marginal Baseline	0.1893	51.42%	0.0230	47.92%
Ours	0.1743	73.06%	0.0230	80.39%
CIFAR-10 ResNet-50
LIRA (n=2)	0.1913	78.55%	0.0262	83.18%
LIRA (n=4)	0.1926	80.52%	0.0261	91.48%
LIRA (n=6)	0.1930	83.19%	0.0260	93.17%
LIRA (n=8)	0.1933	83.00%	0.0260	93.70%
Marginal Baseline	0.2163	48.70%	0.0287	60.94%
Ours	0.2070	62.95%	0.0277	59.68%
(b) CIFAR-100
Figure 5: Comparing the true positive rate vs. false positive rate of our membership inference attack against the state-of-the-art shadow model approach LIRA proposed in Carlini et al. [2022] evaluated at 2, 4, 6, and 8 shadow models, and the marginal baseline proposed in Yeom et al. [2018] Our single-model quantile regression attack can reliably perform membership inference attacks on a ResNet-50 CIFAR-10/CIFAR-100 target model (91% and 68.6% test accuracies respectively) without relying on any knowledge of the target architecture. We include a visual readout of the 256-shadow model attack shown in Carlini et al. [2022] for reference. Our attack's effectiveness is dominates the marginal baseline but falls short of LIRA in this scenario. We find pinball loss to be a strong predictor of performance for membership inference attacks.
Table 7: Precision and pinball loss of all membership inference attack at  $1\%$  and  $0.1\%$  false positive rates on CIFAR100 for ResNet-10, -18, -34, and -50 architectures. Pinball losses are computed on a held out test set. Lower pinball losses consistently predict better membership inference performance.
Method	PB1%	Precision @ 1% FPR	PB0.1%	Precision @ 0.1% FPR
CIFAR-100 ResNet-10
LIRA (n=2)	0.1486	95.44%	0.0188	99.06%
LIRA (n=4)	0.1386	95.95%	0.0170	98.97%
LIRA (n=6)	0.1385	96.19%	0.0170	99.10%
LIRA (n=8)	0.1385	96.14%	0.0170	99.06%
Marginal Baseline	0.2749	50.84%	0.0390	47.92%
Ours	0.1775	83.30%	0.0390	77.37%
CIFAR-100 ResNet-18
LIRA (n=2)	0.1612	96.29%	0.0193	99.11%
LIRA (n=4)	0.1523	96.74%	0.0179	99.46%
LIRA (n=6)	0.1524	97.02%	0.0181	99.51%
LIRA (n=8)	0.1525	96.94%	0.0181	99.48%
Marginal Baseline	0.2754	51.71%	0.0364	59.02%
Ours	0.2492	90.37%	0.0364	89.63%
CIFAR-100 ResNet-34
LIRA (n=2)	0.1840	95.73%	0.0213	99.10%
LIRA (n=4)	0.1732	96.34%	0.0203	99.17%
LIRA (n=6)	0.1726	96.45%	0.0202	99.31%
LIRA (n=8)	0.1727	96.42%	0.0202	99.28%
Marginal Baseline	0.2065	62.13%	0.0270	66.67%
Ours	0.1913	81.80%	0.0270	79.64%
CIFAR-100 ResNet-50
LIRA (n=2)	0.1926	95.21%	0.0239	98.65%
LIRA (n=4)	0.1832	95.87%	0.0227	98.94%
LIRA (n=6)	0.1828	96.20%	0.0228	99.02%
LIRA (n=8)	0.1828	96.07%	0.0228	98.98%
Marginal Baseline	0.2188	58.81%	0.0272	65.75%
Ours	0.2006	79.57%	0.0272	85.41%