SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Learning Rate	Weight Decay	η
CMNIST	{5e-4, 1e-3, 2e-3, 5e-3}	{0.0, 1e-4, 1e-3, 1e-2, 1e-1}	{1.0, 1.25, 1.5}
CCIFAR10	{1e-5, 5e-5, 1e-4, 5e-4, 1e-3}	{0.0, 1e-4, 1e-3, 1e-2, 1e-1}	{1.0, 1.25, 1.5, 1.75, 2.0}
SMPI3D	{1e-5, 5e-5, 1e-4, 5e-4, 1e-3}	{0.0, 1e-4, 1e-3, 1e-2, 1e-1}	1.0
WATERBIRDS	{1e-4, 5e-4, 1e-3, 5e-3}	{0.0, 1e-4, 1e-2}	{1.0, 1.5, 2.0, 2.5, 3.0}
CELEBA	{1e-5, 5e-5, 1e-4, 5e-4}	{0.0, 1e-5, 1e-4, 1e-3, 1e-2}	{1.0, 1.25, 1.5}
τ	Tssl/Tstop
CMNIST	{0.1, 1.0, 2.0}	{10, 20, 50, 100} epochs
CCIFAR10	{0.5, 1.0, 2.0}	{100, 200, 300, 400, 500} epochs
SMPI3D	1.0	{10, 20, 50, 100} epochs
WATERBIRDS	{0.1, 0.5, 1.0, 2.0}	{50, 100, 200, 500, 1000, 2000, 5000} steps
CELEBA	{0.5, 1.0, 2.0}	{10, 20, 50, 100} epochs
epochs for all datasets (except WATERBIRDS which we train for 200), and the best model across the training duration is selected for each trial, by evaluating our proposed validation score at the end of each epoch and monitoring for the maximum value. For cMNIST, CCIFAR10 and sMPI3D, we use our bias-unsupervised group-balanced validation score (see Equation (7)), whereas for WATERBIRDS and CELEBA the corresponding worst-group version.
Self-supervised Pretraining. For all cases, we adapt self-supervised pretraining recipes from solo-learn software library [22]. In particular, we optimize the MoCoV2+ loss with a LARS optimizer [72] using  $\eta_{\mathrm{LARS}} = 0.002$ . Furthermore, an MLP projector with a single hidden layer is used to project features to a hypersphere in a 256-dimensional space. Cosine scheduling is used for the learning rate and for the exponential moving average coefficient of the momentum encoder. We train for a maximum of 100 epochs for the CMNIST, sMPI3D and CELEBA tasks, and for 500 epochs for the CCIFAR10 datasets. We save periodically checkpoints of base models across training, as we are going to decide which one to use as a basis for the bias proxy and for the debiased model as a hyperparameter of ULA. For each dataset, we perform a small random search over contrastive temperature and data augmentation by maximizing the i.i.d. validation accuracy of an online linear classifier on-top of extracted representations. In general, we fix the contrastive temperature to 0.1 and use default Imagenet augmentations for sMPI3D and CCIFAR10, while we search for effective variants of those for CMNIST and CELEBA. Details about the hyperparameter combinations used are displayed at Table 7.
In all cases, we verify that the augmentations at Table 7 do not cause representations to be invariant to variations of the bias attribute. We take extra care in order to delineate that, especially for synthetic tasks, improvements are observed due to our ULA methodology, and not due to the use of data augmentations which nullify the bias attribute; something which, in real-life tasks such as CELEBA, might be impossible to do with hand-crafted transformation. We classify against the bias attribute using a linear classifier on extracted representations, and we find that, for the hyperparameters used, bias information is indeed retained (see Table 6).
Hyperparameter Search for ULA. For each dataset and task, we search over the respective spaces defined at Table 8. In particular, we sample uniformly 64 independent configurations (128 for WATERBIRDS) and we choose the one with the best bias-unsupervised validation score (Equation (7)). To make the space a bit smaller, we consider the same weight decay for the training of the bias proxy and the debiased logit adjusted network, as well as we consider the same number of training epochs
Table 8: Search space used to tune hyperparameters for each of the datasets considered
Learning Rate	Weight Decay	η	τ	Tssl (epochs)	Tstop
CMNIST (1%)	2e-3	1e-3	1.5	2.0	100	100 epochs
CCIFAR10 (1%)	5e-5	1e-4	1.0	1.0	500	500 epochs
sMPI3D (C=3)	5e-5	1e-4	1.0	1.0	100	100 epochs
WATERBIRDS	1e-3	0.0	3.0	0.1	-	50 steps
CELEBA	1e-4	0.0	1.5	0.5	10	10 epochs
Table 9: Best hyperparameters selected by our proposed bias-unsupervised validation score.
for SSL-pretrained base network and the linear head for the bias proxy (which we train online in our implementation). For WATERBIRDS, the linear head is trained on top of a pretrained ResNet50 onImagenet, which is taken from a PyTorch [59] model repository. The pretrained model's parameters remain frozen during the training of the bias proxy, so in this case there is no number of pretraining steps to select. The hyperparameters used for the benchmarking experiments, that we presented in Section 4, can be found at Table 9.
F Extended Experiments
(a) Left: Soft confusion matrix estimate using  $p_{\mathrm{bias}}(y_{\mathrm{bias}}|x)$  of the bias proxy and the formula of Eq. 5 in the main paper. Right: Estimated prior distribution between the target and bias variables using ground-truth bias observations.
(b) Accuracy of predicting the bias variable via arg max  $y p_{\mathrm{bias}}(y_{\mathrm{bias}}|x)$ , even though the bias proxy was trained on target variables.
Figure 6: Left: Ablation of final group-balanced test accuracy  $(\%)$  on cCIFAR10  $(1\%)$  with respect to the number of epochs we train the linear head on-top of a pretrained backbone with BYOL to produce the proxy network for the bias variable. Right: Progression of accuracy metrics  $(\%)$  during training of a debiased network with uLA. Our validation accuracy (orange) tracks better the group-balanced test accuracy (green), than the standard in-distribution validation accuracy. Model selection with our criterion will result in a better model (close to the mode of the test curve) compared to the one using i.i.d. validation.
Figure 5: Heatmaps displaying the effectiveness of the proposed method to derive a proxy network for predicting the bias variable. Backbone is pretrained using BYOL on cCIFAR10 (1%).