SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $p_{\mathrm{bias}}(y_{\mathrm{bias}}|x) \propto \exp (h_{\mathrm{bias}}(x)y_{\mathrm{bias}} / \tau)$  is the biased model conditional. Note that  $p_{\mathrm{bias}}(y_{\mathrm{bias}}|x)$  is post-hoc calibrated by a temperature hyperparameter  $\tau$ . As we rely on the biased network to approximate the spurious correlation structure described in  $p_{\mathrm{data}}(Y,Z)$ , it is crucial that the predicted conditional probabilities of the biased network are calibrated correctly [31, 56, 53]. Afterwards, we are ready to begin training a debiased network with logit adjustment as in Section 2; using only the predictions of  $h_{\mathrm{bias}}$  as the bias variable. The debiased network is initialized at the composition of a random linear classifier with the pretrained network  $f_{\mathrm{base}}$ , and during training we finetune it while adjusting its output logits by
$$
p _ {\theta} (y \mid x) \propto \exp \left(h _ {\theta} (x) _ {y} + \eta \log \hat {p} _ {\text {d a t a}} (y \mid \hat {y} (x; h _ {\text {b i a s}})\right), \tag {6}
$$
where  $\eta \geq 0$  is a hyperparameter controlling the strength of the additive logit bias. Notice that for  $\eta = 0$  we fall back to ERM training. By tuning  $\eta$  we can mitigate calibration errors of the debiased model [53], similar to what  $\tau$  does for the bias proxy. Selected hyperparameter configurations can be found in Appendix E.
3.2 Bias-unsupervised Validation
We re-purpose the pretrained biased classifier  $h_{\mathrm{bias}}$  so that training no longer requires bias-annotated validation data for model selection. Our bias unsupervised validation criterion calculates a balanced accuracy across pairs  $(y, y_{\mathrm{bias}}) \in [K] \times [K]$  of true labels and biased classifier predictions. In practice, we compute
$$
\widehat {\operatorname {B a l A c c}} (f; h _ {\text {b i a s}}) := \frac {1}{K ^ {2}} \sum_ {y, y _ {\text {b i a s}}} \frac {1}{\left| S _ {y , y _ {\text {b i a s}}} \right|} \sum_ {x _ {i}, y _ {i} \in S _ {y, y _ {\text {b i a s}}}} \mathbb {1} \left(y _ {i} = \hat {y} \left(x _ {i}; f\right)\right) \tag {7}
$$
$$
S _ {y, y _ {\text {b i a s}}} := \left\{\left(x _ {i}, y _ {i}\right) \in \mathcal {D} _ {\text {v a l i d}} \mid y _ {i} = y \text {a n d} \hat {y} \left(x _ {i}; h _ {\text {b i a s}}\right) = y _ {\text {b i a s}} \right\},
$$
where  $S_{y,y_{\mathrm{bias}}}$  are partitions of  $\mathcal{D}_{\mathrm{valid}}$  based on the value of predictions of  $h_{\mathrm{bias}}$  on a sample  $x_{i}$  and its ground-truth target label  $y_{i}$ . This corresponds to a form of group-balanced accuracy. Alternatively, we could also calculate a form of worst-group accuracy by taking the minimum across  $S_{y,y_{\mathrm{bias}}}$ . We find that worst-case validation is more suitable for tasks with small number of classes  $K$ .
During training, we evaluate models at every epoch and we select the one that maximizes our validation score across the duration of a training trial. In addition, we use this criterion to tune hyperparameters. In particular, for each task we tune learning rate, weight decay, logit adjustment strength coefficient  $\eta$ , calibration temperature  $\tau$  and, in addition, the number of pretraining steps for the SSL backbone - whenever it is applicable - and for the linear classification probe of the bias proxy network. Fig. 1 depicts our bias-unsupervised training and validation procedures.
Datasets. The tasks we consider are all specific instances of the setup above (see Fig. 2 and Section 2). This spans group robustness challenges like colored MNIST [57, CMNIST], corrupted CIFAR10 [37, CCIFAR10] and WATERBIRDS [62], fair classification benchmarks like CELEBA [51], and systematic generalization tasks such as the contributed sMPI3D. Details about their construction can be found in Appendix D.
Training Setup. For CMNIST, we train a 3-hidden layer MLP, while we use a ResNet18 [34] for CCIFAR10 and sMPI3D, and a ResNet50 for WATERBIRDS and CELEBA. For all datasets except WATERBIRDS, we pretrain the base model with the MOCoV2+ [15] process, while training of the linear probe for the bias network and finetuning for the logit adjusted debiased network happen with
Bias Labels	CMNIST	ccIFAR10
Train	Val	0.5%	1.0%	2.0%	5.0%	0.5%	1.0%	2.0%	5.0%
[61] GROUPDRO†	✓	✓	63.12	68.78	76.30	84.20	33.44	38.30	45.81	57.32
[57] LFF*	X	✓	52.50±2.43	61.89±4.97	71.03±2.44	80.57±3.84	28.57±1.30	33.07±0.77	39.91±0.30	50.27±1.56
[17] DFA*	X	✓	65.22±4.41	81.73±2.34	84.79±0.95	89.66±1.09	29.95±0.71	36.49±1.79	41.78±2.29	51.13±1.28
[50] LC†	X	✓	71.25±3.17	82.25±2.11	86.21±1.02	91.16±0.97	34.56±0.69	37.34±0.69	47.81±2.00	54.55±1.26
ERM*	X	X	35.19±3.49	52.09±2.88	65.86±3.59	82.17±0.74	23.08±1.25	25.82±0.33	30.06±0.71	39.42±0.64
ULA (ours)	X	X	75.13±0.78	81.80±1.41	84.79±1.10	92.79±0.85	34.39±1.14	62.49±0.74	63.88±1.07	74.49±0.58
Table 1: Results using datasets from Chu et al. [17] for various % of bias-conflicting examples in the training set. We report avg. group-balanced test accuracy  $(\%)$  and std. dev. over 5 seeds. *Results from Chu et al. [17]. †Results from Liu et al. [50].
Bias Labels	WATERBIRDS	CELEBA
Train	Val	i.i.d.	worst group	i.i.d.	worst group
[61] GROUPDRO*	✓	✓	93.5	91.4	92.9	88.9
[9] ERM	X	✓	97.6	86.7	93.1	77.8
[57] LFF*	X	✓	97.5	75.2	86.0	77.2
[48] JTT*	X	✓	93.6	86.0	88.0	81.1
[50] LC	X	✓	-	90.5±1.1	-	88.1±0.8
ERM	X	X	97.3	72.6	95.6	47.2
Bardenhagen et al. [9]	X	X	97.5	78.5	88.0	78.9
[3] MASKTUNE	X	X	93.0±0.7	86.4±1.9	91.3±0.1	78.0±1.2
ULA (ours)	X	X	91.5±0.7	86.1±1.5	93.9±0.2	86.5±3.7
Table 2: Results on WATERBIRDS and CELEBA. We report avg. test accuracy  $(\%)$  and std. dev. over 5 seeds. *Results from Liu et al. [48].
AdamW [52] optimizer. For WATERBIRDS instead, we leverage a base model which was pretrained onImagenet [60], following baselines in the literature for fair comparison, and we finetune it using SGD. Finally, for CMNIST, CCIFAR10 and sMPI3D we use our group-balanced bias-unsupervised validation criterion, whereas for WATERBIRDS and CELEBA the worst-group version. Further details are described in Appendix E.
Baselines. We compare ULA with vanilla ERM and a diverse set of group robustness techniques described in Section 5. GROUPDRO [61] provides with a fully bias-supervised baseline, while LFF [57], JrT [48], LC [50] and DFA [17] are bias unsupervised during training although they require bias annotations during validation to achieve robust optimal performance. We also consider two fully bias-unsupervised methods: Bardenhagen et al. [9] propose early stopping networks to derive a proxy for bias-unsupervised validation, and MASKTUNE [3] which provide competitive results under fully bias-unsupervised benchmarks without performing any validation procedure.
4.1 Results on Benchmarks
In Table 1, we report the group-balanced accuracy on cMNIST and cCIFAR10, across different percentages of bias-conflicting examples in the training set. For cMNIST, we observe that our method performs overall competitively against LFF, DFA and LC, even though these baselines use bias annotations during model selection. On the other hand, for cCIFAR10, we observe a significantly improved group robust performance for 3/4 difficulty levels. The highest difference is observed at the  $1.0\%$  task, where our method outperforms GROUPDRO by about  $24\%$  absolute increase in group-balance accuracy.
In Table 2, we observe worst-group accuracy results in the more challenging WATERBIRDS and CELEBA datasets. In both cases, our approach performs again competitively among bias-unsupervised training algorithms that leverage bias information during validation, falling slightly behind LC, which is the best out of the ones considered. Notably, our approach is still the best performing fully bias-unsupervised method, outperforming the supervised learning pretraining validation scheme of Bardenhagen et al. [9], and performing on par with MASKTUNE on WATERBIRDS and better than it on CELEBA by  $\approx 8\%$  absolute worst-group accuracy.
Bias Labels	sMPI3D (various C colors / shape)
Train & Val	2	3	4	5
% i.i.d. samples	-	33.33	50.00	66.66	83.33
[61] GROUPDRO	✓	31.23±1.88	46.01±4.13	69.68±7.82	82.18±8.39
ERM	X	31.94±1.67	47.68±4.06	71.94±7.71	83.10±8.64
[48] JT	X	31.89±0.88	48.93±2.04	67.78±3.14	83.31±3.51
[50] LC*	X	31.29±0.96	45.01±2.31	61.67±5.06	94.62±0.88
ULA (ours)	X	59.58±3.12	80.53±3.16	91.11±3.52	98.05±0.64
Table 3: Results on sMPI3D for various numbers of  $C$  colors per shape value in the training set (see Appendix D). We report avg. group-balanced accuracy (%) and std. dev. over 5 seeds & dataset generations. *GroupMix augmentation was not used for fairness of comparison.
Figure 3: Ablations on the influence of SSL pretraining and on the use of VANILLA process, SLA, or ULA for training against the downstream task. For CCIFAR10 (1%) and sMPI3D ( $C = 3$ ) we report the group-balanced test accuracy, whereas for CELEBA the worst-group. We report avg. accuracies and std. dev. over 5 seeds.
4.2 Systematic Generalization
sMPI3D is our contributed task which we use to study combinatorial systematicity in classifiers. With this benchmark, we aim to study the ability of classifiers to generalize to samples generated from novel combinations of observed generative attributes values, which under  $p_{\mathrm{data}}$  have 0 probability. The target task is to classify the shape of an object, which is spuriously correlated with its color. We devise 4 difficulty levels for this task which we denote by  $C$ , the number of color values present in the training set per shape. In Fig. 2d, we display a possible split between in-distribution and o.o.d. combinations of attributes for  $C = 4$ . Details about its construction are given in Appendix D.
Results. In Table 3, we further validate our approach on our contributed systematic generalization task. Under this setting, all bias-unsupervised approaches are evaluated fairly since their models are validated with exactly the same data resources; access to bias labels cannot give model selection advantage to any algorithm since there are no o.o.d. samples in the validation (just like in the training) split. Our method is the only one which consistently offers group-balanced accuracy improvements across difficulty levels, demonstrating generalization to o.o.d. samples. On the other hand, GROUPDRO, vanilla ERM, JTT and LC are not able to increase group-balanced accuracy over the percentage of i.i.d. samples present in the balanced test set.
We perform a set of studies to understand better the efficacy of our approach. In Fig. 3, we ablate the choice of training paradigm and the influence of a pretrained base model using SSL to the robust performance of a trained model to the target task. The paradigms we choose are among vanilla cross-entropy minimization, ULA and sLA (bias-supervised logit adjustment - see Section 2). For this ablation, a bias-supervised validation procedure was used for comparison against the fully bias-supervised sLA baseline. We find that finetuning the pretrained base model gives the best performance across training paradigms and tasks. Second, forn CELEBA and CCIFAR10, vanilla cross-entropy finetuning does not offer stark performance improvement over vanilla training from scratch. Only when we apply a logit adjustment training procedure, we are able to take significant advantage of the learnt representation space. At the same time, the accuracy gaps between ULA and
Group-balanced Test Acc.
ERM	25.82±0.33
GROUPDRO	38.30
ULA w. MoCoV2+	62.49±0.74
ULA w. BYOL	59.73±2.03
ULA w. BARLOWTWINS	50.08±1.23
Figure 4: Validation vs test on CCIFAR10 (1%).
Table 4: Ablation of SSL pretraining methods for the backbone model. Default  $\eta = 1.0$  and  $\tau = 1.0$  are used. We pretrain for 1000 epochs, and train the linear head of the bias proxy for 100 epochs. We report avg. group-balanced test accuracy (\%) and std. dev. over 5 seeds on cCIFAR10 (1%).
Pretraining method	Pearson correlation
End-to-end supervised	0.690 (0.588, 0.770)
SSL + linear head	0.819 (0.721, 0.885)
Table 5: Effect of pretraining method for a bias proxy network on results of hyperparameter search using our validation criterion. Each training trial's hyperparameters were sampled from the same prior. Pretraining with SSL results in trials whose best validation criterion correlates better with selected model's group-balanced test accuracy on CCIFAR10 (1%). In parenthesis, we compute 95% confidence intervals.
SLA are small which indicates that we are able to recover correctly the bias attribute with the proxy network. On the other hand, for sMPI3D the gaps between ULA and sLA are large. This shows that, in the systematic generalization case, bias extraction remains an open challenge for future work, and that improvement over baseline procedures in Table 3 is due to pretraining with SSL.
In addition, we study the impact of the choice of SSL method that we use to pretrain the backbone. For this reason, we perform an ablation experiment on CCIFAR10 (1%) by changing the pretraining strategy from MOCoV2+ to BYOL [30] or BARLOW TWINS [73]. We find that, while BYOL performs within the error margin of the best MOCoV2+ setting, BARLOW TWINS underperforms. BARLOW TWINS seeks to match the empirical (i.i.d.) cross-correlation between features to the identity. Arguably, we expect that the cross-correlation is different under the shifted test set. In any case, ULA significantly outperforms the non-ULA baselines with any of the considered SSL methods.
Finally, we study how the choice of pretraining paradigm for the bias network influences the quality of hyperparameter search using our proposed validation criterion. In Fig. 4, we present two separate searches on the same space of hyperparameters using two different pretraining approaches. In red we see our approach of pretraining an SSL base model for the bias network, and in blue we see a baseline approach where we pretrain with purely supervised learning. We observe that SSL pretraining enables stronger correlation between the proposed bias-unsupervised group-balanced validation criterion and the corresponding test accuracy on CCIFAR10. It is more difficult to tune hyperparameters with a bias network pretrained with supervised learning, because it may fit the training set entirely. In that case, the validation criterion collapses to the in-distribution test accuracy which is not indicative of the group-balanced test accuracy. On the contrary, classification with a linear probe on top of SSL representations prevents from fitting the training set entirely, having small generalization gaps in-distribution [8]. In this way, the validation criterion remains strongly correlated even in larger validation accuracy values, maintaining its utility in a greater range of hyperparameter configurations.
Prior literature can be grouped according to three main strategies that attempt to improve a model's robustness to dataset bias [54]. (i) Resampling strategies increase or decrease the frequency of biased
attributes in the input space [27, 55] or latent space [23, 74, 17]. (ii) Loss reweighting methods balance class or feature importances during training [74, 21, 12, 65, 40]. Especially relevant to our work are the reweighting methods that improve robustness and generalization when training on biased datasets [61, 57, 17, 48]. (iii) Post-hoc adaptation methods [54, 25, 19, 42, 41, 3, 70] correct the biases learned by already-trained models. Most relevant to our work is the logit adjustment technique proposed by Menon et al. [54] for long-tail learning, which leverages class frequencies to rebalance the model predictions after training or to train with a loss function which is aware of the class prior. We develop a logit adjustment technique for the problem of learning from biased data, which does not require previous knowledge about the dataset's biases.
Several solutions have emerged for the group robustness problem, especially when bias attribute data is available. For example, GROUPDRO [61] leverages explicit bias attributes to reduce worst-case group loss. Our method, however, reduces reliance on such data, recognizing that access to them can be impractical. Bias-unsupervised methods like Learning from Failure (LFF) [57] and Disentangled Feature Augmentation (DFA) [17] reweight the loss of the unbiased model using a constrained biased model, removing the need for bias supervision. Conversely, Just-train-twice (JT) [48] reweights misclassified samples from an initial biased training, emphasizing worst-performing group data. Similarly, He et al. [33] exemplify the same intuition in that a second network is trained on examples that cannot be predicted already using the spurious structure. Liu et al. [50] proposed a bias-unsupervised logit adjustment technique (LC) also based on co-training a biased network. Utilizing domain knowledge, Clark et al. [18] describe a bias-supervised logit adjustment approach to debiasing Visual Question Answering (VQA) models by incorporating a bias proxy which is trained exclusively on question data. These methods only generalize to seen attribute groups during training and require bias knowledge during validation for optimal performance. We tackle these issues by employing SSL to pre-train a network, deriving a bias proxy for debiased model training and validation, as well as using it as initialization.
In order to derive bias-unsupervised solutions, literature has proposed to train reference models as proxies for the missing bias labels. Creager et al. [20] optimize a reference model for group assignments which maximally violate a relaxation of the Environment Invariance Criterion. Chen et al. [16] seek to establish conditional independence between the predictions of the proxy model and the target variable given the inferred groups. Our work follows more closely the approach of [48], in which a biased network is simply trained with ERM. As we demonstrate at Figure 6 of Appendix F, by utilizing a frozen backbone pretrained with SSL, our approach improves on the sensitivity to the number of training steps for the bias proxy.
Bardenhagen et al. [9] suggested a validation scheme dependent on early stopping of bias proxy network training. Our method, using SSL pretraining, avoids this by treating pretraining steps as tunable hyperparameters, maintaining performance of alternatives which were otherwise tuned with bias information. Chen et al. [16] perform experiments using a methodology dubbed as Training Environments Validation (TEV). Similar to us, TEV validates models based on inferred groups from training, however the methodology is unfortunately not well documented in the literature, making its reproducibility difficult. As we show in the ablation study of Figure 4, implementation details can make a large difference in the quality of the criterion. Finally, MASKTUNE [3] eliminates spurious shortcuts by masking input data during a secondary training phase. Despite its resilience in performance without using a bias-unsupervised o.o.d. model selection criterion, the need for a reliable validation strategy for group robustness remains.
We explored group robust classification in synthetic and real tasks, proposing a generalization task with unseen attribute combinations. Current robust classification methods struggle in this setting, motivating our SSL-based logit adjustment approach. Importantly, we introduce a methodology for training and validating robust models without group labels. Empirical evaluations on five datasets show our method outperforms existing fully bias-unsupervised approaches and rivals those using bias annotations during validation. In terms of limitations and broader impact of our contributions, as machine learning systems handle high-stakes applications, ensuring robustness to underrepresented samples is crucial. Our work reduces reliance on known data biases, but existing benchmarks differ from real-life scenarios with unknown biased attribute combinations. To bridge this gap, we proposed a synthetic benchmark and encourage further research on real data, revealing more system limitations.
[76] C. Zhao, J. Mangat, S. Koujalgi, A. Squicciarini, and C. Caragea. Privacayalert: A dataset for image privacy prediction. In Proceedings of the International AAAI Conference on Web and Social Media, 2022.
[77] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 2017.
A Bias-Supervised Training with Logit Adjustment
The goal of this Appendix section is to develop a bias-supervised approach for maximizing the group-balanced accuracy (Equation (3)), which is going to serve as an initial point in the development of our entirely bias-unsupervised methodology in Section 3.
Under the problem formulation we introduced in Section 2, the metric that we would ultimately like to maximize given a scoring function  $f_{Y|X} : \mathcal{X} \to \mathbb{R}^K$  is  $\mathrm{Acc}_{Y|X}(f_{Y|X}; p_{\mathrm{test}})$ , the top-1 accuracy under the test distribution  $p_{\mathrm{test}}$ . Simple algebraic manipulations reveal that this test accuracy corresponds to the group-balanced accuracy:
$$
\begin{array}{l} \operatorname {A c c} _ {Y \mid X} \left(f _ {Y \mid X}; p _ {\text {t e s t}}\right) := \underset {x, y \sim p _ {\text {t e s t}}} {\mathbb {E}} \mathbb {1} \left(y = \underset {y ^ {\prime}} {\operatorname {a r g m a x}} f _ {Y \mid X} (x) _ {y ^ {\prime}}\right) \tag {8} \\ = \underset { \begin{array}{c} y, z \sim p _ {\text {t e s t}} \\ x \sim p _ {\text {t e s t}} (\cdot | y, z) \end{array} } {\mathbb {E}} \mathbb {1} \left(y = \operatorname * {a r g m a x} _ {y ^ {\prime}} f _ {Y | X} (x) _ {y ^ {\prime}}\right) \\ = \frac{1}{KL}\sum_{(y,z)\in [K]\times [L]}\mathbb{E}_{x\sim p_{\text{data}}(\cdot |y,z)}\mathbb{1}\big(y = \operatorname *{argmax}_{y^{\prime}}f_{Y|X}(x)_{y^{\prime}}\big). \\ \end{array}
$$
A related quantity in the literature is the class-balanced top-1 accuracy, which is used in class-imbalanced classification problems [11, 53]. We can make the connection by imagining  $(y,z)$  as a multi-label classification target in a multi-label, but class-imbalanced, problem. The multi-label class-balanced accuracy of a multi-label scoring function  $f_{Y,Z|X}:\mathcal{X}\to \mathbb{R}^K\times \mathbb{R}^L$  would then be
$$
\operatorname {A c c} _ {Y, Z \mid X} \left(f _ {Y, Z \mid X}; p _ {\text {t e s t}}\right) = \frac {1}{K L} \sum_ {(y, z) \in [ K ] \times [ L ]} \mathbb {E} _ {x \sim p _ {\text {d a t a}} (\cdot | y, z)} \mathbb {1} \left(\left(y, z\right) = \underset {y ^ {\prime}, z ^ {\prime}} {\arg \max } f _ {Y, Z \mid X} (x) _ {y ^ {\prime}, z ^ {\prime}}\right). \tag {9}
$$
We will set towards building a multi-label classifier which maximizes multi-label class-balanced accuracy and we will finally show that its part which only predicts  $Y$  from  $X$  can be shown to maximize a lower bound to the group-balanced accuracy.
We assume that  $\mathcal{D}$  and  $\mathcal{D}_{\mathrm{valid}}$  provide us with tuples of observations  $(x,y,z)$  distributed according to  $p_{\mathrm{data}}$ . In this case, results in the class-imbalance literature [53, 44, 19] state that the Bayes-optimal scoring function  $f_{Y,Z|X}^{*}$  for maximizing the class-balanced criterion (9) is given by
$$
\underset {y, z} {\operatorname {a r g m a x}} f _ {Y, Z \mid X} ^ {*} (x) _ {y, z} = \underset {y, z} {\operatorname {a r g m a x}} p _ {\text {t e s t}} (x \mid y, z). \tag {10}
$$
Note that under the invariance of mechanism assumption (2) we have that  $p_{\mathrm{test}}(x|y,z) = p_{\mathrm{data}}(x|y,z)$ . This allows us to compute the mechanism conditional probability in  $p_{\mathrm{data}}$  distribution terms
$$
p _ {\text {t e s t}} (x \mid y, z) = \frac {p _ {\text {d a t a}} (y , z \mid x)}{p _ {\text {d a t a}} (y , z)} p _ {\text {d a t a}} (x) \tag {11}
$$