SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Christos Tsurigotis*
Université de Montréal, Mila, ServiceNow Research
Apple MLR
Aaron Courville†
Université de Montréal, Mila, CIFAR CAI Chair
Empirical risk minimization (ERM) is sensitive to spurious correlations in the training data, which poses a significant risk when deploying systems trained under this paradigm in high-stake applications. While the existing literature focuses on maximizing group-balanced or worst-group accuracy, estimating these accuracies is hindered by costly bias annotations. This study contends that current bias unsupervised approaches to group robustness continue to rely on group information to achieve optimal performance. Firstly, these methods implicitly assume that all group combinations are represented during training. To illustrate this, we introduce a systematic generalization task on the MPI3D dataset and discover that current algorithms fail to improve the ERM baseline when combinations of observed attribute values are missing. Secondly, bias labels are still crucial for effective model selection, restricting the practicality of these methods in real-world scenarios. To address these limitations, we propose a revised methodology for training and validating debiased models in an entirely bias-unsupervised manner. We achieve this by employing pretrained self-supervised models to reliably extract bias information, which enables the integration of a logit adjustment training loss with our validation criterion. Our empirical analysis on synthetic and real-world tasks provides evidence that our approach overcomes the identified challenges and consistently enhances robust accuracy, attaining performance which is competitive with or outperforms that of state-of-the-art methods, which, conversely, rely on bias labels for validation.
Supervised learning algorithms typically rely on the empirical risk minimization (ERM) paradigm – minimizing the average loss on a training set. The ERM paradigm operates under the assumption that the training data is a representative sample of the true data distribution [67]. Consequently, models that achieve low expected loss can still be unfair when the model is tasked to predict outcomes for underrepresented groups [38, 10, 66], and prone to rely on spurious correlations between target annotations and generative attributes of the training data [32, 62] that do not hold up under more general testing conditions. As automated predictors are deployed in failure-critical applications [24] or interact with society, where fairness must be guaranteed [13, 36], robustness and fairness become requirements that standard learning strategies do not satisfy.
*Work done during internship at ServiceNow Research. Author correspondence at tsirigoc@mila.quebec.†This work was also funded, in part, from A. Courville's Sony Research Award. Courville also acknowledges support from his Canada Research Chair.
These limitations have motivated the search for alternatives that perform uniformly across different data subgroups [69, 61] or, equivalently, that rely less on spurious features [2] or "shortcuts" [28]. Part of solving this problem includes making the correct assumptions about data, which should consider spurious statistical correlations due to bias in its generative process. A popular example of this issue is the cow vs. camel classification problem [2].
As one might expect, pictures of cows very often contain a grass background, while camels are usually depicted in a desert. As such, a binary classifier that predicts whether there is grass in the background of an image could achieve a high prediction accuracy on top of natural images of the two animals. However, such a classifier would fail whenever the background changes from the typical occurrences.
While progress has been made, existing methods to improve robustness on the least frequent attribute combinations (e.g., cow on sand), require knowledge
Figure 1: Unsupervised Logit Adjustment (uLA). We train a linear classifier on top of an SSL pre-trained model to obtain biased predictions. These predictions are then leveraged to train a debiased model. No bias information is used during training or cross-validation.
about the bias attributes (bias-supervision) either during training [6, 61], or validation [57, 48, 50]. This limits their applicability in practical scenarios where bias annotations could be too costly or impossible to obtain. Even in cases where they can be obtained somewhat efficiently, human annotators might be biased themselves or sensitive annotations may not be readily available due to privacy concerns [76].
Moreover, access to group information is rarely satisfiable on natural data, where a curse of generative dimensionality implies that given the high number of attributes controlling the generative process, no realistic finite sample can cover all possible - exponential in number - combinations of such attributes. In fact, many group robustness algorithms that rely on data rebalancing or loss reweighting techniques [61, 48] assume that all group combinations are present during training. However, this condition is not satisfied in systematic generalization tasks, where certain data subgroups that are present in test data, are absent from training data. This motivates us to study the extent that training algorithms can successfully operate using group information implicitly present in the training dataset.
Specifically, in the case of these challenging systematically split datasets, we study the generalization of models on unseen combinations of attributes that were independently observed during training. This combinatorial type of generalization draws its name from the cognitive property of systematicity [26]. In deep learning, this question was introduced for the first time by Lake and Baroni [46], who studied the systematicity of RNN models in sequence prediction tasks. Here, we raise the same question in classification. To do so, we introduce a benchmark consisting of systematic splits from the MPI3D (thereby named sMPI3D) image dataset [29]. In particular, we use the 'real' split of MPI3D which consists of photographs of a robotic arm that has a colored rigid object attached to its end effector. The images are captured in a way that controls their generative attributes, such as the shape of the rigid object or the position of the robotic arm. We use it to ask the following question: given a 'shape' classifier that has been trained on objects of all possible shapes and colors but only a subset of their combinations (e.g., red cubes and blue spheres), how would it perform for a new color-shape combination (e.g., blue cubes)? In this example, the 'color' attribute plays the role of a bias, obstructing view of all possible colors that cubes could have. In Appendix D.1 we describe in more detail the construction of the sMPI3D task, while in Figure 2d we illustrate an example of a systematic split.
As discussed above, existing state-of-the-art methods suffer from two limitations. The first one, as demonstrated by Asgari et al. [3, Table 1], is that the robust accuracy of many recent training algorithms, which do not demand any bias annotations during training, degrades severely when there is no access to bias labels during model selection. This highlights that robust algorithms should prescribe a way of performing bias-unsupervised validation, assuming the same access to i.i.d. data
(a) CMNIST
(b) sMPI3D
(c) CELEBA
Figure 2: Left: Example tasks. Circled in dashed lines are samples from training split, exhibiting statistically major attribute groups. Those outside are example test samples, where all groups are equally considered. A classifier trained on a biased training set may misclassify a bias-conflicting test sample - recognizing a 'red three' as 'zero' (2a) or a 'blue sphere' as a 'cone' (2b) - or be unfair to the sensitive gender attribute when tasked to classify a facial attribute, e.g. hair color (2c). Right: An example of a systematic split.  $C$  is the number of color values per shape. 'Crosses' represent groups used to sample the training and validation splits, while 'circles' are entirely out-of-distribution.
(d) A  $C = 4$  split of sMPI3D
resources as training. The second, revealed by our study on sMPI3D (Section 4.2), is that in most cases these methods fail to improve over the ERM baseline, which fails to systematically generalize.
To address these issues, our approach, summarized in Figure 1, is based on pretraining a base encoder using self-supervised learning [7] to extract a proxy for the missing bias labels and to provide an initialization point for finetuning a debiased model. Pretraining a proxy for the bias variable enables us to use this network for two purposes: first, to train a group-robust network and, second, to define a validation criterion for robust model selection. The debiasing training algorithm is based on the logit adjustment paradigm [54]. Our entirely bias unsupervised methodology to group robustness using logit adjustment, which we call ULA, is able to compete with or outperform state-of-the-art counterparts, that otherwise utilize bias labels during model selection, in synthetic and real-world benchmarks. At the same time, it is the only method to consistently offer improvements over the ERM baseline in the sMPI3D systematic generalization task; thus effectively tackling the identified challenges about explicit or implicit use of group information.
Problem Formulation. Consider a multi-class classification task, where  $X \subset \mathcal{X}$  is the input variable and  $Y \subset \mathcal{Y}$ , the categorical target variable with  $|\mathcal{Y}| = K$  classes. We have access to a dataset of observations  $\mathcal{D} := \{(x_n, y_n)\}_{n=1}^N$  sampled i.i.d. from an underlying data distribution  $p_{\mathrm{data}}(X, Y)$  over  $\mathcal{X} \times \mathcal{Y}$ . The setting above may become problematic once we consider that the deployment data  $\mathcal{D}_{\mathrm{test}}$  are sampled from a different testing distribution:  $p_{\mathrm{test}} \neq p_{\mathrm{data}}$ . In other words, we assume that there are two data generating processes; one which generates development data ( $\mathcal{D}$  and  $\mathcal{D}_{\mathrm{valid}}$ , where  $\mathcal{D}_{\mathrm{valid}}$  is used for validation) according to  $p_{\mathrm{data}}$ , and one which generates deployment data ( $\mathcal{D}_{\mathrm{test}}$ ) according to  $p_{\mathrm{test}}$ .
In further detail, we focus on a particular transfer learning problem from  $p_{\mathrm{data}}$  to  $p_{\mathrm{test}}$ , which is due to a distribution shift in attributes which participate in the generative process of the input variable  $X$ . Our study considers anti-causal prediction tasks for which the target variable  $Y$  is one of the generative attributes of  $X$ , and  $Z \subset \mathcal{Z}$  another (possibly unobserved) categorical generative attribute with  $|\mathcal{Z}| = L$  classes.  $Z$  is marginally independent to  $Y$  under  $p_{\mathrm{test}}$ , but it might not be under  $p_{\mathrm{data}}$ . For this reason, we say that the variables  $Y$  and  $Z$  are spuriously correlated in training and validation data, and  $Z$  will also be referred to as the bias attribute. Under this setting, a group  $(y,z) \in [K] \times [L]$  is defined as a combination of target and bias attribute values, and group robustness can be formulated as a transfer learning problem from  $p_{\mathrm{data}}$  to  $p_{\mathrm{test}}$  under the following two assumptions relating the two joint distributions over  $X$ ,  $Y$  and  $Z$ :
$$
p _ {\text {t e s t}} (y, z) \propto \mathbb {1} _ {\operatorname {S u p p} p _ {\text {d a t a}} (Y) \times \operatorname {S u p p} p _ {\text {d a t a}} (Z)} (y, z), \tag {1}
$$
$$
\text {a n d} p _ {\text {t e s t}} (x \mid y, z) = p _ {\text {d a t a}} (x \mid y, z), \tag {2}
$$
where  $\mathbb{1}_S(s) = 1$  if  $s\in S$  else 0, the characteristic function of a set  $S$ , and  $\mathrm{Supp}p$  denotes the support set of a distribution  $p$ . Relation 1 asserts that, during test time, the target and bias variables
are distributed uniformly over the product of their respective marginal supports under  $p_{\mathrm{data}}$ . In other words, all combinations of observed attributes values are considered equally. This also implies that  $Y$  and  $Z$  are marginally independent under  $p_{\mathrm{test}}^3$ . On the other hand, relation 2 assumes the invariance of mechanism, which is an assumption typically found in disentangled causal process literature [64]. These two assumptions underlie the group robustness literature, in that they are equivalent to the evaluation of classifiers under a popular robust performance criterion which is described below.
Performance Criteria. Let  $\hat{y}(x; f) \coloneqq \operatorname{argmax}_y f(x)_y$  be the predictions of a scoring function  $f: \mathcal{X} \to \mathbb{R}^K$ . The accuracy of  $f$  under  $p_{\mathrm{test}}$  corresponds to the group-balanced accuracy
$$
\operatorname {A c c} _ {Y \mid X} (f; p _ {\text {t e s t}}) = \underset { \begin{array}{c} y, z \sim p _ {\text {t e s t}} \\ x \sim p _ {\text {t e s t}} (\cdot | y, z) \end{array} } {\mathbb {E}} \mathbb {1} \left(y = \hat {y} (x; f)\right) = \frac {1}{K L} \sum_ {y, z} \underset {x \sim p _ {\text {d a t a}} (\cdot | y, z)} {\mathbb {E}} \mathbb {1} \left(y = \hat {y} (x; f)\right), \tag {3}
$$
which is a frequently used performance metric in literature [61, 48, 50]. Essentially, this performance criterion first separately computes the accuracy for samples  $x$  of each group  $(y,z)$  and then averages them. The averaging operation per individual group accuracy directly stems from the uniformity assumption in  $p_{\mathrm{test}}$  (rel. 1) and implements a group fairness notion: we care equally about performing well in all groups. Another popular group robustness criterion is the worst-group accuracy [61] which substitutes the average accuracy over individual groups with the minimum (worst) accuracy.
As described in Section 5, there exist different approaches to improve group fairness [61, 48, 50] that depend on knowing biases for cross-validation. Here we focus on logit adjustment techniques like the one proposed by Liu et al. [50], which co-train two models where one corrects the biases of the other. In the following sections, we show how, by decoupling the training of the biased model from the bias-corrected model, it is possible to obtain a proxy criterion that can be used for cross-validation without knowing the bias attribute.
Logit Adjustment was originally developed as a technique for supervised learning under class-imbalanced [25, 53] or long-tailed [54] data. For clarity of presentation, we will assume for the moment that we have access to bias labels. We describe sLA, a bias-supervised training process with logit adjustment. Let  $h_{\theta}:\mathcal{X}\to \mathbb{R}^{|\mathcal{Y}|}$  be the model that we want to train for group robustness, such as a neural network parameterized by  $\theta$  implementing a function from samples in  $\mathcal{X}$  to the unnormalized logit space of  $Y$ . We then train by minimizing the average cross-entropy loss for the following logit adjusted model
$$
p _ {\theta} (y | x, z) \propto \exp \left(h _ {\theta} (x) _ {y} + \log \hat {p} _ {\text {d a t a}} (y | z)\right). \tag {4}
$$
We estimate the conditional  $\hat{p}_{\mathrm{data}}(y|z)$  directly from the available training data, for example via empirical frequencies of the finite number of groups  $(y,z)\in [K]\times [L]$ . Finally, during inference, we cancel out the contribution of the logit bias term  $\hat{p}_{\mathrm{data}}(y|z)$  and predict only according to the optimized neural network  $h_{\theta^{*}}$ .
Note that, since  $Y$  is a categorical variable,  $\log \hat{p}_{\mathrm{data}}(y|z)$  takes non-positive values. For this reason, we can intuitively interpret logit adjustment as a soft masking operation for outputs of  $h_{\theta}$  which are unlikely in the training data when we have observed  $z$ . In this way, we account for the dependency of  $Y$  to  $Z$  which spuriously exists in the training distribution. By fitting the cross-entropy objective under logit adjustment, the network  $h_{\theta}$  has to model the remaining relations for  $Y \mid X$  that are not spurious since those are already accounted for. An expressive enough model class in Equation (4) can achieve this, assuming further that the likelihood ratio  $\frac{p_{\mathrm{data}}(x|y,z)}{p_{\mathrm{data}}(x|z)} = \frac{p_{\mathrm{data}}(y|x,z)}{p_{\mathrm{data}}(y|z)}$  is independent of  $z$ . In appendix A, we derive sLA from first principles, bridging the gap between the well-studied application of this technique to the class-imbalance problem with its application to group robustness. Moreover, we demonstrate that sLA is a well-justified procedure by proving the following proposition.
Proposition 2.1 (SLA optimizes the group-balanced accuracy). Under the assumption that the hypothesis class  $p_{\theta}(y|x,z)$  (eq. 4) contains  $p_{data}(y|x,z)$ , the minimizer network  $h_{\theta^*}$  of the cross-entropy loss maximizes a lower bound to the group-balanced accuracy  $\mathrm{Acc}_{Y|X}(h_{\theta^*};p_{test})$ .
Systematic Generalization. We expect sLA models to systematically generalize, as they optimize for the transfer learning problem defined by Equations (1) and (2). As Figure 2d suggests, systematic generalization is just an extreme case of such problems, for which samples from some combinations
3Generally, the variables, which are uniform over a support which factorizes into a cartesian product of their marginal supports, are independent:  $c\mathbb{1}_{\mathcal{X}\times \mathcal{Y}}(x,y) = c\mathbb{1}(x,y\in \mathcal{X}\times \mathcal{Y}) = c\mathbb{1}(x\in \mathcal{X})\mathbb{1}(y\in \mathcal{Y}).$
have  $p_{\mathrm{data}}(y,z) = 0$  during training, however they populate the test set with  $p_{\mathrm{test}}(y,z) > 0$ . As a counterexample, the setting by Ahmed et al. [1] does not correspond to a systematic generalization task, as we define it. In what they refer to as systematic splits of colored MNIST, all possible combinations of color and digit are exposed to the model during its training, while in our sMPI3D some color and shape combinations are only revealed during test-time. This detail makes our setting significantly more challenging, as the conditions for applying importance sampling are not met, as Supp  $p_{\mathrm{data}}(Y,Z) \ngeq \operatorname{Supp} p_{\mathrm{test}}(Y,Z)$  [4, Chapter 5]. This means that a simple re-weighting of the per-sample loss with  $\frac{1}{p_{\mathrm{data}}(y,z)}$ , in order to estimate  $\mathbb{E}_{p_{\mathrm{test}}}[l_{\mathrm{ce}}(y,h_{\theta}(x))]$ , is not appropriate for the task.
Self-Supervised Learning (SSL) refers to a collection of methods [7] in unsupervised representation learning where unlabeled data provides the supervision by defining tasks where a model is asked to predict some part of the input from other parts, for example by contrasting independently transformed versions of data samples [14, 35, 15]. Methods like SimCLR [14], MoCo [35], BYOL [30] and Barlow Twins [73] provide solutions for low-data generalization, robustness, as well as transferability of learnt representations for image classification. Although self-supervised learning methods are generally more robust to distribution shifts than purely supervised methods [49], they can still be affected by significant shifts, for example in long-tailed data learning [5, 63]. In this work, we explore the limits and utility of SSL in group robust classification. An extended preliminary discussion about self-supervised algorithms can be found in Appendix B.
3 ULA: Bias-unsupervised Logit Adjustment
Here, we introduce bias-unsupervised logit adjustment (ULA), a logit correction approach which improves on the work of Liu et al. [50] by removing dependency on explicit bias annotations both during training and validation. For this reason, we create a proxy variable for the bias via the predictions of a pretrained network using SSL. The advantage of pretraining, over co-training a bias network [57, 17, 50], is two-fold: First, we can reuse the fixed bias proxy network to define a validation criterion for group robustness. This enables us to perform hyperparameter search, but also informs us about when to stop training the debiased model, which is critical for optimal performance [48]. Second, recent literature [39, 43] has demonstrated, in the case of training with bias-supervised data, that a linear classifier on top of pretrained base models provides with substantial group-robustness improvements. Here, we initialize the debiased model from the pretrained base model and finetune it using logit adjustment. Figure 1 provides a summary of our approach and its pseudocode can be found in Appendix C. Further training details are discussed in the following.
3.1 Bias-unsupervised Training
Biased network: pretrain with SSL. We start by training a base model,  $f_{\mathrm{base}}$ , using an SSL method on the unlabeled data of the training set  $\mathcal{D}$ . We decide on the hyperparameters of the SSL algorithm (learning rate, weight decay, temperature, augmentations and potentially others) by maximizing the i.i.d. validation accuracy of an online linear classifier, which probes the representations of the base model. Afterwards, we train a linear classifier  $g_{\phi}$  on top of a frozen  $f_{\mathrm{base}}$  and against target variable labels  $Y$  using a vanilla cross-entropy loss, in order to derive a proxy for the bias variable. Finally, we retrieve the composite neural network  $h_{\mathrm{bias}} = g_{\phi} \circ f_{\mathrm{base}}$ , and use its predictions  $\hat{y}(x; h_{\mathrm{bias}})$  as a proxy for the missing bias variable observations. For the same purpose, Nam et al. [57] trained a bias-extracting network by employing Generalized Cross Entropy [75], an objective which provides robustness to noisy labels. Since the goal of the proxy network is to predict the spurious attribute, bias-conflicting or minority samples can be perceived as mislabeled data points. Here, we follow these observations and leverage the representations learnt with SSL which, when composed with a low-capacity (linear) classifier, provide with a model that is more robust to label-noise [71]. In addition to deriving a bias proxy, we will also use  $f_{\mathrm{base}}$  as an initialization point in the parameter space for training the debiased model.
In this paper, we have chosen to use MoCoV2+ [15] as the SSL algorithm for the image classification tasks we consider. We make this choice since contrastive learning algorithms, like the MoCo family [35, 15], offer relatively stable training since they explicitly prevent representation collapse in their loss function (see Appendix B). However, as we demonstrate in Section 4.3, our method is not restrained to the use of a particular SSL algorithm.
Debiased network: logit adjustment. In the absence of bias labels during training, we need a substitute for the estimate  $\hat{p}_{\mathrm{data}}(Y,Z)$  in eq. 4. We use the predictions  $y_{\mathrm{bias}}$  of the bias proxy network  $h_{\mathrm{bias}}$  to that end. The resulting joint distribution between the target variable and the biased network's predictions,  $\hat{p}_{\mathrm{data}}(y,y_{\mathrm{bias}})$ , can be thought of as a soft confusion matrix of  $h_{\mathrm{bias}}$  and can be computed using the available training data with
$$
\hat {p} _ {\text {d a t a}} \left(y, y _ {\text {b i a s}}\right) = \frac {1}{| \mathcal {D} |} \sum_ {x ^ {\prime}, y ^ {\prime} \in \mathcal {D}} p _ {\text {b i a s}} \left(y _ {\text {b i a s}} \mid x ^ {\prime}\right) \mathbb {1} \left(y = y ^ {\prime}\right), \tag {5}
$$