SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\ell_ {\sigma \text {- d a m p}} (\mathbf {y}, f _ {\theta}) = \ell_ {l o g} \left[ T _ {\mathbf {y}} * 1. 2 7 8 \mathbf {y} f _ {\theta} (1 - \sigma (1. 2 7 8 * \mathbf {y} f _ {\theta})) \right]
$$
The 1.278 comes in to make sure the maximum input to log-loss occurs at  $f_{\theta} = 1$ . However, due to the different temperatures  $T_{1} \neq T_{-1}$ , achieving the same margin on all samples produces lower loss on the class with the larger temperature.
2. With per-class temperatures  $T_{-1}, T_1 > 0$  the variant of  $\sigma$ -stitch is
$$
\ell_ {\sigma - \text {s t i c h}} (\mathbf {y} f _ {\theta}) = \ell_ {l o g} \left(T _ {\mathbf {y}} [ \mathbf {1} [ \mathbf {y} f _ {\theta} (\mathbf {x}) <   1 ] \times \mathbf {y} f _ {\theta} (\mathbf {x}) + \mathbf {1} [ \mathbf {y} f _ {\theta} (\mathbf {x}) > 1 ] \times (2 - \mathbf {y} f _ {\theta} (\mathbf {x})) ]\right)
$$
3. With per-class function output targets  $\gamma_{-1},\gamma_1 > 0$  the variant of MARG-LOG is
$$
\ell_ {\text {M A R G - L O G}} (\mathbf {y} f _ {\theta}) = \ell_ {\log} (\mathbf {y} f _ {\theta}) + \lambda \log (1 + | f _ {\theta} - \gamma_ {\mathbf {y}} | ^ {2}).
$$
These per-class variants are only for training; at test time, the predicted label is  $\mathrm{sign}(f_{\theta})$
Details of the vision and language experiments. We use the same datasets from Liu et al. [2021], downloaded via the scripts in the code from [Idrissi et al., 2022]; see [Idrissi et al., 2022] for sample sizes and the group proportions. For the vision datasets, we finetune a resnet50 fromImagenet-pretrained weights and for Civilcomments, we finetune a BERT model.
Figure 15: Images mis-classified by a model trained on CelebA data with equal group sizes, i.e. without a shortcut. Samples with blonde as the true label have a white strip at the bottom while samples with non-blonde as the true label have a black strip at the bottom. The figure demonstrates that many images with blonde people in the image have the non-blonde label, thus demonstrating label noise. For example, see a blonde man in the first row that is labelled non-blonde and a nonblonde lady in the third row that is labelled blonde. Yet, MARG-CTRL improves over ERM for many LR and WD combinations; see fig. 16.
Optimization details. For all methods and datasets, we tune over the following weight decay (WD) parameters:  $10^{-1}, 10^{-2}, 10^{-3}, 10^{-4}$ . For the vision datasets, we tune learning rate (LR) over  $10^{-4}, 10^{-5}$  and for CivilComments, we tune over  $10^{-5}, 10^{-6}$ . For CivilComments, we use the AdamW optimizer while for the vision datasets, we use the Adam optimizer; these are the standard optimizers for their respective tasks [Puli et al., 2022, Gulrajani and Lopez-Paz, 2021]. We use a batch size of 128 for both CelebA and Waterbirds, and train for 20 and 100 epochs respectively. For CivilComments we train for 10 epochs with a batch size of 16.
Per-method Hyperparameters. Like in [Pezeshki et al., 2021], the per-class temperatures  $T_{-1}, T_1$  for  $\sigma$ -damp and  $\sigma$ -stitch, and the function output targets  $\gamma_{-1}, \gamma_1$  for MARG-LOG are hyperparameters that we tune using the worst-group accuracy or label-balanced average accuracy computed on the validation dataset, averaged over 2 seeds.
1. For  $\sigma$ -stitch, we select from  $T_{-1} \in \{1, 2\}$  and  $T_1 \in \{2, 4, 8, 12\}$  such that  $T_1 > T_{-1}$ .
2. For  $\sigma$ -damp, we search over  $T_{-1} \in \{1, 2\}$  and  $T_1 \in \{2, 4\}$  such that  $T_1 > T_{-1}$ .
3. For SD and MARG-LOG, we search over  $\gamma_{-1} \in \{-1,0,1\}$  and  $\gamma_1 \in \{1,2,2.5,3\}$  for the image datasets and  $\gamma_1 \in \{1,2\}$  for the text dataset, and the penalty coefficient is set to be  $\lambda = 0.1$
4. For JTT, we search over the following parameters: the number of epochs  $T \in \{1,2\}$  for CelebA and Civilcomments and  $T \in \{10,20,30\}$  for Waterbirds, and the upweighting constant  $\lambda \in \{20,50,100\}$  for the vision datasets and  $\lambda \in \{4,5,6\}$  for Civilcomments.
5. For CNC, we search over the same hyperparameter as [Zhang et al., 2022]: the temperature in  $\tau \in \{0.05, 0.1\}$ , the contrastive weight  $\lambda \in \{0.5, 0.75\}$ , and the gradient accumulation steps  $s \in \{32, 64\}$ . For the language task in Civilcomments, we also try one additional  $s = 128$ .
B.7 MARG-CTRL improves over default-ERM on CelebA even without the stable feature being perfect.
CelebA is a perception task in that the stable feature is the color of the hair in the image. But unlike the synthetic experiments, MARG-CTRL does not achieve a  $100\%$  test accuracy on CelebA. We investigated this and found that CelebA in fact has some label noise.
We trained a model via the MARG-CTRL method  $\sigma$ -damp on CelebA data with no shortcut; this data is constructed by subsampling the groups to all equal size, (5000 samples). This achieves a test worst-group accuracy of  $89\%$ . We visualized the images that were misclassified by this model and found that many images with blond-haired people were classified as having non-blonde hair. Figure 15 shows 56 misclassified images where samples with blonde as the true label have a white strip at the bottom while samples with non-blonde as the true label have a black strip at the bottom. The figure shows that images with blonde people can have the non-blonde label, thus demonstrating label noise. Thus, MARG-CTRL improves over ERM even on datasets like CelebA where the stable features do not determine the label.
Figure 16: Test worst-group accuracy on CelebA of default-ERM and MARG-CTRL for different values of LR and WD. Default-ERM's performance changes more with LR and WD than MARG-CTRL, which shows that default-ERM is more sensitive than MARG-CTRL. Only 2 combinations of LR and WD improve ERM beyond a test worst-group accuracy of  $60\%$ , while every MARG-CTRL method achieves more than  $70\%$  test worst-group accuracy for every combination of LR and WD.
B.8 Sensitivity of ERM and MARG-CTRL to varying LR and WD
In fig. 16, we compare the test worst-group accuracy of default-ERM and MARG-CTRL on CelebA, for different values of LR and WD. There are 8 combinations of LR and WD for which ERM is run. For each combination of LR and WD, the hyperparameters of the MARG-CTRL method (values of  $\lambda, T, v$ ) are tuned using validation group annotations, and the test worst-group accuracy corresponds to the best method hyperparameters. Default-ERM's performance changes more with LR and WD than MARG-CTRL, which shows that default-ERM is more sensitive than MARG-CTRL. Only 2 combinations of LR and WD improve ERM beyond a test worst-group accuracy of  $60\%$ , while every MARG-CTRL method achieves more than  $70\%$  test worst-group accuracy for every combination of LR and WD.