SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} = \sum_ {j \in g _ {1} \cup g _ {2}} \beta_ {j} \sum_ {i = 1} ^ {d} w _ {j i} \Delta x _ {i} + \text {c o n s t a n t} \\ \Longrightarrow = \Theta (\| \beta \| \| \vec {w} \| _ {F} \| \Delta \mathbf {x} \|) \quad \text {C a u c h y - S c h w a r t z i n e q u a l i t y} \\ = \Theta ((\| \beta \| ^ {2} + \| \bar {w} \| _ {F} ^ {2} \|) \| \Delta \mathbf {x} \|) \\ \end{array}
$$
E Further Experiment Details
E.1 Hyperparameters.
We picked the learning rate, optimizer, weight decay, and initialization for best performance with ERM baseline on validation data, which are not further tuned for other baselines unless stated otherwise. We picked the best  $\lambda$  for Grad-Reg and CDEP from [1, 10, 100, 1000]. Additionally, we also tuned  $\beta$  (weight decay) for Grad-Reg from [1e-4, 1e-2, 1, 10]. For Avg-Ex, perturbations were drawn from 0 mean and  $\sigma^2$  variance Gaussian noise, where  $\sigma$  was chosen from [0.03, 0.3, 1, 1.5, 2]. In PGD-Ex, the worst perturbation was optimized from  $\ell_{\infty}$  norm  $\epsilon$ -ball through seven PGD iterations, where the best  $\epsilon$  is picked from the range 0.03-5. We did not see much gains when increasing PGD iterations beyond 7, Appendix F contains some results when the number of iterations is varied. In IBP-Ex, we follow the standard procedure of Gowal et al. (2018) to linearly dampen the value of  $\alpha$  from 1 to 0.5 and linearly increase the value of  $\epsilon$  from 0 to  $\epsilon_{max}$ , where  $\epsilon_{max}$  is picked from 0.01 to 2. We usually just picked the maximum possible value for  $\epsilon_{max}$  that converges. For IBP-Ex+Grad-Reg, we have the additional hyperparameter  $\lambda$  (Eqn. 4), which we found to be relatively stable and we set it to 1 for all experiments.
Relative Core Sensitivity (RCS) (Singla et al., 2022). The metric measures the relative dependence of the model on core features and is normalised such that the best value is 100. Higher value of RCS imply that the model is exploiting core features more than the spurious features.
$$
R C S = 1 0 0 \times \frac {a c c ^ {(C)} - a c c ^ {(S)}}{2 \min  (\bar {a} , 1 - \bar {a})}, \text {w h e r e} \bar {a} = \frac {a c c ^ {(C)} + a c c ^ {(S)}}{2}
$$
$$
a c c ^ {C} \triangleq \frac {1}{N} \sum_ {n} \mathbb {1} \left(f \left(\mathbf {x} ^ {(n)} + \sigma (\mathbf {z} \odot \mathbf {m} ^ {(n)}); \theta\right) = y ^ {(n)}\right)
$$
$$
a c c ^ {S} \triangleq \frac {1}{N} \sum_ {n} \mathbb {1} \left(f \left(\mathbf {x} ^ {(n)} + \sigma (\mathbf {z} \odot (1 - \mathbf {m} ^ {(n)})); \theta\right) = y ^ {(n)}\right)
$$
where  $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$  and  $\sigma = 0.25$  for all experiments. The interpretation of  $acc^{(C)}$  is the accuracy when noise is added outside of core region, and  $acc^{(S)}$  is the accuracy when noise is added outside of spurious region.
E.3 Data splits
We randomly split available labelled data in to training, validation, and test sets in the ratio of (0.75, 0.1, 0.15) for ISIC, (0.65, 0.1, 0.25) for Plant (similar to Schramowski et al. (2020)) and (0.6, 0.15, 0.25) for Salient-Imagenet. We use the standard train-test splits on MNIST.
E.4 Datasets
ISIC dataset The ISIC dataset consists of 2,282 cancerous (C) and 19,372 non-cancerous (NC) skin cancer images of 299 by 299 size, each with a ground-truth diagnostic label. We follow the standard setup and dataset released by Rieger et al. (2020), which included masks with patch segmentations. In half of the NC images, there is a spurious correlation in which colorful patches are only attached next to the lesion. This group is referred to as patch non-cancerous (PNC) and the other half is referred to as not-patched non-cancerous (NPNC) Codella et al. (2019). Since trained models tend to learn easy-to-learn and useful features, they tend to take a shortcut by learning spurious features instead of understanding the desired diagnostic phenomena. Therefore, our goal is to make the model invariant to such colorful patches by providing a human specification mask indicating where they are.
decoy-MNIST dataset The MNIST dataset consists of 70,000 images of handwriting digit from 0 to 9. Each class has about 7,000 images of 28 by 28 size. We use three-fully connected layers for multi classification with 512 hidden dimension and 3 channels.
Salient-Imagenet. The six classes we considered are Rhinoceros Beetle, Dowitcher, Alaskan Tundra Wolf, Dragonfly, Gorilla, Snoek Fish.
E.5 Computational cost
Run time and memory usage Table 3 presents the computation costs, including run time and memory usage, for each method using GTX 1080 Ti. It is worth noting that IBP-Ex has significantly less run time and memory usage compared to PGD-Ex, with a 10-fold reduction in run time and a 2.5-fold reduction in memory usage. Considering that PGD-Ex and IBP-Ex have similar performance in terms of worst group accuracy, as shown in Table 5, IBP-Ex+Grad-Reg appears to be comparably effective and efficient for model modification. Additionally, the combined method IBP-Ex+Grad-Reg, which presents the best performance in terms of averaged and worst group accuracy compared to PGD-Ex, also has a 3-fold reduction in run time and a 2-fold reduction in memory usage compared to PGD-Ex.
Grad-Reg	PGD-Ex	IBP-Ex	IBP-Ex+Grad-Reg	PGD-Ex+Grad-Reg
×2.3	×4.9	×2.2	×3.5	×7.0
Table 3: Running time in comparison to ERM on the ISIC dataset
E.6 Network Architecture
Model architecture on the decoy-MNIST dataset
Sequential(
(0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))  
(1): ReLU()  
(2): Conv2d(32, 32, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))  
(3): ReLU()  
(4): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))  
(5): ReLU()  
(6): Conv2d(64, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))  
(7): ReLU()  
(8): Flatten(start_dim=1, end_dim=-1)  
(9): Linear(in_features=200704, out_features=1024, bias=True)  
(10): ReLU()  
(11): Linear(in_features=1024, out_features=1024, bias=True)  
(12): ReLU()  
(13): Linear(in_features=1024, out_features=2, bias=True)
Model architecture on the ISIC dataset
Sequential(
(0): Flatten(start_dim=1, end_dim=-1)  
(1): Linear(in_features=2352, out_features=512, bias=True)  
(2): ReLU()  
(3): Linear(in_features=512, out_features=512, bias=True)  
(4): ReLU()  
(5): Linear(in_features=512, out_features=512, bias=True)  
(6): ReLU()  
(7): Linear(in_features=512, out_features=10, bias=True)
Model architecture on the Plant phenotyping dataset
Sequential(
(0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))  
(1): ReLU()  
(2): Conv2d(32, 32, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))  
(3): ReLU()  
(4): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))  
(5): ReLU()  
(6): Conv2d(64, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
(7):ReLU()
(8): Flatten(start_dim=1, end_dim=-1)
(9): Linear(in_features=200704, out_features=1024, bias=True)
(10):ReLU()
(11): Linear(in_features=1024, out_features=1024, bias=True)
(12):ReLU()
(13): Linear(in_features=1024, out_features=2, bias=True)
）
F.1 Standard deviations
We repeated all our experiments on Decoy-MNIST, Plant and ISIC dataset three times and report the mean and standard deviation in Table 4. Similarly, we report in Table 5, the standard deviations for the corresponding Table 2 of the main paper.
Dataset→	Decoy-MNIST	Plant	ISIC
Method↓	Avg Acc	Wg Acc	Avg Acc	Wg Acc	Avg Acc	Wg Acc
ERM	15.1 ± 1.3	10.5 ± 5.4	71.3 ± 2.5	54.8 ± 1.3	77.3 ± 2.4	55.9 ± 2.3
G-DRO	64.1 ± 0.1	28.1 ± 0.1	74.2 ± 5.8	58.0 ± 4.6	66.6 ± 5.4	58.5 ± 10.7
Grad-Reg	72.5 ± 1.7	46.2 ± 1.1	72.4 ± 1.3	68.2 ± 1.4	76.4 ± 2.4	60.2 ± 7.4
CDEP	14.5 ± 1.8	10.0 ± 0.7	67.9 ± 10.3	54.2 ± 24.7	73.4 ± 1.0	60.9 ± 3.0
Avg-Ex	29.5 ± 0.3	19.5 ± 1.4	76.3 ± 0.3	64.5 ± 0.3	77.1 ± 2.1	55.2 ± 6.6
PGD-Ex	67.6 ± 1.6	51.4 ± 0.3	79.8 ± 0.3	78.5 ± 0.3	78.7 ± 0.5	64.4 ± 4.3
IBP-Ex	68.1 ± 2.2	47.6 ± 2.0	76.6 ± 3.5	73.8 ± 1.7	75.1 ± 1.2	64.2 ± 1.2
P+G	96.9 ± 0.3	95.8 ± 0.4	79.4 ± 0.5	76.7 ± 2.8	79.6 ± 0.5	67.5 ± 1.1
I+G	96.9 ± 0.2	95.0 ± 0.6	81.7 ± 0.2	80.1 ± 0.3	78.4 ± 0.5	65.2 ± 1.8
Table 4: Macro-averaged (Avg) accuracy and worst group (Wg) accuracy on (a) decoy-MNIST, (b) plant dataset, (c) ISIC dataset. Results are averaged over three runs and their standard deviation is shown after  $\pm$ . I+G is short for IBP-Ex+Grad-Reg and P+G for PGD-Ex+Grad-Reg. See text for more details.
Method	NPNC	PNC	C	Avg	Wg
ERM	55.9 ± 2.3	96.5 ± 2.4	79.6 ± 6.6	77.3 ± 2.4	55.9 ± 2.3
G-DRO	72.4 ± 4.0	63.2 ± 14.8	64.1 ± 5.6	66.6 ± 5.4	58.5 ± 10.7
Grad-Reg	67.1 ± 4.8	99.0 ± 1.0	63.2 ± 11.3	76.4 ± 2.4	60.2 ± 7.4
CDEP	72.1 ± 5.4	98.9 ± 0.7	62.2 ± 4.7	73.4 ± 1.0	60.9 ± 3.0
Avg-Ex	62.3 ± 11.7	97.8 ± 0.8	71.0 ± 16.7	77.1 ± 2.1	55.2 ± 6.6
PGD-Ex	65.4 ± 5.4	99.0 ± 0.3	71.7 ± 6.7	78.7 ± 0.5	64.4 ± 4.3
IBP-Ex	68.4 ± 3.4	98.5 ± 1.0	67.7 ± 4.8	75.1 ± 1.2	64.2 ± 1.2
P+G	69.6 ± 2.8	98.84 ± 0.6	70.4 ± 4.1	79.6 ± 0.5	67.5 ± 1.1
I+G	66.6 ± 3.1	99.6 ± 0.2	68.9 ± 4.7	78.4 ± 0.5	65.2 ± 1.8
Table 5: Macro-averaged (Avg) accuracy and worst group (Wg) accuracy on ISIC dataset. Also shown are the average precision scores for each of the three groups. All the results are averaged over three runs and their standard deviation is shown after  $\pm$ . Note that the worst group for each run can be different
F.2 Additional results on Salient-Imagenet
In Table 6, we show average accuracy and accuracy when noise (drawn from standard normal) is added to spurious or irrelevant regions (N-Acc column). We observe that for PGD-Ex + Grad-Reg, the accuracy did not diminish by much when noise is added to the spurious region.
F.3 Comparison of PGD-Ex and IBP-Ex
In Table 5, it is difficult to compare the worst group accuracy of IBP-Ex (64.2) and PGD-Ex (64.4) due to the comparably high standard deviation of PGD-Ex (4.3). Therefore, we additionally compare
Method	Accuracy	N-Acc ↑	RCS ↑
ERM	96.4	87.5	47.9
Grad-Reg	88.3	82.2	52.5
PGD-Ex	93.8	90.2	58.7
PGD-Ex + Grad-Reg	94.6	93.8	65.0
Table 6: The columns in that order are the average accuracy, accuracy when noise is added to spurious (or irrelevant) regions and RCS value for our Salient-Imagenet data setup.
the accuracy drop when colorful patches are removed from images in the PNC group in Table 7. We replace the colorful patch of the image with its mean value, making it looks like a background skin color. Note that we evaluate the robustness to concept-level perturbations rather than pixel-level perturbations, as our focus is on avoiding spurious concept features rather than robustness to adversarial attacks. Interestingly, the accuracy drops about  $17\%$  and  $37\%$  in IBP-Ex and PGD-Ex, respectively, showing that IBP-Ex is more robust to concept perturbations. This can be explained by the effectiveness of robustness methods in covering the epsilon ball with the center of each input point defined in a low-dimensional manifold annotated in the human specification mask. IBP guarantees robustness on any possible pixel combination within the epsilon ball while PGD only considers the worst case in the epsilon ball. When the inner maximization to find the PGD attack is non-convex, an inappropriate local worst case is found instead of the global one. Thus, IBP-Ex shows better robustness when spurious concepts are removed, which involves large perturbations on irrelevant parts within the defined epsilon ball. The combined method IBP-Ex+Grad-Reg, where Grad-Reg compensates for the practical limitations of the training procedure of IBP-Ex, shows about  $1\%$  higher worst group accuracy than IBP-Ex alone.
Method	PNC	PNC (Remove patch)
PGD-Ex	99.0 ± 0.3	62.2 ± 17.0
IBP-Ex	98.5 ± 1.0	81.6 ± 16.5
IBP-Ex+Grad-Reg	99.6 ± 0.2	82.5 ± 9.5
Table 7: Comparison between robustness based methods. Macro-averaged accuracy and regval loss before and after removing color patch part of images in PNC group on ISIC dataset.
F.4 Results of PGD-Ex with different epsilon and iteration number.
We experimented with different values of epsilon and iteration numbers on the ISIC and Plant phenotyping datasets. The epsilon values tested were 0.03, 0.3, 1, 3, and 5, and the iteration numbers were 7 and 25. In Figure 4, the results on the ISIC dataset showed that using an iteration of 7 with different epsilon values resulted in stable results, but using an iteration of 25 resulted in unstable worst group accuracy. However, in the Plant phenotyping dataset, we found that both average and worst group accuracy were similar regardless of the epsilon and iteration values used.
(a) PGD-Ex on the ISIC data
(b) PGD-Ex on the Plant phenotyping data
Figure 4: PGD-Ex results on the ISIC and Plant phenotyping dataset with different epsilon and iteration numbers in (a) and (b), respectively.
F.5 Out-of-distribution scenarios on the Plant data
In the main paper, we follow the dataset construction from Schramowski et al. (2020) to replace background with the average pixel value, which is obtained from train split. Here, in Table 8 we evaluated on a test set obtained by adding varying magnitude of noise to the background to test methods under out-of-distribution scenarios. We observe that robustness and regularization methods when combined led to a model that is far more robust to noise in the background, aligning with our original results on the plant dataset.
Noise (N(0,1))	Noise (N(0,10))	Noise (N(0,30))
Avg Acc	Wg Acc	Avg Acc	Wg Acc	Avg Acc	Wg Acc
ERM	59.8 ± 11.9	43.5 ± 2.0	57.4 ± 7.6	38.1 ± 4.8	55.8 ± 1.7	22.0 ± 3.7
Grad-Reg	71.6 ± 2.0	66.1 ± 1.8	68.7 ± 6.2	53.4 ± 4.3	56.1 ± 3.3	34.8 ± 1.8
PGD-Ex+Grad-Reg	69.8 ± 1.8	67.2 ± 2.1	69.5 ± 3.7	60.6 ± 4.8	67.5 ± 4.5	50.8 ± 2.4
F.6 Generality to new explanation methods: Integrated-gradient and CDEP
In Table 9, we introduced evaluation using Integrated-gradient (Sundararajan et al., 2017) based regularization and also added evaluation with PGD-Ex+CDEP that we did not originally include on Decoy-MNIST dataset.
Table 8: Out-of-distribution scenarios on the Plant data
Alg.	Avg Acc	Wg Acc
Integrated-Grad	26.7 ±1.3	17.6 ±1.2
CDEP	14.5 ±1.8	10.0 ±0.7
PGD-Ex	67.6 ±1.6	51.4 ±0.3
PGD-Ex+Integrated-Grad	80.5 ±2.1	62.1 ± 6.8
PGD-Ex+CDEP	84.8 ±0.8	64.2 ±1.6
F.7 Generality to new model architecture: Attention map-based (ViT)
Using a Visual transformer architecture (of depth 3 and width 128), we evaluated regularization using local explanations obtained using an attention map - regularization based on attention maps was used to supervise prior knowledge in Miao et al. (2022) and is called SPAN. We obtained saliency explanations on inputs using the procedure proposed in Miao et al. (2022), shown in Table 10
Table 9: PGD-Ex+Integrated-Grad and PGD-Ex+CDEP on the Decoy-MNIST data
Alg.	Avg Acc	Wg Acc
ERM	10.0 ±0.3	8.1 ±0.3
SPAN	19.0 ±0.3	8.1 ±0.3
PGD-Ex	64.6 ±4.7	37.4 ±3.5
PGD-Ex+SPAN	63.1 ±2.6	39.4 ± 2.9
Table 10: Attention Map based local explanations with ViT on the Decoy-MNIST data
F.8 Sensitivity to hyperparameters on Decoy-MNIST dataset
In Table 11, we show sensitivity to hyperparameters on Decoy-MNIST dataset. In summary, results are broadly stable with the choice of hyperparameters, and we did not extensively search for the best hyperparameters.
Decoy-MNIST	Lambda (Grad-Reg)	Eps (PGD-Ex)	Avg -Acc	Wg -Acc
PGD-Ex + Grad-Reg	1	3	96.9 ± 0.3	95.8 ± 0.4
0.1	3	96.8 ± 0.8	94.2 ± 0.2
1.5	3	95.6 ± 1.0	93.0 ± 1.0
5	3	91.6 ± 0.9	87.6 ± 2.3
0.0001	3	75.5 ± 0.9	57.2 ± 3.6
1	1	95.1 ± 2.0	91.3 ± 3.9
1	2	95.4 ± 1.8	92.0 ± 1.6
1	4	97.5 ± 0.2	95.4 ± 0.8
1	5	93.8 ± 1.6	86.3 ± 3.1
1	0.1	58.5 ± 7.9	30.0 ± 2.7
1	0.0001	59.5 ± 1.1	40.1 ± 2.0
PGD-Ex	0	3	67.6 ± 1.6	51.4 ± 0.3
0	0.0001	16.5 ± 1.1	15.4 ± 2.7
0	0.1	19.1 ± 0.7	13.6 ± 0.6
0	1	62.5 ± 1.0	40.1 ± 2.2
0	2	74.6 ± 5.6	52.8 ± 4.8
0	4	71.9 ± 8.5	58.6 ± 12.9
0	5	57.0 ± 3.1	42.6 ± 4.2
Grad-Reg	10	0	64.1 ± 0.1	28.1 ± 0.1
5	0	39.2 ± 2.2	21.5 ± 0.6
20	0	49.1 ± 2.7	33.2 ± 4.3
100	0	50.1 ± 0.4	35.2 ± 2.3
500	0	48.0 ± 0.9	36.2 ± 1.7
1000	0	49.6 ± 1.7	30.5 ± 6.1
Table 11: Sensitivity to hyperparameters on Decoy-MNIST dataset
G Discussion on poor CDEP performance
Regarding ISIC dataset discrepancy: In Table 5, CDEP demonstrates better performance in worst group accuracy compared to ERM on the ISIC dataset. However, it fails to surpass RRR, which contradicts results from previous research in Rieger et al. (2020) where CDEP was found to perform better than RRR. This discrepancy may be attributed to the fact that Rieger et al. (2020) used different metrics (F1 and AUC) and employed a pretrained VGG model to estimate the contribution of mask features, whereas in our study we used worst group accuracy and employed a four-layer CNN followed by three fully connected layers without any pretraining. We do not use a pre-trained model for CDEP in order to make a fair comparison to other methods. As a result, CDEP also fails to improve worst group accuracy over ERM on the Plant Phenotyping and Decoy-MNIST datasets. We further illustrate the interpretations of CDEP on the Plant Phenotyping dataset using Smooth Gradient in Figure 5. In comparison to the interpretations of other methods shown in Figure 3 in the main paper, CDEP appears to focus primarily on the spurious agar part instead of the main leaf part.
Regarding DecoyMNIST dataset discrepancy: Note that our Decoy-MNIST setting is inspired from decoy-mnist of CDEP (Rieger et al., 2020), but not the same. All the methods were found to be equally good on the original decoy-mnist dataset (Rieger et al., 2020), which is why we had to alter the dataset to be more challenging. A key difference is that the volume of spurious/simple features in our version of decoy-mnist dataset is much higher, making it harder to remove dependence of a model on decoy/spurious features. This explains why there is the performance gap on this dataset reported in our paper and CDEP (Rieger et al., 2020).
Figure 5: Visual heatmap of salient features for CDEP on three sample images from the train split of Plant phenotyping data. Importance score from SmoothGrad Smilkov et al. (2017) method is normalized between 0 to 1 and visualized with a threshold 0.6.