SlowGuess/NeurIPS_2023_OCR huggingface dataset:
that given the inherent differences between the two, making a fair comparison is challenging. However, we find that our setup can "boost" not only ERM but also other robust base learners i.e. if we use these robust optimization methods to find our base learners under analogous RAI constraints, we are able to further enhance the robust performance of these algorithms. The results are shown in Table 4. We hypothesize that individually robust base learners are able to help the ensemble generalize well, allowing our approach to further optimize through ensembles.
I.4 Synthetic Datasets
In this section, we use synthetic datasets to illustrate how our RAI algorithms converge, and how different constraints on  $W$  translate into performance across various responsible metrics. We use the following distributions to construct the datasets, and use class labels as the group labels.
- Dataset-I:  $P(X|Y = 0) = \mathcal{N}((0,0),\pmb {I})$ ,  $P(X|Y = 1) = \frac{1}{3}\mathcal{N}((-3,1),\pmb {I})) + \frac{1}{3}\mathcal{N}((3,0),\pmb {I})) + \frac{1}{3}\mathcal{N}((0, - 3),\pmb {I}))$ ,  $P(Y = 0) = 0.7$ ,  $P(Y = 1) = 0.3$ .
- **Dataset-II:**  $P(X|Y = 0) = \frac{5}{12}\mathcal{N}((-2, -2), 0.5\pmb{I}) + \frac{2}{12}\mathcal{N}((-2, -2), 0.5\pmb{I}) + \frac{5}{12}\mathcal{N}((2, 2), 0.5\pmb{I}), P(X|Y = 1) = \frac{2}{5}\mathcal{N}((-3, 0), 0.3\pmb{I})) + \frac{3}{5}\mathcal{N}((3, 0), 0.3\pmb{I})), P(Y = 0) = 0.7, P(Y = 1) = 0.3$
We sample 1000 points each for both training and testing from both distributions. Note that these datasets deliberately exhibit: 1. Class imbalance (particularly in Dataset-I) 2. Multiple minority sub-populations (within and between classes) 3. Varying noise levels in the sub-populations (predominantly in Dataset-II). Such characteristics are frequently encountered in real-world scenarios and demand responsible classifiers.
Models For base learners, we use linear classifiers for Dataset-I and neural network classifiers with a single hidden layer of size 4 and ReLU activations for Dataset-II. We find that base learners can be models with varying complexity.
Hyperparameters Due to the limited size of the datasets, we forgo the warm-up stage. At every round, we run 1000 iterations with a mini-batch size of 32. We run  $\alpha$ -LPBoost with the default value  $\eta = 1$ . For  $\alpha$ -CVaR experiments, we take  $\alpha = 0.7$  across all experiments. For lower values of  $\alpha$ , we observe similar results and comparisons, albeit with a substantial reduction in average metrics. Consequently, we opt for conservative values to standardize average performance across all models and subsequently compare worst-case performance in responsible settings.
I.4.1 Results and Discussion
- **Domain Oblivious (DO)** To begin, we run ERM, AdaBoost,  $\alpha$ -LPBoost, and RAI games on Dataset-I. For RAI-GA and RAI-FW games, we use  $\alpha$ -CVaR uncertainty set as  $W$ . Given the class imbalance,  $Y = 0$  and  $Y = 1$  represent good candidates for subpopulations of interest. The results are reported in Figure 1. We immediately observe the following:
- Both proposed methods RAI-FW and RAI-GA effectively decrease the objective value and achieve lower worst-class classification loss, as compared to both ERM and AdaBoost.
- They closely follow the  $\alpha$ -LPBoost iterates. Intuitively, our quasi-boosting updates resemble  $\alpha$ -LPBoost for the CVaR objective, and that is reflected in similar objective values.
- Domain Aware (DA) For this setting, we run ERM, AdaBoost, Online GDRO, RAI-GA, and RAI-FW on Dataset-II. We use Group DRO over the five gaussian groups as the uncertainty set  $W$ . Although Dataset-II was selected due to the presence of more pronounced subpopulation behavior, we get similar results for Dataset-I as well. The results are reported in Figure 2 and Table 5.
- Partially Domain-Aware (PDA) For this setting, we run our algorithms for Dataset-II. Similar to gaussian memberships, the class labels  $Y$  provide another secondary definition of implicit grouping in the dataset. We report the results in Table 5. A critical observation from the  $DA$  setting results is that Online GDRO, and RAI (Group) all exhibit inferior performance according to the secondary class definition i.e. although they optimize for the known groups (gaussian), they fail to optimize for unknown groups (class labels). Thus, a natural solution is to run RAI updates over the intersection
Algorithm	Synthetic Datset-I
Average Loss	Worst Class Loss
ERM	27.3	82.6
AdaBoost	26.5	27.7
α-LPBoost	23.5	23.7
RAI-GA	23.9	24.4
RAI-FW	23.9	24.2
Figure 1: Results for Dataset-I. Left: Average and worst class losses for baselines and proposed RAI updates. Right:  $\alpha$ -CVaR objective values vs number of rounds for train and test splits
Figure 2: Results for Dataset-II. Left: Visualization of classifiers learned. For ensembles, we derandomized them using Definition 3. RAI methods (and Online GDRO) effectively prioritize minority and noisy instances (orange). Right: Testing Group DRO objective values vs the number of rounds (train values are similar and omitted to improve figure clarity). Quantitative results are reported in Table 5
of  $\chi^2$  (for unknown groups) and Group (for known groups) constraints. As seen in the Table, we see that both RAI-GA and RAI-FW achieve a middle ground by significantly improving worst-case performance for both known and unknown groups.
Algorithm	Synthetic Datset-II
Group 1	Group 2	Group 3	Group 4	Group 5	Worst Group	Average	Worst Class
ERM	0.0	3.1	15.2	22.9	1.6	22.9	5.5	5.7
RAI-GA (χ2)	3.1	5.6	10.1	13.3	2.5	13.3	5.3	5.9
RAI-FW (χ2)	2.1	4.7	8.9	13.4	2.9	13.4	4.9	5.1
Online GRO	5.1	3.7	5.8	10.2	5.6	10.2	6.1	6.2
RAI-GA (Group)	5.0	4.2	7.4	10.3	3.2	10.3	5.4	5.7
RAI-FW (Group)	10.0	4.7	6.5	9.5	5.5	10.0	6.9	7.5
RAI-GA (χ2∩ Group)	3.4	4.0	7.3	10.3	3.4	10.3	5.1	5.5
RAI-FW (χ2∩ Group)	4.7	4.9	9.2	10.6	2.6	10.6	5.2	6.1
Table 5: Average, worst group, and worst class losses for Synthetic Dataset-II. All the algorithms listed after Online GDRO have access to group information.