SelecMix: Debiased Learning by Contradicting-pair Sampling
Authors: Inwoo Hwang
Sangjun Lee
Yunhyeok Kwak
Seong Joon Oh
Damien Teney
Jin-Hwa Kim
Byoung-Tak Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: debias
spurious correlation
mixup

Abstract: Neural networks trained with ERM (empirical risk minimization) sometimes learn unintended decision rules, in particular when their training data is biased, i.e., when training labels are strongly correlated with undesirable features. To prevent a network from learning such features, recent methods augment training data such that examples displaying spurious correlations (i.e., bias-aligned examples) become a minority, whereas the other, bias-conflicting examples become prevalent. However, these approaches are sometimes difficult to train and scale to real-world data because they rely on generative models or disentangled representations. We propose an alternative based on mixup, a popular augmentation that creates convex combinations of training examples. Our method, coined SelecMix, applies mixup to contradicting pairs of examples, defined as showing either (i) the same label but dissimilar biased features, or (ii) different labels but similar biased features. Identifying such pairs requires comparing examples with respect to unknown biased features. For this, we utilize an auxiliary contrastive model with the popular heuristic that biased features are learned preferentially during training. Experiments on standard benchmarks demonstrate the effectiveness of the method, in particular when label noise complicates the identification of bias-conflicting examples.

SelecMix: Debiased Learning by Contradicting-pair Sampling
Inwoo Hwang'
Sangjun Lee' Yunhyeok Kwak
Seong Joon Oh
Damien Teney
Jin-Hwa Kim 12
Byoung-Tak Zhang"
AI Institute, Seoul National University University of Tübingen
"NAVER Al Lab
*Idiap Research Institute
Abstract
Neural networks trained with ERM (empirical risk minimization) sometimes learn unintended decision rules, in particular when their training data is biased, i.c., when training labels are strongly correlated with undesirable features. To prevent network from learning such features, recent methods augment training data such that examples displaying spurious correlations (i.e., bias-aligned examples) become minority, whereas the other, bias-conflicting examples become prevalent. However, these approaches are sometimes difficult to train and scale to real-world data because they rely on generative models disentangled representations. We propose an alternative based on mixup, a popular augmentation that creates convex combinations of training examples. Our method, coined SelecMix, applies mixup to contradicting pairs of examples, defined as showing either (i) the same label but dissimilar biased features, or (ii) different labels but similar biased features. Identifying such pairs requires comparing examples with respect to unknown biased features. For this, we utilize an auxiliary contrastive model with the popular heuristic that biased features are learned preferentially during training. Experiments on standard benchmarks demonstrate the effectiveness of the method, in particular when label noise complicates the identification of bias-conflicting examples.
Introduction
The inductive biases contributing the success of deep neural networks (DNNs) can sometimes limit their capabilities for out-of-distribution (OOD) generalization. DNNs are prone to learn simple, linear predictive patterns from their training data, sometimes ignoring more complex but important ones [30, 32). It has been suggested that the simplest correlations in the data are often spurious [7]. DNN relying on such simple, spurious patterns will therefore display poor OOD generalization. Spurious correlations in a dataset are often the result of a selection bias, and such datasets are therefore said be biased. This paper is about debiased learning, also known as debiasing, i.e., methods that prevent network from relying on spurious correlations when trained on biased dataset.
Biased datasets typically contain majority of so-called bias-aligned examples and minority of bias-conflicting ones. In bias-aligned examples, ground truth labels are correlated with both robust and biased f In bias-conflicting examples, labels correlated only with robust features. Clearly, the issues of models trained on biased datasets stem from the prevalence of bias-aligned samples. Various approaches for debiased learning encourage models to ignore biased features. Since
Corresponning authors.
AA feature is biased it displays a pattern that is statistically predictive of labels over the dataset, though not necessarily on every example. For instance, a blue background may present most (but not all) images of birds. These images are said to he bias-aligned.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

the identification of biased features from i.i.d. data is ill-defined, it requires additional assumptions, or supervision from heterogeneous (non-i.i.d) training samples [28].
In this work, we approach debiased learning with the assumption that biased features are "easier to learn" than robust ones, meaning that they are incorporated in the model earlier during training (29,30) 3. Existing works based on this heuristic typically train two models: (i) an auxiliary model that purposefully relies on biased features, and (ii) the desired debiased model. The auxiliary one guides the training of the debiased one [24, 27J. For example, Nam et al. [24] trains the auxiliary model with generalized cross-entropy (GCE) loss [40] that strengthens its reliance on biased, easy-to-leam features. The training of the debiased model either augments the data with novel bias cias-confli examples [15, 20] or upweights existing ones [22, 24]. On the one hand, upweighting baased based methods are simple but their debiasing capabilities are limited when bias- conffciifii examples are scarce. On the other hand, augmentation-ban based methods rely on carefully tuned generative models or disentangled representations that are difficult to apply to real-world data.
We propose a simple and effective method based on mixup [37], a popular data augmentation method that creates convex combinations of randomly-chosen pairs of examples and their labels. Our method, coined SelecMix, is an application of mixup to selected contradicting pairs of examples generate new bias-conflicting examples. We define contradicting pairs as having either (i) the same ground truth label but dissimilar biased features, or (ii) different labels but similar biased features. To compare examples with respect to their biased features and thereby identify the contradicting pairs, we train an auxiliary contrastive model with a novel generalized supervised contrastive (GSC) loss that amplifies the reliance on easy-to-learn features. As a result, feature clustering in the embedding space serves as a good indicator of the similarity of the biased features. We train the auxiliary model and the desired debiased model simultaneously. The auxiliary model identifies contradicting pairs, while the debiased model is trained on data augmented by SelecMix. Compared to past approaches, our method generates bias-conflicting examples without generative models or disentanglement, while implicitly upweighting existing ones since they are frequently selected for the mixup.
We evaluate our method on standard benchmarks for debiasing. Experimental results suggest that SelecMix consistently outperforms prior methods, especially when bias-confficting samples are scarce. In addition, our method maintains its superior performance in the presence of label noise that complicates the identification of bias-conflicting examples. Ablation studies show advantages of both (i) the selective mixup strategy compared to other mixup variants, and (ii) the GSC loss, which strengthens the reliance on biased features and allows measuring the similarity of examples with respect to the biased features.
2 Related work
Debiasing with known forms of bias or bias labels. Early works on debiasing assume some knowledge about the bias. Some methods require that each training example is provided with bias label, i.e., a precise value for its biased features [11, 14, 21, 26, 31J. Other methods use knowledge of the general form of the bias, such as color texture in images. This information is typically used to design custom architectures [1, 34]. For example, ReBias [1] uses a BagNet [4] as an auxiliary model because it focuses texture, which is assumed to be the biased feature. The auxiliary model guides the training of a debiased model robust to unusual variations in texture.
Debiasing with the easy-to-learn heuristic. A number of recent works assume that biased features are learned more quickly than robust ones [20, 24]. A popular approach is to train an auxiliary model that intentionally relies primarily on the easy-to-learn biased features, e.g., with the GCE loss [40]. The auxiliary model guides the training of a debiased model that focuses on other, presumably non-biased features. For example, LfF [24] learns biased and debiased models simultaneously. Bias-conflicting training examples are identified based on the relative losses from the biased and
debiased models, then upweighted for the training the debiased model. BiaSwap [15] trains an image translation model to generate new bias-conflicting training examples. Building on LiF, DFA [20] disentangles robust and biased features and swaps them randomly for augmentation. Disentangiement is however an ill ill-posed problem in itself [23] and a challenge with real-world data. In contrast, our
See [38] for discussion of the disputed relevance of this assumption real-world data.

---

Label =
Label =
Labol =
Label =
Query sample
Label 1
99%
* 15
Generated sample
99%
Figure I: Overview of the proposed SelecMix. gray ellipse with the symbols represents sharing label on the embedding space. The selective mixup is applied to the pairs of samples (dashed lines) having (top) the same label but dissimilar biased features, and (below) the different labels but similar biased features (Sec. 3.3). The orange triangles of the top and below ellipses represent the generated samples by the previous two procedures, respectively. The legend illustrates the distribution of the exemplary dataset of two labels and two bias labels.
method augments bias-conf data by mixing existing examples, without generative models nor disentangled representations.
Method
We first describe the use of mixup debiasing strategy, while assuming that each training example is (unrealistically) provided with bias labels (Sec. 3.1). Then, we move to realistic scenario where bias labels are not available. We introduce an auxiliary contrastive model that identifies the biased features, which are assumed to be "easier to learn" than robust ones. This allows comparing examples with respect to these biased features (Sec. 3.2). Finally, describe the complete SelecMix method that combines the selective mixup with our auxiliary biased model (Sec. 3.3).
3.1 Mixup for augmenting bias-conflicting samples
In highly biased dataset, bias-conflicingg samples constitute only small fraction of the training data. This is the root issue in the cases we consider. The goal is thus to increase the fraction of bias-conflicting examples the training data, which will then reduce the reliance of the Iearned model on biased features. Our idea is to use mixup [37] to augment the existing pool of bias-conflicting examples. Mixup a popular augmentation method [2, 16, 36] known to improve various measures of robustness [25, 39]. It constructs convex combinations of pairs of examples and their labels:
(T., J.) (A (1-A)23, Ay. + 11 v,),
where (I,,, y4) and (Iz. yi) are two original training examples (e.g.. image and one-hot label vector) and À is random mixing coefficient, À U'00, 1].
Assuming for now that bias labels are available, we could generate bias-conflin conf-iiniig examples by applying mixup on pairs having either (i) the same ground truth label but different bias labels or (ii) different labels the same bias label. Any such pair includes least one bias-conflicting example, so that mixup generates additional ones as long as this original example is assigned higher mixing weight in Eq. (1). An overview of this mixup strategy shown in Fig. Next, we describe how to identify such desired pairs when bias labels are not available.

---

Colored MNIST
Compppted CIFAR10
BFFHQ
1.0
1.0
Coolre Cereeereee
0.5
08
64
por
0/100
000g
www
0100g
120
200
150
Fpoch
Fpoch
Coaree simbarty
Ocdire dintrrly
0.5
Rog C000
-05
neg
0/16g
Fpoch
Figure 2: Illustration of the similarity of positive, negative, and contradicting pairs when trained with (top) the SC and (bottom) the proposed GSC losses. The solid line shows the average cosine similarity of (i) the pairs with the same label (positives) and (ii) the pairs with different labels (negatives). The dotted line represents (iii) the pairs with the same label but different bias labels (contradicting positives) and (iv) the pairs with the different labels but the same bias label (contradicting negatives). Observations: As training proceeds with the SC loss, the similarity of contradicting positives increases while it decreases for the contradicting negatives. In contrast, the proposed GSC loss amplifies the reliance on the biased features, thus the clustering in embedding space remains good indicator of the similarity of the biased features during the whole training process.
3.2 Replacing bias labels with an auxiliary model
We utilize an auxiliary model to compare training examples with respect to their biased features. We assume that biased features are easier to learn than robust ones, because they are involved in simpler (e.g., linear) predictive patterns. Our auxiliary model is trained to rely primarily on biased features. We train the auxiliary model with a contrastive objective [6. 10, 13]. This is known to induce a clustering in embedding space (better than standard cross-entropy) that reflects the similarity of training examples in terms of the learned features. These are biased features by our assumption, such that the clustering reflects the similarity of examples w.r.t their (unknown) bias labels. We use the supervised contrastive (SC) loss of Khosla et al. [13]:
log where Pi,k = exp (z. 2x/T) EyEBY( exp (z) 21/2
Lsc =
Σ iEB
(2)
PPal
REP.
where z; is the normalized embedding of image I,, B current mini-batch, P; {k E [i) y4 y is the set of positive examples relative to the example i (i.e., with the same label), and the scalar is a temperature hyperparameter.
..,, B} is the set of indices in the
As an experiment confirm that the clustering of training examples in embedding space is indeed based on biased features, we train the auxiliary model on the Colored MNIST, the Corrupted CIFAR- 10, and the BFFHQ datasets (see Appendix A. with the SC loss and compute the cosine similarity =Tz, of the embeddings of all pairs of examples. Fig. 2 shows that the examples are clustered according the biased features early in the training. other words, predictive patterns involving biased features are learned faster than those involving robust features, as desired. Since the higher cosine similarity =._ z, implies high probability Pi.j. we interpret it as the likelihood of the pair (1,j) having similar biased features.
To further amplify the reliance of the auxiliary model on the biased features, define the generalized SC (GSC) loss as follows:
Losc =
PL. log
(3)
P
iEB
REP.

---

Algorithm SELECMIX
Algorithm 2 Training with SELECMIX
1: Input: batch {(x), M biased model Je
2: Output: batch B {Gx, }t)
1: Input: a dataset D = {2,, 34) a model fo.
biased model ga. the number of iterations '''
3: Sample p
U|0 1]. 8 Ø
2: Output: a debiased model fo
4: for 1, ................................ B do
3: Initialize 0 and
5: Sample ~ U[O, 1] and À &- min(A. A)
4: for = 1,... T do
6: if pf 0.5: h = argmin go(z;)) jep;
5: Draw batch B = {(.,,,))) from D
7: clse:
k argmax go (Z)))))())))
6: B = SELECMIX(B, go)
JEN;
7: Update with Ccr(B)
8:
(T. + (Az, A)EE, 9k)
8: Update @ with Losc
(Eq. (3))
9: B- BU (T.,
9: end for
10:
end for
where pi is scalar having the same value as Př meaning that the gradient is not back-propagated through it. The term P assigns a higher weight to sample pairs with high probability Pt,k and thus amplifies the reliance on the biased features. We discuss the relationship between the GCE and GSC losses in Appendix C.2.
3.3 Complete proposed method: selective mixup with biased embedding space
We now have an auxiliary model for quantifying the similarity of the training examples in terms of the biased features. We use it to apply mixup on contradicting pairs, which have either (i) the same label but dissimilar biased features (contradicting positives) or (ii) different labels but similar biased features (contradicting negutives).
Contradicting positives. For each instance (at, Ui) in the current mini-batch (i.e., the "query"), pick another one (Ix, yks) with the lowest similarity (measured in the space of their embeddings produced by the auxiliary model) among the set of positive examples (i.c., with the same label as x;):
k argmin Pi. = argmin =[=;,, where P, (i E B (1) 1 = ys}. jEP:
jEP.
where B- = {1, 2, B} is the set of the sample indices in the current mini-batch, and z; and zj the normalized embeddings produced by our auxiliary model Jø- i.e., 2; = (r,)) Since select the pair among the set of positives, the training CE loss of the mixed example is ((T,, T)) = ((xz; (1 A)rKk Az: + (1 A)yx) I(Ax; + (1 A)zk,, yk). Considering that most examples are bias-aligned in the training set, the query (I;; Us) is also likely to be bias-aligned. In addition, since the query x; and the selected example zg have the same label but dissimilar biased features, is also likely that the biased features of 23 are not correlated with the label, i.e.. the selected example (x4, y/) likely to be bias-conflicting. Thus, to effectively generate an example that contradicts the prediction based on biased features, we sample x ~ U100, 1| and assign the smaller value among À and to x; and the larger one to Tk.
Contradicting negatives. For each query (x,, v:). we pick another one with the highest similarity among the set of negative examples:
k argmax Pi.j argmax 27 zj. where NG {j B Ue + vz}- JEAr
(5)
jEM:
Similarly, we sample X ~ 00, 1] and let À + min(A, - A). In standard mixup, the training loss of the mixed sample is: 1(5,, 3 = I(xx, + A) Ay; (1 - A)yx)) = X- A-((xr; + (1 A))ra y.) (1- A) I(Ar; + (1 A)Tx 3/.). However, considering that (i) the query (z,, v:) is likely to be bias- aligned, and (ii) the pair (Iz, shares similar biased features, the first term A-((Ar; (1- A)arK yi) acts as bias-aligned example since the biased feature shared with and ø is predictive of the label V:- Thus, rather than interpolating the label, we simply assign T/s <- yk- The pseudo-code of the proposed algorithm is presented in Alg. I and Alg. 2.
Intuitive interpretation. Unlike standard mixup, we assign the label of the generated sample the label of the selected sample, which is likely to be bias-conflicting. Therefore, the proposed method
5

---

can be viewed as generating new bias-conflicting samples by injecting bias-aligned samples as noise to existing bias-conflicting samples. The method can also be interpreted as implicitly upweighting existing bias-conflicting samples, since they are frequently chosen for the mixup pair.
4 Experiments
We now validate the effectiveness of the proposed method on debiasing benchmarks. We also evaluate it under the presence of label noise, which is challenging but realistic scenario, yet less explored so far (Sec. 4.1). We also provide a detailed analysis of each component of our method: (i) the auxiliary contrastive model (Sec. 4.2) and (ii) the selective mixup strategy (Sec. 4.3).
Datasets. The Colored MNIST is a modified MNIST [19] that consists of colored images of ten digits where each digit is correlated with the color (e.g.. most of the images of "0" are colored with red). Here, the label is a digit (i.e.. 0 ~ 9) and the biased feature is color. The Corrupted CIFAR 10 is constructed by applying different types of corruptions the corresponding objects in the original CIFAR-10 10 [18] dataset (c.g.. most of the images of dogs are corrupted with GAUSSIAN BLUR noise). The Biased FFHQ (BFFHQ) [20] is constructed based on the real-world dataset FFHQ [12]. where the label is age and the biased feature is gender. The ratio of bias-confiictnng samples in the training set a (0.5%, 1%, 2%.5% 5%} for [Colored MNIST, Corrupted CIFAR10) and = 0.5% for BFFHQ. All datasets are available in the official repository of DEA [20]. We defer the detailed description of the datasets in Appendix A.
Baselines. To evaluate the effectiveness of our method in debiasing, we compare it with the prior methods LfF [24] and DFA [20] which also rely on the easy-to-learn property of the biased features. LfF trains an auxiliary model with the GCE loss to amplify its reliance on the biased features, then reweights examples for training a debiased model. DFA disentangles the biased and robust features with a similar principle as LIF, then augments the data for training a debiased model by swapping the biased features across examples. We also include EnD [31], ReBias [1], and HEX [34]. EnD leverages explicit bias labels. ReBias and Hex are designed for a specific, known form of biased features such as color and texture in images.
Table 1: Main results. (*) Denotes methods tailored predefined forms of bias, (9) methods using bias labels, and (f) methods relying on the casy-to-learn heuristic. Numbers for HEX are from [20].
Dataset
Ratio (%)
Vanilla
HEX [34]
ReBias [1]
EnD O [31]
LIF ! [24]
DFA [20]
V+Ours
L+Ours
0.5
3087
30.33
71.42:141
:6.98:485
63.86:281
67.37
70.47
70.00
Colored MNIST
1.0
20.51
43.7311550
86.50.09
73.83
21.64
80.20 86
83.55104m
82.801001
2.0
65.40=116
56 85
92.95
82.28:118
84.95=171
85. 61 20.76
87.03
87. 36.16:
5.0
82.12=
74.62+3.20
96.92=suu
89.26
89.42=008
89.86
91 11.5620.1
21.57+000
0.5
23.26=029
13.87a006
22. 13:2.3
22.54=205
29.36=018
30.04
38. 144..1
39,44
Corrupted
1.0
26. 10-122
14.8 14.814042
26.05=00.e
26.20=a0.9
33.50=0052
33.80
41 4188440.4
43.68=0051
CIFAR-IN
2.0
31.04:044
15.20
32.00:0::
32.99:1.31
40.65:1123
42. 10
47.70
49,70:054
5.0
11.98g012
16.04gaex
44.00.00
44.90.0.3
50.95.00.0
49.23
54.0016.3 34
57.03
BFFHQ
0.5
56.20
52.83
56.80 16.80
56.5310m
65.60.1.4 40
61.60
71.60
20.80
4.1
Main results
We apply our method to a vanilla ResNet18 (V+Ours) and to LfF (L+Ours). We use the Colored MNIST, Corrupted CIFAR1O, and BFFHQ datasets. As shown in Table 1, our method consistently outperforms baselines, except ReBias [1] on the Colored MNIST. Their bias-capturing model Bag- Net [4] is specifically tailored to color and texture as biased features by relying on local image patches as input. Both DFA and our method augment the pool of bias-conflicting training examples, but DFA's reliance on disentangled representations seems problematic on the more complex datasets. In contrast, our method performs well on all datasets, owing to the simplicity of the mixup strategy. In particular, our method outperforms DFA by a large margin on BFFHQ while the gap is smaller on Colored MNIST where disentanglement is easier. See Appendix A.2 for experimental details.
Debiasing under the presence of label noise. Label noise can negatively affect debiasing methods that need to identify bias-conflicting examples. This is because training examples with noisy (i.e., incorrect) labels are difficult to distinguish from the bias-conflicting examples that we wish to identify. Fig. 3 shows that our method maintains better performance under the presence of label noise than

---

40
70
V-Out
Acursa)
80-
30
Nory
Nory
(a) Colored MNIST
(b) Corrupted CIFAR-10 10
Figure Unbiased accuracy under the presence of label noise.
Table 2: Ablation study of various metrics on the biased embedding space. GT indicates that we used ground truth bias label for SelecMix. The cos and 12 denote cosine distance and lp distance on the embedding space of the biased model trained with GCE loss, respectively. Ours denote the cosine distance the embedding space of the proposed biased contrastive model.
SelecMix
GT
cos
12
Kl.-divergence
Ours
Contradicting positives
40.31 40.71
38.56:1.25
40.8220.4
42.94
Contradicting negatives Both
41.70
33.69
44.55
30.99
41.82
43.35
38.05:0083
38. 28.902295
39.94:148
43.68
baselines. We hypothesize that the robustness of our method comes from the nature of mixup, which known to improve robustness [39].
4.2 Detailed analysis of the biased contrastive model
Comparison of the biased embedding spaces for measuring bias similarity. Our auxiliary contrastive model trained with the GSC loss learns the embedding space which reflects the similarity of the examples w.r.t. their biased features. is then used for SelecMix to identify the contradicting pairs by measuring the cosine similarity. We replace our auxiliary model with the biased model of LIF [24] which trained with the GCE loss, and use it for SelecMix. For the similarity measures, we used cosine distance and (2 distance in the embedding space learned by their biased model, and the KL-divergence of the softmax outputs of the classification head. As shown in Table 2, our auxiliary contrastive model achieves the best performance. Especially, the performance gap is significant for the contradicting negatives. This supports our claim Sec. 3.2 that the proposed GSC loss induces the feature clustering in the embedding space w.r.t. the biased features. In contrast, the GCE loss learns the decision boundary and does not explicitly cluster the features, thus selecting the contradicting negatives, i.e., the pairs with different labels but similar biased features, seems underperform ours.
Table 3: Bias label prediction accuracy.
Corrupted CIFAR-10
BFFHQ
Biased model
0.5%
1.0%
2.0%
5.0%
0.5%
LfF [24]
77.44-094
73.01=170
67.00
55.58-40.6
51.07+3.6
DFA [20]
1..5
72.301071
64.71
52.01.1.70
46.2011100
Ours
95.45
93.39
87.2810.20
59.1310.25
Bias label prediction of the biased model. Table 3 shows the accuracy of the bias label prediction of the auxiliary biased model for each method. Our biased contrastive model does not have classification head since it is trained with contrastive loss. Thus, we attach linear classifier on top of the model and fine-tune it. Note that the bias labels are used only for the evaluation. As shown in Table 3, the biased contrastive model shows the best performance. As the ratio of the bias-conflicting samples increases, they degrade the performance. Notice our biased contrastive model is robust to this effect compared with the prior methods relying on the GCE loss.
7

---

(a) LFF [24]
(b) DFA [20]
(c) SC loss [13]
(d) GSC loss
Figure Visualization with t-SNE features extracted from (a) the biased model of LfF [24], (b) the biased model of DFA [20], (c) the model trained with the SC loss, and (d) our auxiliary contrastive
model trained with the GSC loss. (Top) a=1%. (Bottom) a=5%.
Visualization. Fig. 4 illustrates t-SNE [33] plot of the features extracted from the corresponding
models trained on the Corrupted CIFARIO. As shown in Fig. 4(d), which corresponds to the proposed biased contrastive model, we observe that the samples are well clustered on the embedding space of the biased contrastive model. Similar to the results in Table 3, the GCE-based biased models, i.c., LiF and DFA, suffer from the increasing number of bias-conficcting samples.
Pretraining vs. simultaneous training of the auxiliary model. For our method in the main experiments, the auxil- iary biased model is simultaneously trained with the debi- ased model. To analyze the performance of the pretrained biased model, we equip SelecMix with the pretrained bi- ased contrastive model and evaluate the performance of our method with the varying total number of epochs for pretraining. The solid line represents the unbiased accu- racy of our method with the pretrained biased model. The dotted line represents the unbiased accuracy of method, ie., simultancous training of the auxiliary biased model. As shown in Fig. 5, vanilla SC loss exhibits the high vari- ance of the performance with respect to the number of the epochs for pretraining, compared to the proposed GSC loss.
40-
38
36-
CSC Phaazel
84
0SC Gamulannocal
32
Pro-training apoch
Figure 5: Comparison of simultaneous training vs. pretraining of the auxiliary model.
4.3 Detailed analysis of the selective mixup strategy of SelecMix
Ablation study. As shown in Table 4, we obtain the best performance when utilizing both the contradicting positives (A) and the contradicting negatives (B) all together in most cases. This is because it generates more diverse bias-conflicting examples compared to when using only one of them alone. On the other hand, standard mixup degrades the generalization capability many cases which implies that the selection of the pairs is crucial.
Comparison with a mixup variant when bias label is available. We compare SelecMix with the recently proposed LISA [35], the mixup strategy for addressing domain generalization and subpopulation shift problems. LISA similarly applies the vanilla mixup on the pair of (i) the samples with the same label but different domains, and (ii) the samples with different labels but the same domain. Here, the domain label corresponds to the bias label in our experiment. The key differences between SelecMix and LISA are (i) ours does not require the bias label, (ii) we do not interpolate the labels of the generated samples, and (iii) we sample and let + min(A, assign the higher weight to the selected sample and the lower weight to the query sample (Sec. 3.3). Table empirically confirms the better performance of SelecMix, using the explicit bias label for fair comparison, over LISA, implying that augmenting the samples close to the bias bias-conflicting
A), i.e., we always
8

---

Table 4: Detailed analysis of SelecMix, (A) denotes SelecMix with only contradicting positives. (B)
denotes SelecMix with only contradicting negatives. (AB) corresponds to the proposed SelecMix.
Dataset
Colored MNIST
Corrupted CIFAR-I0
BFFHQ
Ratio (%)
0.5
2.0
5.0
Vanilla
50.51
65.40.143
82.12-152
23.26=022
26. 10:077
31.04a0.4
41,98a0112
56, 36.20-035
+ mixup
36.90=157
19.19
59.444261
76.28=351
23.53=a00
25.48
31.88
43.41
52.80=a
Ours (A)
58.49:07
74.59
83.162u14
89.32:001
37.16-nm
41.38:041
48.00
54.40:025
66.20
+ Ours (B)
71.23
81 87.0
85.91
90,53.001
35.17
37.74
42.77
49.59
70.33
+ Ours (AB)
70.47=1.06
83.55
87.03
91.56=0017
38.14
41.87
47.70+1.35
54.00
71.60-191
LfF
63.86+281
78. 1..64
84.95s1.71
89.42:00s
29.36=au1
33.50sa.52
40.65+1.23
50,95
65.60=1.4
+ mixup
44.30
58. 28.22
72.44:2.19
85,28
22.71:a0
26.32
32.67:10.45
45. 16:622222
57.53_004
Ours (A)
17.28
74.4418.6
44.20
90.33
38.46.000
49.32.10.
56.11
67.60
Ours (B)
70.26=1.0
83,14
86,
86.4440.49
91.49=0011
37.15=131
41. 82:
47.01
53.68
70.40=222
Ours (AB)
=0.00
82.80+
87.162042
91.57=0020
39.44=022
43.68
49.70
57.03::048
70.80-296
Table 5: Comparison with the mixup variants when the bias label is accessible.
Dataset
Colored MNIST
Comupted CIFAR-10
BFFHQ
Ratio (%)
0.5
1.0
2.0
5.0
0.5
1.0
2.0
5.0
0.5
Vanilla
35.71=0x3
50.51
65,40=1.
82.12+1.52
23.26:0.29
26.10sa.1
31.04=044
41.98aa12
56.20-003
-LSAA (A)
48.95-005
67.101637
78.28:0.
87.04
33.29.0si
38 38.621aa1
45.79
53.41
63.20..022
Ours (A)
18.07-1
72.251000
82,35
90.1 90.11
36. 10
40.55
46,70
46.70_0m
53.901.0
67.67-113
+ .IISA (B)
52.32.22.
12.
79.74
86.44
29.56.n91
34. 34.23
40.47_am
17.61
58.93_00
Ours (B)
72.02:1.3
81 113941143
86.04
88 88.68
35.47
39.27
43.13
48.444086
77.40=20
+ LISA (AB)
60,85=1772
74.42+1.7
83,20 $3,20
88.7346.3p
32.71=1.
38.18
44.15=00.9
51.574045
65.20-053
+ Ours (AB)
70.57:29
83.38
87.22:n.00
90.23
37.02
41.6611..0
48.35=0.
53.47
75.00
samples crucial on the biased datasets. More discussions on the comparison between the SelecMix (using the explicit bias label) and LISA is provided in Appendix C.3.
5 Conclusions
We presented SelecMix, a method for debiased learning that augments bias-confiiting examples using mixup on contradicting pairs of examples. The selection of these pairs is the critical part of the method. It relies on an auxiliary model trained with a contrastive loss designed to amplify reliance on the biased features. The biased features are assumed to be "easy to learn" and incorporated earlier than others during training by SGD. SelecMix outperforms baselines on debiasing benchmarks and remains effective under the presence of label noise.
Limitations. The "easy-to-learn" assumption may not be correct, i.e., biased features are not always guaranteed to be learned faster than robust ones. Our method should only be used when this assumption holds, but it unclear how to determine such cases a priori, and how frequent they are in real-world data [38]. In the worst case, our method could increase reliance on biased features and worsen robustness. Our experiments use semi- comi-synthetic datasets where the assumption made valid by construction. Thus, our results are not a sign of the broad real-world applicability of the method.
Societal impact. Our contributions should have positive impact since our aim is to make machine learning systems more reliable. However, our method is only one step in this direction and the problems addressed should not be considered as solved.
Acknowledgments and Disclosure of Funding
We would like to thank Sangdoo Yun for the useful discussions and Yeonji Song for the suggestions on the writing. We also like to thank the anonymous reviewers for their constructive comments.
This work was supported by the SNU-NAVER Hyperscale AI Center and the Institute of Information & Communications Technology Planning & Evaluation 10005-00000--W..... Sta.Staaaab///OO abb 2019- D.01371-Baby LBA/10%, 2022-0-0955-PICA/5509 grant funded by the Korean government.
2022-0-00951-

---

References
[1] Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning (ICML), 2020.
[2] Duhyeon Bang. Kyungjune Back, Jiwoo Kim, Yunho Jeon, Jin- Kim, Jiwon Kim, Jongwuk Lee, and Hyunjung Shim. Logit mixing training for more reliable and accurate prediction. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 2812-2811 18, 2022.
[3] Yujia Bao, Shiyu Chang, and Regina Barzilay. Learning stable classifiers by transferring unstable features. In International Conference on Machine Learning, pages 1483-1507. PMLR, 2022.
[4] Wieland Brendel and Matthias Bethge. Approximating cnns with bag-of-local-feature models works surprisingly well imagenet. In International Conference on Learning Representations, 2018.
[5] Remi Cadene, Corentin Dancette, Matthicu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. Advances in neural information processing systems, 32, 2019.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[7] Nikolay Dagaev, Brett D Roads, Xiaoliang Luo, Daniel N Barry, Kaustubh R Patil, and Bradley C Love. A noo-godd-to-be-tie true prior to reduce shorteut reliance. arXiv preprint 1rXiiv2102.064000 2021.
[8] Giorgio Giannone, Serhii Havrylov, Jordan Massiah, Emine Yilmaz, and Yunlong Jiao. Just mix once: Mixing samples with implicit group distribution. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
[9] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[10] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9733 2020.
[11] Youngkyu Hong and Eunho Yang. Unbiased classification through bias-contrastive and bias- balanced learning. Advances in Neural Information Processing Systems, 34, 2021.
[12] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401 4401-4410, 2019.
[13] Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33: 33:18661-18673, 2020.
[14] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9012-9020, 2019.
[15] Eungyeup Kim, Jihyeon Lee, and Jaegul Choo. Biaswap: Removing dataset bias with bias- tailored swapping augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14992- 15001, 2021.
[16] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statisties for optimal mixup. International Conference on Machine Learning, pages 5275-5285. PMLR, 2020.
10

---

[17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[18] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[19] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:
//yann. lecun. com/ wwxe//nmsss/////////e
[20] Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debi- ased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 34, 2021.
[21] Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9572 9581, 2019.
[22] Evan Liu, Behzad Haghgoo, Annie Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781-6792.
PMLR, 2021.
[23] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14- 4124. PMLR, 2019.
[24] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems, 33:20673-22 20684,, 2020.
[25] Chanwoo Park, Sangdoo Yun, and Sanghyuk Chun. A unified analysis of mixed sample data augmentation: A loss function perspective. arXiv preprint arXiv:2208.099) 09900, 2022.
[26] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2019.
[27] Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M Rush. Learning from others' mistakes: Avoiding dataset biases without modeling them. In International Conference on Learning Representations, 2020.
[28] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10965)x612-634, 2021.
[29] Luca Scimeca, Seong Joon Oh, Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which shorteut cues will DNNs choose? study from the parameter-space perspective. In International Conference on Learning Representations, 2022.
[30] Harshay Shah, Kaustay Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 03:9573-9585. 2020.
[31] Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and disentan- gling deep representations for bias correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13508-13517, 2021.
[32] Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton van den Hengel. Evading the simplicity bias: Training diverse set of models discovers solutions with superior ood gen- eralization. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, 2022.
[33] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.

---

[34] Haohan Wang, Zexue He, Zachary C Lipton, and Eric P Xing. Learning robust representations by projecting superficial statisties out. In International Conference on Learning Representations. 2018.
[35] Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. arXiv preprini urXiv:2201.00299 2022.
[36] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[37] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
[38] Jianyu Zhang, David Lopez-Paz, and Léon Bottou. Rich feature construction for the optimizaation-gneralizattoo dilemma. arXiv preprint arXiv:2203. 15516, 2022.
[39] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In International Conference on Learning Representations, 2020.
[40] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [No]
12

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "UNAhzt5NFT", "reviewer": "Reviewer_YfvZ", "summary": "This paper proposes SelectMix for debiased laerning. Contradicting pairs are selected for mixup with different training loss to achieve the learning goal. The experiments show improvement over previous methods.", "questions": "see above", "limitations": "Limitations and societal impact are presented.", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Comprehensive experiments and analysis are provided. The results also show improvement.\n\nWeaknesses\n1. The organization of the paper is not very easy to follow.\n2. The core contribution of the paper is SelectMix, but also a contrastive objective is included in training. The analysis and improvement of adding the contrastive loss is not shown. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FAltFZdPHDP", "reviewer": "Reviewer_HQZ4", "summary": "The authors proposed selective mixup for learning robust classifiers. Specifically, they apply mixup to \n1. input pairs that have the same label but different bias attribute values.\n2. input pairs with different label but with similar bias attribute values.\nSince the bias attribute values are usually not given, they estimate it by a bias-amplified model (that favors easy-to-learn shallow features during training). On MNIST, CIFAR10 and BFFHQ, combining selective mixup with previous methods consistently improves the robust performance.", "questions": "1. I think comparing with JTT [1] is necessary and will strengthen the paper.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. Adapting input mixup for de-biasing is interesting.\n2. Consistently good performance across three datasets.\n\nWeakness:\n1. The idea is clearly presented (nice figure) but the paper is poorly written.\n    + For example, you cannot just simply name *bias attributes* as *bias*. Bias often refers to unwanted correlations between the input and the output. The sentence \"the same label but dissimilar bias [line 11]\" is just very confusing.\n2. A few missing references.\n    + [1] uses group DRO to learn a robust classifier from the training mistakes. It outperforms LfF by a significant margin, I am very curious to see how does SelecMix compare to this method.\n    + [2] proposes to use a biased model to guide the training of a debased model. I think they are one of the first among this line and it should definitely be cited.\n    + [3] uses different data environments to learn a metric space that encodes examples based on their bias attributes. \n\n\n[1] Liu, Evan Z., et al. \"Just train twice: Improving group robustness without training group information.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Sanh, Victor, et al. \"Learning from others' mistakes: Avoiding dataset biases without modeling them.\" International Conference on Learning Representations. 2020.\n\n[3] Bao, Yujia, Shiyu Chang, and Regina Barzilay. \"Learning Stable Classifiers by Transferring Unstable Features.\" ICML 2022.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "EjPQyn8amSM", "reviewer": "Reviewer_b6K4", "summary": "This work deals with the problem of groups (bias) robustness under distribution shift between train and test set. The authors propose a mixup strategy where samples with i) same label but different groups, and samples with ii) different labels but similar bias are mixed without having access to bias (or group) annotation. Cosine similarity is used as a proxy to find clusters of different groups training an additional model via supervised contrastive learning. The authors provide experiments on vision datasets to corroborate their method.", "questions": "* Mixup-based augmentation has been employed to handle distribution shifts finetuning large models, for example in this recent work - https://openreview.net/forum?id=HI2ilxFli0W - class-conditional mixup is employed to improve group robustness in presence of group shift between train and test in absence of group annotation. How would your method, in particular, `i) the same label but dissimilar bias` part of SelecMix, compare to this or a similar approach for transfer learning or distribution shift?\n\n* For SelecMix, other than i) you also mix `ii) samples with different labels but similar bias`. How important is this second mixing for SelecMix to work? I would assume that using standard mixup on a train set with highly unbalanced groups (99% vs 1% or 95% vs 5%) the results should be identical to applying ii) and shouldn't help group robustness. Can you elaborate on why using ii) is helpful compared to just applying mixup in Table 4?\n\n* Do you think you could use your method to de-bias language or a different modality?\n\n* What is the metric in Table 1? Average accuracy? It would be interesting to see the worst-group (label, bias) accuracy for the different methods.", "limitations": "* The effectiveness of the method is based on how well the cosine similarity can cluster samples from the same group together. This is a reasonable assumption for images but not necessarily for other modalities, and for this reason, the method is (or could be) limited in applicability.\n* Additionally I think this approach will have issues with larger datasets with more variety and number of classes and biases.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths\n\n* Important work that explores the use of mixup strategy in distribution shift with possible applications to foundational models.\n* The paper is well written and clear with a solid set of experiments and ablation studies.\n* The method is effective and experiments are provided on vision benchmarks.\n\nweaknesses\n \n* Cosine similarity to cluster samples from the same group can be ineffective for more complex datasets with a large number of classes or a large variety of biases.\n* Method limited to images.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept