Online black-box adaptation to label-shift in the presence of conditional-shift
Authors: Faruk Ahmed
Aaron Courville
Venue: Submitted to ICLR 2023 (2023)
Keywords: label-shift
online
black-box
adaptation
Bayesian

Abstract: We consider an out-of-distribution setting where trained predictive models are deployed online in new locations (inducing conditional-shift), such that these locations are also associated with differently skewed target distributions (label-shift). While approaches for online adaptation to label-shift have recently been discussed by Wu et al. (2021), the potential presence of concurrent conditional-shift has not been considered in the literature, although one might anticipate such distributional shifts in realistic deployments. In this paper, we empirically explore the effectiveness of online adaptation methods in such situations on three synthetic and two realistic datasets, comprising both classification and regression problems. We show that it is possible to improve performance in these settings by learning additional hyper-parameters to account for the presence of conditional-shift by using appropriate validation sets. 

Under review as a conference paper at ICLR 2023
ONLINE BLACK-BOX ADAPTATION TO LABEL-SHIFT IN THE PRESENCE OF CONDITIONAL-SHIFT
Anonymous authors Paper under double-blind review
ABSTRACT
We consider an deployed online in new locations (inducing conditional-shiff) such that these locations are also associated with differently skewed target distributions (label- shift). While approaches for online adaptation to label-shift have recently been discussed by [Wu al. (202I), the potential presence of concurrent conditional- shift has not been considered in the literature, although one might anticipate such distributional shifts in realistic deployments. In this paper, we empirically explore the effectiveness of online adaptation methods in such situations on three synthetic and two realistic datasets, comprising both classification and regression problems. We show that it is possible to improve performance in these settings by learning additional ppprrppraaeters to account for the presence of conditional-shift by using appropriate validation sets.
setting where trained predictive models are
INTRODUCTION
We consider a setting where we have black-box access to a predictive model which we are interested in deploying online in different places with skewed label distributions. For example, such situations can arise when a cloud-baed, proprietary service trained on large, private datasets (like Google's Vision APIs) serves several clients real-time different locations. Every new deployment can be associated with Isiel-shift. Recently, Wu et al. (2021) discuss the problem of online adaptation to label-shift, proposing two variants based on classical adaptation strategies Online Gradient Descent (OGD) and Follow The Leader (FTH), Adapting the output of model to new label-distribution without an accompanying change in the label-conditioned input distribution only requires an adjustment to predictive distribution (in principle). Therefore, both methods lend themselves to online black-box adaptation to label-shift, which makes on-device, post-hoc adjustments to the predictive distribution feasible under resource constraints.
In this paper, we empirically explore such methods when the underlying assumption of an invariant conditional distribution is broken. Such situations likely arise in reality. For example, in healthcare settings there are often differing rates of disease-incidence (label-shift) across different regions (Vos al. 2020) accompanied by conditional-shift in input features different deployment locations, for example in diagnostic radiology Cohen et al. (202T). In notation, for input variable and target variable y. we have that pnew (x: 3) # P( y) and pnew (3) * P(y), for training distribution P and test distribution pnew in a new deployment location.
Contributions Our contributions are follows.
We conduct an empirical study of the FTH and OGD methods introduced by Wu et al. (2021) in black-box label-shift settings with concurrent conditional-shi a situation likely to arise in realistic deployments.
We explore the question of how potentially improve performance in such practical settings by computing confusion matrices on OOD validation sets, and show that adding extra hyper- parameerrss can contribute further improvements.
We reinterpret a simplified variant FTH under a more general Bayesian perspective, enabling to develop analogous baseline for online adaptation in regression problems.

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND
We begin with a brief review of online adaptation methods for label-shift for classification problems, based on the recent discussion in Wu et al. (202T). While their motivation is temporal drift in aabel-distributions, we consider the case where a single model is serving several clients online in different locations, each with their own skewed label-distribution that does not change even further with time. If the training set label-distribution is P(y) and the label-distribution in the new location pnew(y) and if we assume pnew (x) y) = ((x y), then the following holds
P Dnew y x) P(x y) Paec (v) Ply x)) P()) pnew (3)
pnew (y) P(u)
P(y x).
(1)
Puew (x)
P(y)
Pnew
(z)
Le., the location-adjusted output distribution is simply reweighting of the output distribution from the base underlying predictive model. Wu et al. (202)) follow along past work on label-shift adaptation by restricting the hypothesis space for f to be that of re-weighted classifiers, since Eq. [ iimplies that one only needs to re- re-weight the predictive distribution to account for label-shift. The parameter vector for this classifier is simply the vector of probabilities in pnew (y), henceforth referred as p. and we will similarly use to represent the training-set probability distribution, Ply). Given an underlying predictive model f. the adjusted classifier rule therefore given by
pby Pr(y x))
g(x; f. q.
arg max vE(K)
(2)
q
where (y x) is the predictive distribution produced by an underlying base model f; for example, a softmax distribution produced by a neural network, and there are classes our dataset.
2.1 ONLINE ADAPTATION ALGORITHMS
Wu et al. 1 present two online updating methods to estimate p Online Gradient Descent (OGD) and Follow The History (FTH).
If we assume knowledge of confusion matrix for classifier f in new location, C.eww (f) R such that CTTTH [] = Pg--Pue- (a) (zly=i) (z() = j), then Wu et al. (202I) show that the expected error rate in this new location can be derived as function of the label-distributee
pnew (y). If we represent puew (y) as K -ddimensional probability vector geew, the expected error rate is given as
K
nnew (1)
Σ (1- Po- (xl== (f(x)) i)) 1=1
a nnw [i] (1 diag ((t nex),
where the all-ones vector. Since we have assumed no conditional -shift so far, C Cj, i.e. the confusion matrix remains invariant under label-shift. This implies one can optimize the expected error rate in the new deployment location using a confusion matrix estimated from a large in-distribution validation set, Cy, in place of Ciper in Eq.
Online Gradient Descent (OGD) Assuming that diag( Cy) is differentiable wrtf, f. we can update f to minimize the expected error rate. We would typically not be aware of the true label-distribution in the new deployment location. However, when the confusion matrix Cy is invertible, we can compute an unbiased estimate of this distribution, given qnew
(CJ) -1 e, where is one-hot vector for the predicted category. Using this, Wu et al. 2222 present an unbiased gradient of gnew (f).
jnew (
E pnow of [1 diag(C,))]
new
When the hypothesis space is restricted to the space of re-weighted classifiers g (Eq. 2 this gradient only over p. Ww et al. (2021) show how might effective numerical methods estimate this gradient. In the online setting, is updated after seeing new examples, hence the / + 1-th gradient update is performed by computing the gradient the current point pc, followed by projection the probability simplex.
i
inew (p)
= E. puew
1 diag(C,))
gTEw]
(5)
p
3p
D=P.
p=p.
2

---

Under review as a conference paper at ICLR 2023
P++1 Projax-1
P: - 11 Ene
where 1) the learning rate and Proj is the projection operator.
Follow The History (FTH) The update rule for p. in FTH is simpler and more efficient (in terms of memory and time complexity), given by
P++1
4
where grew is the estimate for the label distribution at the 7-th iteration. Empirical evidence in Wu et al. (202T) suggests that FTH performs very competitively with OGD, and might be preferred in highly resource-constrained settings.
UNMET ASSUMPTIONS IN PRACTICE
We now consider applying the above strategies in cases where some of the assumptions in the above section are broken. While it is difficult to make conclusive theoretical statements in situations when these assumptions break, we propose some heuristics which we evaluate empirically.
3.1 THE ASSUMPTION INVARIANT P( y) CAN BREAK
In realistic deployments in new locations, it is likely that along with a differently skewed label- distribution, the conditional distribution will change as well, ic. pnew (x y) * P(# y). In our study, we will assume that this distributional shift only takes place within the same domain, and along (potentially warriously-correattee non-semantic features, leaving the semantic features intact, setting likely to be manifested in different deployment locations.
HEURISTIC One possibility to adapt the above methods to settings with concurrent conditional- shift is estimate the confusion matrix on an OOD validation set. Intuitively, an IID-estimated confusion matrix is likely to be over-confident, and a surrogate-OOD validation set can better reflect performance at test-time OOD settings.
HEURISTIC 2
We propose to add extra scaling hyper- peeer-arrmeters in the decision rule in Eq. Specifically, we add the scaling hyper-parameters Xy and X, before making a test prediction,
( q.
arg max log Py x)) Au log ply] Xy, log gly).
(8)
yE[K]
where we have rewritten the rule in log- space. In this formulation, log Py (v x)) = logit (xx). so we can drop the normalizing term. This results predictive rule that adjustment (Menon et al., 2021). Intuitively, these hyper-parameters play the role of determining how much of the training prior to "subtract", and how much weight to assign to the pseudo-label based re-adjustment. When these magnitudes learned validation sets representing combination of label-shift and conditional-shife one can hope to further improve at novel test-time deployments.
form of logil-
3.2 CONFUSION MATRICES CAN BE NON-INVERTIBLE
Existing work on label-shift based on confusion matrices rely on significantly large held-out validation set to estimate a robust confusion matrix. When the underlying dataset is highly class- imbalanced with several categories and limited-size validation sets, one can easily end up with non-invertible confusion matrix. [Lipton et al. (2018) suggests two main possibilities use of a soft confusion matrix, or pseudo-inverse. In our experiments on a large-scale realistic dataset, we find both choices to lead to degraded performance. We find that simply using identity matrix approximation can recover some the performance drops (see Appendix E When using FTH with identity Cy, this corresponds to simply using the pseudo-labels up time to estimate the label- distribbtin......... However, naively using the identity matrix in Eq. might lead to practical problem: after seeing the first data-point, p would be one-hot vector, and thus enforce the same
3

---

Under review as a conference paper at ICLR 2023
prediction the next iteration when using Eq. A fix would be "pseudo-count to smooth initial conditions, which is reminiscent of Bayesian posterior updates. In the next section, we use this realization starting point suggest simpler well as more general framework. This framework then enables us to develop an equivalent online label-shift adaptation method for regression problems.
4 A BAYESIAN PERSPECTIVE
we use the vector a to keep online counts of predictions, with an initialized Cko, such that
ax [k]
Σ
1 k) 00 1 = k] + a.-1(k),
(9)
then using an identity confusion matrix in Eq. 7
corresponds the following update rule,
[k]
Pc+1
[k]
(10)
Σ k==1 [e(k')
We recognize that this update-rul rule corresponds exactly the posterior predictive distribution com- puted using Categorical likelihood with a Dirichlet prior, and using recursive rule for updating the posterior. More precisely, we use
Dir(a:): y/ Φ Cat(o),
(11)
(12)
where & E K- are the parameters of the Categorical distribution, in the following update equations
P((0) & P(y 6) Pt-1(0),
(13)
P4) / P Φ) Po(p) do.
(14)
then we arrive at Eq. I0 using Eq. [4] and Eq. 9 using Eq. 13 See Appendix[AJfor a derivation of Eq. 13
In practice, Y/ not available to us, and we use the pseudo-label ÿt. instead, as in FTH.
4.1
EXTENSION TO REGRESSION PROBLEMS
While adaptation for regression problems has been discussed more generally (Cortes & Mohri 2011 2014 Zhang al. 2013), analogous discussion for online black-box label-shift adaptation is missing for regression. We adapt the general online update rules in Eq. 13 14 regression problems undergoing similar concurrent test-time distributional shifts. A natural choice is to use Gaussians to model the distributions over the continuous target variable,
(-
A
Py x)) exp
2 y
f(x))
(15)
Ay
P(y) exp
y
(16)
2
where Ag, Ay are the precision parameters and m is the training set mean. The parameters & in Eq. 13 now the mean and precision parameters for y in the new deployment location. We the Normal-Gamme distribution to model the posterior over these parameters, since this is the conjugate distribution for Gaussians with unknown mean and precision (DeGroot] 2004).
P(p new new
N/(1 new
Ga(Aneas new
a, b).
(17)
neww
Combined with the Gaussian likelihood in Eq. 14 this yields pnew (y) in the form of Student's t-distribution.
2011
L (v 21)
puew (v)
1
(18)
2a
4

---

Under review as a conference paper at ICLR 2023
Train (r = 0.99)
7 s 007<0
Validation (opposite colors with r = 0.75)
BUbyKORO
1120472911
Test (r = -10)
10
(a) Synthetic variant of the MNIST dataset constructed by using colors correspond to sources with skewed tabel-distibuttions. The colors are flipped for validation and test with different correlation strengths, correspond- ing to (almost completely) reversing the label-skew the sources test-time.
data. Differently colored regions along the z-axis correspond to training. validation and test samples, with different regions of the color corresponding different sources/locations.
(b) Synthetic
Figure I: Synthetic MNIST and Gaussian datasets.
are where 2a is the number degrees of freedom, and L = b(x++) (in log-space) takes the form
Using these, our predictive function
Au (v- Ay 2
2
2a log (1 L 2a
(w-
arg min
y m
(19)
Setting the derivative wrt y to zero yields cubic equation (see Appendix B.I which we can solve to find roots. A positive sign of the second derivative of the objective tells us if solution is (local) minima. When we have one real solution with a positive second derivative, we use this; when we have multiple real solutions with positive second derivatives, smallest objective; when we have no real solutions with positive second derivatives, we do not update Py I)) retaining fxx as the solution. Empirically, we find that the condition for no local minima does not arise for optimal choices of
pick the that corresponds the
(also see Appendix B.2).
The update equations at the f-th step follow from the computation of the posterior using Eq. (see Murphy (2007), for example, for the derivation of these update steps) and are given as:
13
Kel00+1 pt)2 Keff + Dc+1, by41 b. Ki
0++1 + 1/2; Kr+1 + 1; Pe+1
(20)
2( +
The pypae-paasmetees AS (output precision) and и (equivalent of the smoothing pseudo-count ogg in classification) are picked on the validation set, along with scaling pre-multiplier for the precision Xy (analogous to the classification setup). In order to place uniform priors over the output range, we will simulate uniform set of samples over the output range. = Eeyddo is the mean of the pseudo-samples, and B is initialized as 0.5(x (ar (see Appendix B.3 for details).
EXPERIMENTS
We compare variants of online label-shift methods based on our discussion above on a mix of synthetic and realistic datasets to the un-adjusted model performance (BASE).
These are the variants proposed in Wu et al. (021). We evaluate both for choices of confusion matrices each computed using the in-distribution validation set, and using the out-of-distribution validation set (our HEURISTIC 1), We refer to these two alternatives as (C-IID) and (C-OOD).
FTH and OGD:
FTH-H and OGD-H: H: These hyper-parameters proposed in HEURISTIC 2. For both variants, we again evaluate two versions each, using (C-IID) and (C-OOD).
modifications of FTH and OGD using the scaling
FTH-H-B: This is modification of FTH, with an additional pseudo-count hyper- parameter added for smoothing. The sarame-arameters are learned on the OOD validation sets. We call the regression variant FT (R).

---

Under review as a conference paper at ICLR 2023
Figure 2: Skewed COCO-on-Placess Synthetic dataset constructed by superimposing COCO ob- jects (Lin et al. 2014) on scenes from the Places dataset (Zhou et al.) 2017). The 5 columns correspond sources of data, where the backgrounds correspond examples of particular scenes, and the skew in number of examples per row correspond to the skew in label distribution we impose. Different background scenes are used for training, validation, and test sets.
OPTIMAL FIXED CLASSIFIERS: These oracle methods are derived by replacing in Eq with the empirical location- label distributions, providing a sense of achievable gains one were aware of the true dibribdistisbutions from the get-go. We include two variants OFC, which uses Eq. / and OFC-H, which uses the modified update rule in Eq. 8 where the hyper-parameter are oracle hyper-parameters learned on the test-set.
When using OGD, we use the surrogate loss implementation in Wu et al. (202)) since is both better-perfee netter-performing well as much faster. This variant involves using smooth approximation of the 0-1 loss allowing for direct gradient computation instead of numerical approximation.
5.1
CLASSIFICATION PROBLEMS
5.1.1 SYNTHETIC: SKEWED-MNISH
We split MNIST classes into two subsets: [0, 1, 2, 5, 9] and [3, 4, 6, 7, 8]. We use different colors to correspond to different deployment locations, similar to Arjovsky et al. (2019), In the training set, we color digits in a particular subset particular color 99% of the time. This corresponds to a 99% skew in dibbr-iistiibutions across the two locations. The 1% cross-over instructs some color-invarian but not strongly enough to completely overcome the bias. The validation set uses opposing colours for the subsets, but with 75% correlation this represents scenario where the class-distributions in different locations change from that in training. Finally, the test set uses completely flipped colors in the two subsets compared to the training set this implies reversed babel-distributions, resulting in poorer baseline performance.
Since the overall class frequencies are balanced in the training set, we drop the P(y) from the update rule in Eq. and [ With a 3-layer CNN trained for 20 epochs to 100% training set accuracy and 99.6% in-distribution test accuracy, we find, in Table [ that using online adjustments at test-time can lead to marked improvements for the base model in the test set. The numbers are averaged over independent rounds of base-model training, with validation and test sets randomly shufiled for trials for each round of training. (More details about dataset construction in Appendix C.I
5.1.2 SYNTHETIC: SLEWEDCCOOOOOO
We construct a second, more photo-realistic, synthetic dataset by superimposing segmented objects from COCO Lin et al. (2014)
scenes from the PLACES dataset Zhou al, (2017), in Ahmed et al. (202T) The scenes correspond to the notion of deployment location, albeit with significant intra- location variation. For every such -epreseepeesented source. we use a different distriiisti to simulate source-specifie skews in the label distribution. In Fig. 2 the relative number of images per row represent the relative frequency 10K training images, ~ 2.5K validation images (cach for seen and unseen sources), and ~ 6K test images (cach for seen and unseen sources).
particular class at a specific source. There are a total of
The validation and test sets are constructed similarly. For in-distribution validation and test sets, the same set of scenes as for training is used (with different instances), and for new-location validation and test sets, different sets of scenes We train a 5esNet-50 for 400 epochs with SGD+Momentum for the underlying model, achieving an on-distribution test accuracy of 75%. Since the overall distribution of classes is close uniform,
used. See Appendix C.3 for details about dataset construction.

---

Under review as a conference paper at ICLR 2023
Table 1: Classification problems: Average accuracy on SKEWED-MNIST. SKEWED-COCO-OD PLACES, and WILDS-IWILDCAM (also reporting macro Fi-score for IWILDCAM)) Overall trends indicate that our heuristics are helpful, and FTH-H-B is competitive or better without needing confusion matrix.
Method
S-MNIST
S-COCO-ON-PLACEES
|WILDCAM (Avg.)
IWILDCAM (FI)
BASE
82.59 = 1.82
56.09 + 0.66
73.10 3.26
32,70 0.16
FTH (C-IID).
93.12 1.57
58.50 + 0.55 58.94 + 0.63
71.41 4.91
29.57 0.93
FTH (C-OOD)
96.04 + 1.03
71.41 4.91
29.57 + 0.93
OGD (C-IID)
88.32 2.06
57.37 0.51
71.66 4.56
32.56 0.27
OGD (C-OOD)
95.75 + 0.70
57.75 + 0.29
73.11 + 3.05
32.49 1 0.41
FTH-H (C-IID)
98.21 0.47
56.72 1 0.84
73.75 3.77
32.46
FTH-H (C-OOD)
98.69 + 0.31
57.81 + 0.74
73.75 4 3.77
32.46 1 0.31
OGD-H (C-IID)
96.07 + 1.76
57.58 - 0.79
72.89 3.30
31,74 + 0,51
OGD-H (C-OOD)
98.91 + 0.20
57.12 + 0.15
73.36 + 3.51
31,36 + 0.41
FTH H-B
97.46 + 0.64
58,42 + 0.49
74.10 3.56
33.33 + 1.31
OFC OFC-H
99.24 0.20
75.88 0.33
79.19 1.76
48.61 0.27
99.26 + 0.20
75.88 + 0.33
81.07 + 0.79
48.61 + 0.27
we again drop the marginal P(y) term in Eq. over the unadjusted base model for all variants. Accuracy is aggregated across 20 random orderings of the test set (since the test-sets are smaller for this specific dataset), for 3 rounds of base-model training each.
2 and g In Table we again find improved performance
5.1.3 WILDS
We the variant of the IWILDCAM 2020 dataset Beery al. (2021 curated by the WILDS set of benchmarks for -uitt--distribution (OOD) generalization Koh et al. (202T). The data consists of burst images taken at camera traps, triggered by animal motion. The task is to identify the species in the picture, and the locations correspond to the unique camera trap the pictures are from. There are a total of 182 species in this version of the dataset across total of 323 camera traps. There significant skew in terms of species distribution across different camera traps, as well as the number of images available for each trap. The training set consists of 130K images from 243 traps; the in-distribution validation set consists of ~ 7.3K images from the same traps that in the training set but on different dates; the OOD validation set consists of ~ 15K images taken at 32 traps that different from the ones in the training set; the in-distribution test set consists of ~ 8.1K images taken by the same camera traps as in the training set, but on different dates from both training and validation; finally, the OOD test set consists of 43K images taken 48 camera traps that different from those for all other splits.
Koh al. (222)) trained ResNet-50 based models along with their curation of this dataset, also evaluating several methods for OOD generalization and releasing all models. We use their models trained with the domain generalization method CORAL Sun & Saenko (016), since this model has improved performance over the ERM baseline. They released three sets of weights, trained with three random seeds. We evaluate all variants for each of the three seeds, with random orderings each of the test set, and report aggregates in Table I with both average accuracy as well as macro-F1 (since some species the dataset are rare). We perform evaluation with both metrics, but use own trained models for average accuracy this because Koh et al. (202I) trained their models optimizing for macro FI. We similarly trained CORAL-augmented base models optimizing the penalty coefficient and choice of early stopping.
Koh et al. (2021) recommend evaluation
We replace the confusion matrix with an identity matrix for evaluating methods on this dataset (for methods where validation-sed set estimated confusion matrix is required). Confusion matrices evaluated on the validation sets are non-invertible for this dataset due to sparse rtassereprsstntttion and we found common alternatives perform poorly (see Appendix
7

---

Under review as a conference paper at ICLR 2023
Table 2: Regression problems: For the GAUSSIANS dataset the metric is mean squared error (lower is better), and for the Pov folds the metric is Pearson's s correlation co-efficient (higher is better), computed separately for average (ALL) and worst-group (WG) performance.
Dataset
BASE
FTH-H H-- (R)
MIX-OF- GAUSSIANS
9.17 + 2.17
4.35 + 1.48
MAVERTYMAP Fold
BASE
FTH-H-B (R)
POVERTY Fold
BASE
FTH-H-B (R)
(ALL)
0.84
0.84 0.00
A (WG)
0,42
0.43 = 0.00
(ALL))
0.83
0.82 0.00
(WG)
0.52
0.50 + 0.01
C (ALL)
0.80
0.83 0.00
C (WG))
0.42
0.56 + 0.01
D (ALL)
0.77
0.77 177+0000 0.00
(WG)
0.50
0.56 = 0.01
(ALL)
0.75
0.75 0.00
E (WG)
0.34
0.37 - 0.00
5.2
REGRESSION PROBLEMS
5.2. SYNTHETIC: MIX-OP-GAUSSIANS
We create a synthetic regression dataset by constructing a curve from mixture of Gaussians. We pick regions on the r-axis to correspond to training, validation, and test sets, such that every set samples data from two regions each, corresponding to two locations (see Appendix C.2 In Figure [b we depict the curve, along with sampling indicators for the different sets and sources. The points have been placed at different heights for clearer visualization of overlaps. 500 points are sampled from the two training regions, and 250 each for the validation and test sets from their assigned regions. We train a 3-layer MLP with BatchNorm and ReLU activations and a mean squared loss for 100 epochs, yielding in-distribution test mean squared error (MSE) of ~ 0.15. In Table we find that online updating reduces the OOD test MSE significantly. Results are aggregates over five trials, with different random sampling of all data, followed by training and validation each time. Full results and more experimental details are in Appendix C.2)
5.2.2 SELD--POVERTYMA
We use the WILDS variant poverty mapping dataset Ych et (2020). This is a dataset for estimating average household economic conditions in region through satellite imagery, measured by an asset wealth index computed from survey data. The data comprises 8-channel satellite images with data from 23 African countries. The locations here correspond to different countries. Due to the smaller size of the dataset, Koh et al. (202T) recommend a five-fold evaluation, where every fold is approximately constructed as follows 10K images from 13- 14 countries in the training set; 1K images from the same countries for in-distribution validation; K images from these countries for in-distribution testing; 4K images from 4-5 countries not in the training set for OOD validation; and 4K images from 4-5 countries in neither training nor validation sets for OOD test.
The evaluation metric is Pearson's 's correlation between predicted economic index vs. actual index, as standard the literature (Yeh et al., 2020). Following Koh et al. (2021), we split the assessment into overall average as well as worst worst-group performance, which picks the worst performance across rural/urban subgroups. As with IWILDCAM, we use the CORAL-augmented base networks and weights released by Koh al. (2021), but with our retrained versions for average correlation coefficient (since the validation choices for the released weights were for worst group performance). We evaluate separately for each fold (which have quite a bit of variance base performance) with random orderings of each of the test sets. In Table 2 we find that while there seems generally little to no improvement for average correlation, there are more significant improvements for three of five folds in terms of worst-group performance. As noted in Koh et al. (2021), a wide range of differences along many dimensions such as infrastructure, agriculture, development, cultural aspects play a role not only in determining wealth-distribution. but also in terms of how the features manifest in different places. Such real-world issues imply that validating for OOD performance is bound to be sensitive to problem types and the specific choices of validation sets used to tune dyper-paramene differences that may arise between an OOD validation set and an OOD test set. This issue extends generally all attempts OOD generalization.
and the
8

---

Under review as a conference paper at ICLR 2023
5.3 TAKEAWAYS
Our experiments are generally suggestive of the following takeaways.
While invertible confusion matrices are not always achievable due to data scarcity (as modelled in our experiments with matrix free methods such FTH-H-B, which we find to provide competitive or improved performance. Using OOD validation sets to estimate confusion matrices can improve results relative using IID validation set, although confusion matrices estimated smaller- sized sets can be noisy.
a practitioner can adopt confusion-
Learning additional scaling hyper-paramettes can be useful for further improvements. We find this trend to not hold for FTH-H-B). We suspect this is likely due to instability from the relatively smaller size of the validation set - when picking oracle scaling hyper-parameters on the test set, we achieve accuracy of 59.37 + 0.89. In Appendix [D] we compare performance when learning hyper-parameters on different validation sets IID/OOOD/tst (oracle).
S-L---CCCCO-O---C-CCCCCCCC (FTH outperforms FTH-H and
RELATED WORK
Label-shift for classifiers Saerens et al. (0022 provides a seminal discussion about adapting the output distribution of a classifier when the test undergoes label-shift. This approach presumes access to the entire test set up front, or a sufficiently representative sample. More recent works have investigated other ways to estimate label-shift (Lipton al., 2018 Azizzadenesheli et al.) 2019) using confusion matrices, which partially inspired the methods in Wu et al. foundation. It has been recently suggested (Alexandari et al.) 2020 Garg et (0020) that the simple correction method in Saerens et al. (2002) often outperforms these later methods when combined with calibration. While Alexandari et al. (2020) perform their calibration using a held-out IID validation set for their iterative method, we adapt this strategy to the out-of-distributin setting by picking scaling hhper-parametee parameters on an OOD validation set.
that we use as our
Test-time training Another emerging line of literature focuses on updating neural network pa- rameters using test data without being able match training statistics with test statistics, due the potential lack of access to training data for the same topical reasons - data privacy and large datasets. Some examples include updating the Batch-Norm statistics optimizing for minimum test-time en- tropy Wang et al. (2021), or using self-superviser pseudo-labels to adapt the feature extraction part of the network Liang al. (2020). Our setup here can be viewed as form of test-time training, but in more constrained setting, with inaccessible model parameters and no resources to replicate an unsite-modes by querying the black-box model, c.g. using distillation (Hinton et al. 2015).
-nttrf-distribbttin generalization There has been a recent surge in interest for methods aiming learn stable or invariant features across different domains/enirrommetts/roousss (Sun Saenko (016);; Arjovsky et al. (2019); Krueger et al. (2020): (Sagwa et al.) (2020). Such approaches have been demonstrated to be useful for certain types of distributional shifts, such as with improved minority group robustness Sagawa et al.] (220)) and systematic generalization cetruliiaaiioAhmmee AAhmed et al. (2021). Our discussion in this paper is complementary to this set methods in OOD generalization research. One can use underlying model trained with cross-group penalties that result in improved OOD generalization, and further improve performance by factoring in useful contextual information.
CONCLUSION
In this paper, we empirically investigated the effectiveness of online black-box adaptation methods for label-shift when a key underlying assumption of invariant class-conditional input distributions is broken. We found that while existing methods can be effective to an extent regardless conditional- shift, performance confusion matrices on OOD validation sets, and learning additional scaling prper-parseeters the output adjustment step account for shifting distributions.
be improved by adopting intuitive heuristics in particular, estimating

---

Under review as a conference paper at ICLR 2023
REFERENCES
Faruk Ahmed, Yoshua Bengio, Harm van Seijen, and Aaron Courville. Systematic generalisation with group invariant predictions. In 9th International Conference on Learning Representations (ICLR), 2021.
Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation. In International Conference on Machine Learning, pp. 222-232. PMLR, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. CoRR, 2019.
Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. arXiv preprint WrXivv:903.097.. 2019.
Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iwildcam 2021 competition dataset. arXiv preprini arXiv:2105.0344 034444 2021.
Joseph Paul Cohen, Tianshi Cao, Joseph D Viviano, Chin- Wei Huang, Michael Fralick, Marzyeh Ghassemi, Muhammad Mamdani, Russell Greiner, and Yoshua Bengio. Problems in the deploy- ment of machine-learned models in health care. CMAJ, 193(35)E11311E1139, 2021.
Corinna Cortes and Mehryar Mohri. Domain adaptation in regression. In International Conference Algorithmic Learning Theory, pp. 308 1833..3.2350 Springer, 2011.
Corinna Cortes and Mehryar Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519:103-126, 2014.
Morris H. DeGroot. Optimal Statistical Decisions, chapter 9, pp. 185-189. John Wiley Sons, Ltd, 2004.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary C Lipton. A unified view of label shift estimation. arXiv preprint arXiv:2003.07554, 2020.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0253/ 2(7), 2015.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021.
David Krueger, Ethan Caballero, foern-Hente Jacobsen, Amy Zhang, Jonathan Binas, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). CoRR, 2020.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 6028-6039. PMLR, 2020.
Tsung- Yi Lin, M. Maire, Serge J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. ArXiv, abs/1 401.401501000 2014.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3122-3130. PMLR, 10-15 Jul 2018
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. ICLR, 2021.
Kevin P Murphy. Conjugate bayesian analysis of the gaussian distribution. https: //ww www cs ube. ea/ http://mnhyyt//peerra//aeauuuuue pdf 2007. [Online; accessed 19-January- 2022].
10

---

Under review as a conference paper at ICLR 2023
Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure. Neural Comput., 14(1):21-41, 41, jan 2002. ISSN 0899-7667.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ICLR, 2020.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. Computer Vision - ECCV 2016 Workshops, pp. 443-450, 2016.
Theo Vos et al. Global burden of 369 diseases and injuries in 204 countries and territories, 1990-2019: a systematic analysis for the global burden of disease study 2019. Lancet, 396( 196(10058):1204--22 1222, 2020.
Dequan Wang. Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test- time adaptation by entropy minimization. In International Conference on Learning Representations, 2021. /RLLAtts:///ppeeeee
net/ forum? dinetiddddxxia0 3b2kkk
Ruihan Wu, Chuan Guo, Su, and Kilian Q Weinberger. Online adaptation to label distribution shift. Advances in Neural Information Processing Systems, 34, 2021.
Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature communications, 11(1):1-11, 2020
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet. and Zhikun Wang. Domain adaptation under target and conditional shift. In Proceedings of the 30th International Conference on International Conference on Machine Learning Volume 28, 2013.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017.

---

Under review as a conference paper at ICLR 2023
A POSTERIOR UPDATE
We derive the posterior update equation (Eq. 13 specifying the conditions under which this rule holds. The key assumption is that in the new deployment location, categories are encountered in an IID manner in the location, i.e, yj IL yk--
Po( P( V1, , ,V)):
(21)
P(yi,'., Vt () P(ò)
(Bayes rule)
(22)
P(y .80)
x P(yn." :y 6) P(Ø),
(dropping terms independent of ()
(23)
= III P(u I () P(o) i=1
(using assumption yj TL ykk)
(24)
1-1
= -P(ye (b)
II 1=1
P(y: () P(t)
(regrouping terms)
(25)
P(y 6) Pt-1(0),
(by definition)
(26)
B REGRESSION MODEL
B. FINDING THE OPTIMAL SOLUTION FROM THE PREDICTIVE RULE
The required distributions are defined as
(-
(-
dx (y f(x(
Ply x) x exp
(27)
2
2n+1
L (y - 12)
pnew x
|+
-1)2
(28)
2a
Au (y (y-my m))
)2 2)
P(y) x exp
(29)
2
(30)
which gives us the objective J - log p(y x) expressed as
J= - P(y xz- - -lo pnew (y) + logg P(v)
(31)
r (
Ay
2a +
1
L 2a
y m
+
log 1 +
(y 11)²)
(32)
2
2
2
The derivative of this objective wrt is
20+1 1 L 2,y 11
OJ
Az f(x)) - Ay(y mm) +
(33)
0y
1 L (y - 11)2
2a
(2a= + 1) L (y - 14)
-o - -(())) Ay m) +
2a
(34)
1 + L (g) n1)2
2a
M
L
(2a + 1)
(y 11)
=
-by) y + (2ym - fx)))))
1+ L (y
2a
2
(35)
Ta
TM
AM (y- - 11)
Tay + Tp +
(36)
+ M 11)2
Setting to zero, we have
= 0 Tay + Tm) (1+ ((0--11)) + +MM(y - -M =
(37)
12

---

Under review as a conference paper at ICLR 2023
Tay + Ta) 1 My² + Mp² 2M + AM - 11) =0
(38)
Tay + MTayI + M M - pTa + +Tu + MT + MT 2M + AMy - AM 0
(39)
MTay" + (MT 2M + (Ta My²ra 2M + AM)y + (Tp + MT AM O (40)
which is the equation we shall solve for y. We use NUMPYS polynomial solver to find roots. cubic equation either has one real and pair of conjugate imaginary roots, or all real roots. We test the real solutions for a positive curvature (implying local minima), and pick the minima resulting in smallest value of the objective J.
B.2
SECOND DERIVATIVE TEST FOR SOLUTIONS
The second derivative of J is given by
2AM2 - 11)2
AM
Td
(1 + Myy 2))))
1 M - u2)
(41)
Writing y μl D, we have
AM
22
AM
2MD²
AM(1 - MD)
1
(1+ MD3)
(1 + MD2Y2
1 MD2
1+ MD2
(1+ MD232
(42)
When this expression is positive, we have local minima.
For the first term to be positive, we require that PO > O. which has straightforward intuitive interpretation: Tu > Ty. i.e. output precision should be higher than marginal adjjsstmttt precision. This is reasonable condition which expect to be fulfilled, since we typically expect to rely more strongly on the underlying predictive model than simply the marginal.
In the second term, AM is always non-negative, for positive pseudo-count. The denominator is always positive. Substituting expressions for the values after the 1-th update, we have
m2)
MD
Ki+1
(43)
(0++1 24)
When this term is S 1, we are guaranteed positivity (strictly speaking, Ta provides the second term with some room for negative values, but we ignore it for simplified reasoning). This condition implies
Rit
R7 Kr +1
(y - n=)²
(0r+1 -2)²
(44)
which then implies that the following range for y allows local minima
t-1 Kr
+
Kr
Kt
16r
fe
(0r+1- 1, < S me
(0r+1 22)
(45)
+
An intuitive interpretation of this condition is that valid updates are allowed within an increasing range as function of the total observed variances up the 1-th test example. In practice, find that validation tends to pick values for > Ty, and that the case for no-local- lololl-miiiiaa typically does arise for the optimal hyper- parameters our experiments.
B.3 INITIALIZING PRIORS
For initializing priors, we might endeavour to stay unbiased, since we assume that deployment locations can have significantly different target distributions than we might anticipate from the marginal over the training set. For classification, built this in by using uniform pseudo-count for all classes and sources. For regression, we simulate a pseudo-count of uniform samples from the output range.
13

---

In this view, & corresponds to the pseudo-count (as per the interpretation of the parameters of the Normal-Gamma conjugate prior as in Murphy (2007)) ) a is defined in terms of K. To improve stability, we will set to the middle of the output range rather than actually estimate the mean of our Under review as a conference paper at ICLR 2023
If we start with reference prior for the Normal-Gamma distribution with parameter settings
# 0, -0.5, 3 0,
(46)
then after observing N data-points (yi, UN ~ U[, H] (the uniformly sampled points we will simulate), the resulting posterior is
N
V/+
(47)
N 1=1
K = N,
(48)
N --1
(49)
2
N
8
(3)- u1)
(50)
2
uniform pseudo-samples. Likewise, we will 3 by estimating its value as function of and using the expression for variance of a uniform distribution,
(rr Var(w)) (s- (1) (H L)
B
(51)
24
C EXPERIMENTAL DETAILS
C.I SYNTHETIC MNIST
The splitting of digits into two sets is performed by observing mis-classification matrices after 200 iterations of training neural network averaged across 100 runs digits are put into opposing sets if they tend to be confused, while also trying to keep the set-sizes balanced.
The network architecture consists CONV layers with 64, 128 and 256 channels, each followed by MAXPOOI,, BATCHNORM, and RELU. After the third layer, we spatially mean-pool activations and use a linear layer to map the logits. A weight-decay of 5e Training is conducted for 20 epochs with batches of size 256 where training accuracy saturates to 100%. An initial learning rate of 0. is used, which is cut by at the 6-th, 12-th and 16-th epochs.
is applied on all parameters.
The datapoint-counts in the train/val/tess environments are follows.
Train
red
4889
5614
4915
38
49 4810
4664
57
59
41
4946
cyan
53
5116
4801
1052
985
10
949
IID validation
12
1023
973
11
955
1082
997
12
OOD validation
eyan
687
714
689
313
304
635
310
315
301
664
red
304
350
301
717
679
280
657
775
708
297
980
1135
1032
0
892
0
1009
OOD Test
0
1010
982
958
1028
974
C.2 SYNTHETIC GAUSSIAN
The synthetic data for this experiment is generated with the following function
((x) = 10N(s) zi 2, 0.5) + 3N T: = 2, 0.5) + 6y z; 0,a = 1)
Training points: Training points are sampled from two regions on the axiss x ~ ~N(22.0.4) and (22, 0.2), with 250 points each.
14

---

Under review as a conference paper at ICLR 2023
OOD validation points: OOD validation points sampled from W 0.2) and (1,, 0.2), with 250 points each.
OOD test points: OOD test points sampled from (0 0.2) and (3,, 0.2), with 250 points each.
For OOD sets, the different sampling distributions correspond to different locations. For different trials, repeat the whole experiment from scratch, sampling new training, validation, and test sets, and performing validation every time.
The network architecture is layer MLP with 128 hidden units, with BATCHNORM and RELU after hidden activations. weight decay le is applied on all parameters. We train for 100 epochs with batch-sizes of 100, with SGD) Momentum (0.9), starting with initial learning rate of 0.01 and scaling it by 0.95 after every epoch.
We include the non-aggregated MSEs below confirm that there consistent improvements every base model/data-sampling individually.
Seed
IID-Base
OOD-Base
OOD-Online
0 0.08
11.23
3.14
1
0.13
12.37
3.82
2
0.16
6.13
3.00
3
0.19
9.14
5.50
4
0.21
7.00
6.31
C.3 SYNTHETIC
We chose the following objects for this synthetic classification task: bicycle, train, cat, chair, horse, motorcycle, bus, dog, couch, and zebra; and the following scenes to simulate different sources.
Training: beach, canyon, building facade, desert/sand, iceberg
OOD validation: house, orchard, crevasse, ball pit, viaduct OOD test:
toter, staircase, waterfall, bamboo forest, zen-garden
When there are multiple instances class in an image, we pick the instance occupying largest area, such that only images with objects occupying at least 10K pixels are retained. All images resized to 256 x 256.
Across the 5 sources, the number of examples for training, validation, and test sets follows.
Table 3: Training set
bicycle
train
cat
chair
horse
motorcycle
bus
dog
couch
zebra
beach canyon
669
429
176
46
0
135
513
513
329
35
0
building_facade
5
34
132
322
503
503
322
132
34
5
desert/sand
0
6
35
135
329
513
513
329
135
iceberg
0
0
0
7
46
176
429
669
669
Table 4: Validation sets
bicycle
train
cat
chair
horse
motorcycle
bus
dog
couch
zebra
beach
167
167
107
44
11
0
0
0
canyon
33
82
128
128
82
33
8
1
()
0
building facade
33
80
125
125
80
33
8
1
desert/sand
1
8
33
128
128
82
33
iccherg
0
0
44
107
167
167
15

---

Under review as a conference paper at ICLR 2023
Table Test sets
bicyclc
train
cat
chair
horse
motorcycle
bus
dog
couch
zebra
beach
401
401
257
105
27
4
0
0
0
0
canyon
81
197
308
308
197
81
21
3
0
building_faad facade
79
193
302
302
193
79
20
3
desert/sand
3
21
81
197
308
308
197
81
iccberg
0
0
0
4
27
105
257
401
401
Note that the pattern of label-shift is the same across validation and test subsets (albeit of smaller size). This proof-of-concept experiment is intended as middle-ground between the COLORED MNIST and WILDS-IWILDCAM experiments, in the potential learning hyper-parameters to account for conditional shift is tested while keeping label-shift pattern fixed).
We train for 400 epochs with SGD + Momentum (0.9), using batch sizes 128, with an initial learning rate of 0.1 which is cut by at the 240th, 320th, 360th epochs. An L2 weight decay regulariser is applied on all parameters with a coefficient of 5e-4. We normalize images with the training set mean and standard deviation per channel, and apply data augmentation of random erops to 224 x 224 and random horizontal reflections.
D HYPER-PARAMETER SELECTION
We contrast performance when methods use IID validation sets vs. OOD validation sets vs. the test itself, Table 6. We observe that, generally speaking, OOD validation can improve over IID validation.
E IDENTITY APPROXIMATION FOR CONFUSION MATRIX
Degenerate confusion matrices can arise when there are missing categories in the validation set used to compute (leading to zero-rows), or if two or more rows are exactly the same (for example, when multiple rare categories both get categorized the same way). Two options are to use soft-confusion matrix, or pseudo-inverse (Lipton et al., 2018). Since /WILDCAM dataset is significantly long-tailed with large number of classes not represented in the validation sets, end up with a number of zero rows for the soft-confusion matrix. For such rows, we simply placed a in the diagonal element.
In Table 7 we find these alternatives to result in degraded performance for IWILDCAM, generally much worse than our identity approximation. We hypothesize that part of the reason is to do with the fact that both our zero-confusion heuristic for dealing with missing classes for the soft-confusion matrix, well the same underlying effect being applied by the pseudo-inverse results in misleading effect: rare classes, absent from validation sets, are in fact more likely to be confused than the frequent ones. This is one possibility for why the less presumptive identity approximation performs better. The inherent difficulty is estimating robust confusion matrices has been recognized the literature, with the typical approach being to hold out significantly large validation sets in order to reliably estimate less noisy confusion matrices. In Table 8, we include numbers from an identity approximation in the synthetic datasets where the confusion matrices were invertible.
On the whole, we suggest to practitioners that in difficult, real-life situations, simpler approximations might continue serve us well, while more sophisticated methods can pose specific requirements to be successful.
F HYPERPARAMETERS. COMPUTE, AND CODE AND DATA LICENSES.
The hyper-paraae pareeerrrameterss involved are the two calibration terms Xus Ay and the pseudo-count term du for classification, and Xy, Ay, K for the regression problems. These were picked via grid-search on the OOD validation sets, optimizing for OOD performance in all cases. For OGD methods, additional
16

---

Under review as a conference paper at ICLR 2023
Table (top) Classification problems: Performance when picking hyper-parameetes IID, OOD validation sets, or on (Oracle) test sets. (bottom) Regression problems: Performance when picking paper-parameters on IID, OOD validation sets, or on (Oracle) test sets. For MIX-OF- MIX-OF-GAUSSE GAUSSIANS, we use mean squared enror as the metric (lower is better), while for POVERTYMAP the metric is the Pearson's 's correlation co-efficient (higher is better).
Datasets
Methods
IID validation
OOD validation
Oracle
FTH-H
82.67 + 1.79
98.69 0.30
98.69 + 0,30
OGD
82.75 + 1.77
95.75 = 0.70
95.75
S-MNIST
OGD-H
82.59 + 1.82
98.91 0.20
98.91 0.20
FTH-H-B
83.00 + 1.79
97.46 0.64
98.35 + 0.52
FTH-H
57.42 0.53
57.81 0.74
59.05 = 0.53
OGD
57.72 + 0.31
57.75 = 0.29
57.75 = 0.29
S-COCO-ON-PLARR
OGD-H
57.31 0.68
57.12 0.15
58.10 + 0.85
FTH-H-B
58.59 + 1.02
58.42 + 0.49
59.37 + 0.89
FTH-H
73.52 3.36
73.75 3.77
74.13 3.54
OGD
69.42 5.10
73.11 3.05
73.16 3.15
IWILDCAM (AVG)
OGD-H
73.41 3.42
73.36 3.51
73.53 3.29
FTH-H-B
73.90 + 3.93
74.10 + 3.56
74.41 3.65
FTH-H
31.93 + 1.56
32.46 = 0.31
33.81 0.30
OGD
29.37 2.15
32.49 0.41
32.72 + 0.06
WWILDCAM (F1)
OGD-H
32.09 + 0.29
31.36 + 0.41
32.72 + 0.15
FTH-H-B
32.73 + 2.78
33.33 1.31
33.33 + 1.31
Datasets
IID validation
OOD validation
Oracle validation
MIX-OF-GAUSSIANS
9.24 + 2.76
4.35 + 1.48
1.76 + 0.59
POVERTYMAP-A (ALL)
0.80 0.00
0.84 + 0.00
0,84 + 0.00
POVERTYMAP-B (ALL)
0.82 0.00
0.82 + 0.00
0.83 + 0.00
POVERTYMAP-B (ALL)
0.82 0.00
0.83 + 0.00
0.83 + 0.00
POV (ALL)
0.78 0.01
0.77
0.78 + 0.00
POVERTYMAP-B (ALL)
1.5:
0.75 + 0.00
POVERTYMAP-A (WG)
0.43 0.00
0.43 0.00
0.45 + 0.02
POVERTYMAP-A (WG)
0.33 + 0.03 0.03
0.50 0.01
0.52 + 0.00
POVERTYMAP-A (WG)
0.50 0.01
0.56+0.00
0.58 + 0.02
POVERTYMAP-A (WG)
+ 0.04 0.04
0.56 + 0.01
0.57 + 0.02
POVERTYMAP-A (WG)
0.36 = 0.02
0.37 1 0.00
0.37 1 0.00
hyper-parameter is the learning rate used for updating This learning rate is searched over range from 1e-8 10 in steps of 10.
V100 GPUs were used to train base models (in cases where we trained our own models), and the online adjustment experiments were performed on an Apple Macbook Air with saved outputs from the models.
We reused code from http::://gttuuu.cmmm//o..
released under the MIT wwleeee_atptiooo lalooontttaeeeeeeeeeeeeeeeeeeeee
License, and code from battps://gil www.g.thuuu..o... aecom/whh14 iesstributiionnhhiiie
publicly released by Wu al. (2021). We also used data from MS-COCO, released under the CREATIVE COMMONS ATTRIBUTION 4.0 LICENSE. WILDS- IWILDCAM is under COMMUNITY DATA LICENSE AGREEMENT - PERMISSIVE V1.0, and the TILDS-POVERTYM... data is U.S. PUBLIC DOMAIN (LANDSAT/DMSSP/VIII))
17

---

Under review as a conference paper at ICLR 2023
Table We compare use of a soft-confusion matrix and the pscudo-inverse with our approximation with identity matrix for IWILDCAM. We find that FTH performance drops strongly, and for OGD, the optimal learning rate most often zero, leading to no differences with base performance. For OGD, we find the optimal learning rate the test-set for all choices of confusion matrix, reporting best-case performance.
Dataset
Method
Soft confusion matrix
Pseudo-Inverse
Identity
FTH (C-IID)
43.41 + 21.80
37.23 = 19.34
71.41 + 4.91
FTH (C-OOD)
34.56 + 16.71
28.20 = 13.74
71.41 = 4.91
IWILDCAM (AVG)
OGD (C-IID)
73.10 + 3.26
73.29 3.04
73.16 1 3.33
OGD (C-OOD)
73.10 + 3.26
73.10 3.26
73.17 3.18
FTH (C-IID)
22.42 + 4.33
11.33 0.26
29.57 0.93
FTH (C-OOD)
23.73 3.36
10.82 4.64
29.57 0.93
TWILDCAM (MACRO-FI)
OGD (C-IID)
32.71 + 0.18
32.70 0.16
32.75 2
OGD (C-OOD)
32.71 + 0.14
32.70+
+.16
Table 8: Identity approximation with S S-MNIST and S-COCO-ON-PLACEE with test-time perfor- mance using the original confusion matrix Cý for reference. When using the identity approximation, OGD (IID) uses the IID validation set to estimate C, and OGD (OOD) uses the OOD validation set.
Dataset
Method
Identity approximation
Original
FTH
96.02 +: 1.07
96.04 +:2 1.03
S-MNIST
OGD (IID)
89.47 + 1.96
88.32 2.06
OGD (OOD)
95.70 + 0.68
95.75 = 0.70
FTH
59.27 + 0.64
58.94 + 0.63
S-COCO-O0 -ON--ON--LLCSS
OGD (IID)
57.48 + 0.52
57.37 0.51
OGD (OOD)
56.02 + 0.35
57.75 + 0.29
18


Reviews:
[{"review_id": "qq8jgLMvhmL", "reviewer": "Reviewer_eNZr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This draft considers the problem of online learning with label shift in the presence of additional conditional shift. In addition to the change in the class-priors $\\Pr[y]$, the posterior probability $\\Pr[x|y]$ can also change over time. Based on the previous work of Wu et al. (2021), the authors propose three heuristics to improve its empirical performance for both classification and regression tasks when the additional conditional shift appears. In their empirical studies, the authors suggest 1) using the OOD validation set instead of the ID validation set when estimating confusion matrices (a key component in the work of Wu et al. (2021)), and 2) adding scaling hyperparameters to the original loss function to improve the performance. ", "summary_of_the_review": "This draft considers the problem of online learning with label shift. The authors propose three heuristics to handle the additional conditional shift based on the previous work of Wu et al. (2021). However, the empirical studies do not show significant improvements over the original approach. The proposed approach lacks theoretical support, and its proposed heuristics lack intuitive support, as well.", "strength_and_weaknesses": "Strength:\n+ This work studies a well-motivated problem, which covers many real-world applications.\n\nWeakness:\n- The authors introduce three heuristics to improve the empirical performance of the algorithm proposed by Wu et al. (2021) when an additional conditional shift appears. There are, however, no significant or distinguishable improvements over the original method based on their empirical studies. For example, in Table 1, the proposed algorithm's performance on S-COCO-ON-PLACES and IWILDCAM (Avg.) is nearly the same compared with the original algorithm, and even worse than the original one in IWILDCAM (F1).\n- Heuristic 3 (adding a tunable scalar to the diagonal and renormalizing rows to the confusion matrix) is a common practice to avoid the non-invertible problem.\n- A discussion of the intuition behind the heuristics is suggested.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "irwCZ1KaL0i", "reviewer": "Reviewer_s3Qi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how to adapt a black box to the testing distribution in an online fashion under the label shift condition. The main contributions are to propose several heuristics to improve the algorithm proposed by Wu et al., (2021) when the label-shift assumption is broken, or the confusion matrix is non-invertible. Empirical studies are conducted to validate the effectiveness of the proposed methods.\n", "summary_of_the_review": "This paper considers how to improve the previous work [Wu, et al., 2021] to learn beyond the label shift assumption. This is an interesting problem, but the proposed method is somewhat unconvincing to me as it requires an additional validation dataset and the reweighed classifier still implicitly relies on the label shift assumption. Besides, the background, problem setup, and method parts of the paper are not clearly written. Given the above concerns, I tend to reject this paper.", "strength_and_weaknesses": "### strength\n+ This paper considers an interesting problem on how to relax the label shift assumption made in the previous work.\n+ This paper extends the online label shift problem to the regression setting.\n### weaknesses:\n- About the written quality: Although this paper is well-organized, some parts of the presentations, particularly the algorithm design part, are not totally clear. The unclear parts are listed as follows.\n\t- about the background: the notation $f$ in Eq. (2) and Eq. (4) are very confusing. In Eq. (2), $f$ is used for an underlying base model, but Eq. (4)  uses $f$ for the model to be learned. \n\t- about Heuristic 1: it is unclear to me how the validation set is collected. Does the learner collect the validation data once at the beginning of the testing online test stage, or do the data just appear in an online fashion? The former one seems less promising in the online adaption problem since the underlying distribution $P_t$ could be different for every iteration.\n\t- about the Bayesian methods: the notation $\\hat{y}_\\tau$ in Eq. (9) and $y_t$ in Eq.(13) is not defined, though I can guess they are the pseudo-label and the true label. It is unclear to me why Eq. (13) updates with the true label while Eq. (9) updates with the pseudo-label. It is a very strong requirement to obtain the true label for each iteration.\n\n- About the soundness of the proposed heuristics:\n\t- about the validation set: the main difference between this work and Wu et al., 2020 is that the latter does not require the validation set. When a validate set sampled from the testing distribution is available, a strong baseline is that we can just estimate $\\hat{\\mathbf{q}}^{\\mathrm{new}}$ by the labeled validate set. I think it would be necessary to compare the proposed method with such a baseline.\n\t- about Heuristics 1: Heuristics 1 aims to solve the problem when the label-shift condition is broken. But the proposed methods are still based on the reweighted classifier Eq (2). Such a kind of reweighting mechanism still crucially relies on the label shift assumption. In this sense, I believe only the adjustment on the estimation of the confusion matrix is not sufficient.\n\t- about Heuristic 3: it seems that the use of an identity matrix instead of the confusion matrix will sacrifice the unbiasedness of the gradient estimator. I am not sure whether such a method can perform well when the confusion matrix is invertible. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "amCPlH_wBx", "reviewer": "Reviewer_iFod", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Using standard label shift handling techniques/equations, it is proposed to handle additional manifestation-shift (p(x/y) shift) by simple heuristics based on additional hyperparameters (like in (8)) or using Bayesian models over labels etc. (19). Empirically it is shown that the proposed heuristics improve over those in Wu et.al.'21.\n", "summary_of_the_review": "This seems to be a work in preliminary form with many missing details and justifications. Hence I tend to not accept the paper.", "strength_and_weaknesses": "Weakness:\n1. The writeup does not seem to be self contained and the reader may have to refer to Wu et.al.'21 to understand some background.\n2. Some parts may require more details. For example, how is p estimated in (8) ? Do the additional hyperparameters effect p's estimation?\n3. While the Bayesian model equations are standard, the connection of it for the problem at hand in section 4 is not clear to me. Perhaps some re-writing might help here.\n4. Apart for some cryptic justification, none of the propositions seem to have theoretical justifications. This makes it very hard for me to evaluate the work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Ex3nvEICZOl", "reviewer": "Reviewer_9gjn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "It is well known that the performance of machine learning models is highly dependent on the distribution of the data on which it is evaluated: model performance deteriorates when tested on data generated from a distribution shifted with respect to the training data generating process. Identifying and mitigating the effects of distribution shifts is a major open challenge for machine learning practitioners, as distribution shifts are ubiquitous in an ever-changing world. In the supervised learning context, evaluating test performance and mitigating it usually require labelled testing data, which is often difficult or impossible to obtain. \n\nLittle can be done about arbitrary distribution shifts – generalization from training to test data is only possible if the shift leaves some structure in the data unchanged. Label shift is a basic example of such a distribution shift, where the conditional probability P(X|Y) remains fixed, and only P(Y) changes. Here X are the covariates, Y the label, and P(X,Y) = P(X|Y)P(Y) is their joint distribution. Recent years saw much progress with the analysis of label shift, and methods have been developed to mitigate its impact on black box models – with deep learning a primary application -- in both offline and online settings. Essentially, these methods rely on re-weighting model predictions using the distribution of predicted (pseudo-)labels, and thus do not require true labels for the test data. \n\nThe current paper follows three goals related to label shift adaptation:\n\n1. The paper’s main effort focuses on examining how previously proposed label shift mitigation methods perform on shifted distributions do not satisfy the label shift condition – a scenario highly relevant to real-world applications, where often as pure label shifts are rare. In an online learning setting, albeit one in which the distribution does not shift continuously, the paper examines empirically how recently proposed algorithms for online adaptation to label shift perform on a few synthetic and realistic datasets that exemplify different kinds of “non-label” distribution shift. The empirical investigation also considers a couple of heuristically-motivated extensions to these algorithms, most notably performing model selection on an OOD validation set which is shifter with respect to both the training and test sets. The findings of these investigations are not clear cut, but suggest that in some cases, the proposed algorithms provide an improved adaptation to the distribution shift. The takeaway is that label shift adaptation methods (or some heuristic generalization thereof) might sometimes be useful to mitigate general distribution shifts, even if this practice has no known theoretical justification.\n\nThe paper considers two further issues related to label shift adaptation:\n\n2. Past work on label shift adaptation has mostly focused on classification problems. The paper proposes an algorithm for label shift adaptation in regression settings, and studies it empirically.\n3. Past algorithms for online label shift adaptation require the inversion of an empirically measured confusion matrix. The paper suggests a heuristic fix for the case when this matrix is non-invertible and studies it empirically. \n\nThe latter two issues are discussed briefly (compared to the main topic of the paper), and here too the investigations do not provide clear cut conclusions on the efficacy of the proposed methods, but in some cases these methods perform better than the baseline.\n", "summary_of_the_review": "The work presented in this paper is novel, seems technically correct, and addresses a key problem to many real-world scenarios. I believe that the work in its current state with some corrections/improvements could and should merit publication in some venue. However, with the flaws described above, I do not believe this paper is ready for publication. ", "strength_and_weaknesses": "Major Strengths: \n\n1. The problem investigated is well motivated. Distribution shifts are indeed a big and relevant problem when machine learning models are deployed in the real world. Much of the work to date has focused on idealized types of shifts, like label or covariate shift. It is natural to wonder how much methods developed for idealized shifts might be useful in more realistic settings. Furthermore, if label shift adaptation methods generalize to realistic shift scenarios, they are attractive from a practical standpoint, as they do not require labelled test data. \n2. Empirical results are, for the most part (except for some comments below), clearly presented: I could understand what was done and believe I have enough information to attempt to reproduce the results. \n3. The paper is quite honest about the inconclusive nature of much of the results, and does not try to oversell the proposed methods. \n\nMajor Weaknesses: \n\n1. A systematic or principled approach to the types of distribution shifts considered is missing. Distributions can shift in many ways and for many reasons. Adding conditional shift to label shift is tantamount to considering general distribution shifts. Indeed, the paper considers two examples with no label shift (P(Y) is not changed in the synthetic MNIST and COCO-on-Places datasets), an example with covariate shift (Mixture of Gaussians), and two general distribution shifts (from the WILDS dataset). Framing the issue as “label shift in the presence of conditional shift” might give a wrong impression that the conditional shift is a perturbation of the label shift condition. I find it clearer to state that general distribution shifts are considered. \nLittle can be said about distribution shifts in general, without focusing on particular types or characteristics of the shifts, such as label/covariate shift, subpopulation shift [6], or shifts where the data generating process has a fixed known causal structure [3]-[5]. Since experiments in the paper do not belong to a particular type of shift, it is hard to compare results or to generalize from them to general shifts. \nThe lack of a systematic approach to general label shifts is reflected also in the absence of discussion of relevant work on this issue, including refs [1]—[6].\n\n2. Given the vast scope of possible distribution shifts, with no systemic understanding of how they relate to or differ from label shift, and with heuristic methods lacking a theoretical foundation – given these, a major and comprehensive empirical study is necessary in order to ascertain the usefulness of the proposed methods. The paper offers modest experiments, in terms of types and strengths of shift, types of data, and alternative baselines/methods. This severely limits the usefulness of the results, as it is unclear when the suggested methods can be expected to improve upon baselines, and how good such improvement are compared to alternative methods. As it stands, few generalizable insights can be drawn from the empirical scope of the paper. The paper itself is honest about the modest and tentative nature of the findings, when it concludes that the experiments are “suggestive” that the proposed methods show “promising trends for the most part” in the limited scope in which they were tested. \nConcretely, for the experiments performed, here are some suggestions of baselines/methods that might provide a wider context for obtained results: \n  a. An optimal fixed classifier, as considered by Wu et al. (2021).\n  b. Results obtained from offline domain adaptation methods (Garg et al., 2020).\n  c. Results obtained from known domain generalization methods such as those mentioned in the related works section of the paper, or the ones surveyed by Gulrajani & Lopez-Paz (2020). In particular, if I understand correctly, CORAL was used for the two WILDS datasets considered in the paper, but not the others. It might be more informative to test all datasets with and without CORALS (and/or other domain adaptation methods). \n  d. The paper emphasizes the importance of the use of an OOD validation set. It would thus be useful to test the effect of this OOD validation set on test performance by considering the effect of different validation sets, preferably with different characteristics. For example, for the synthetic colored MNIST dataset, one could use validation sets that are more or less correlated with the test sets. \n\n3. Goals 2+3 above are not explored in detail in the paper. No references are given to prior work on regression label shift / domain adaptation (e.g., [7]-[8] below), nor to the discussion in Lipton et al. (2018, section 7) about remedies to non-invertible empirical confusion matrices. The corresponding experiments provide only an initial investigation into them. The paper provides some interesting but embryonic discussion/exploration of both. Their inclusion in the current form of the paper \n\n4. Some key definitions and explanations are lacking in the paper, making it difficult to understand some sections of it.\n  a. “Conditional shift” is not defined. While it is a term used in the literature and whose meaning might be intuitive, many other terms are used in the literature as well. To guarantee that there are no misunderstandings regarding this central concept, its definition should be provided. \n  b. Method FTH-H-B and FTH-H-B (R) are never clearly defined (what is the “pseudo-count hyper-parameter” mentioned? I did not understand).\n  c. In equation (3), the definition of the expected error rate \\ell^{\\test{new}} is only given in words, not in a formula. \n  d. In section 4.1, what are a, b, kappa, and mu?   \n  e. In appendix A, none of the notation is defined, and in fact no information is given about the context and goal of the derivation there. \n\nFurther comments\n\n1. Online vs offline methods. The scope of label shifts considered in this paper is more limited than those considered by Wu et al.: here only constant shifts are considered (test data is drawn from a fixed shifted distribution), whereas Wu et al considered distributions that keep changing throughout training. An important strength of online methods are their ability to deal with continual changes. Considering only constant changes reduces (but does not invalidate) the usefulness of online methods compared to offline ones. The decision to focus on online methods should be motivated in the paper. \n\n2. OOD validation: the concept of OOD validation is introduced in Heuristic 1 without being properly defined/explained. As this is a central tenet in the proposed methods, the idea and procedure should have a clear and detailed explanation. Furthermore, in Heuristic 1 it is written that OOD validation is a standard practice of model selection, with a reference to Gularjani & Lopez-Paz (2020). As far as I can tell, this reference (which emphasizes the importance of validation set details in the context of domain generalization) does not advocate the use of validation on a separate OOD set. Rather, it attributes this method to Krueger et al. (2020), who indeed mention it in an appendix. \nRegarding the method itself OOD validation itself: why should it work? I can understand that it might be useful when the shifts in the validation and test sets are somehow related (like the Skewed-MNIST example where test is a more severe shift of the same type as validation), but why would it help in examples like the mixture of Gaussians, or the WILDS datasets? Looking at the experiment results, it indeed seems to me that OOD validation is helpful only for the skewed-MNIST example. If my reading is correct, this should be stated clearly, and the appropriate qualifications should be made in the conclusions about the merits of OOD validation. Currently, section 5.3 states that “Using OOD validation sets … improves results on the whole” – but for S-COCO-on-Places and iWildCam (Avg) I do not see any improvement more significant than the noise level, and for iWildCam (F1) there is a small deterioration (which is also consistent with noise). \nFrom a practical perspective, performing OOD validation is not always possible as it requires more labelled data – it would be useful to emphasize this fact. Technically – what are all the optimization steps performed on this validation set? I.e., which hyper-parameters are calculated on this validation set, other than the confusion matrix? \n\n3. Non-invertible confusion matrices. The methods proposed in Heuristic 3 surely generate invertible matrices, but why would they be expected to work for label shift and general distribution shift adaptation? They seem to me ad-hoc and unmotivated. What would be their merit compared to using a pseudo inverse, or the soft probability matrix suggested by Lipton et al. (2018)? \n\n4. Section 4: The role of this Bayesian discussion is not clear to me. What insights are gained from this Bayesian perspective? Are these insights relevant also to cases of pure-label shift, or only general distribution shifts? I found the discussion around equations (11)-(14) confusing on first reading. The notation in equations (11)-(12) is confusing, perhaps Y|\\phi ~ Cat(\\alpha) and \\phi ~ Dir(\\alpha). The notation in equation (13)-(14) – P_t(\\phi), P^{new}_{t+1} is not defined anywhere. \nI found the whole of section 4.1 confusing. How is the discussion related to label shifts in regression problems? What are the takeaways or results of this section? Are the results valid only for the Gaussian example with a conjugate prior, or more generally applicable? What kind of calibration is performed in this section, and why is it useful?\n\n5. Experiment details. Right before section 5.1: \n  a. It would be worthwhile to provide the details of “the surrogate loss implementation of Wu et al.” \n  b. What are the details of the grid search used for the parameter of OGD? On which validation set is it taking place.\n  c. Skewed-MNIST should reference the inspiration from color MNIST of Arjovsky et al. (2019). A table with the makeup (number of digits of each color) of each of the train/val/test datasets would be useful. It is stated that “Since the overall class frequencies are balanced … we drop the P(Y)”. Drop it from where (same comment for skewed COCO on Places)? Appendix C.1 describes how digits were split into two sets – was there a precise protocol for this? How is the “tend(ency) to be confused” measured context? What was the optimizer used for training  - SGD? \n  d. WILDS-iWildCam: it is stated that “We use Heuristic 3 for evaluating methods on this dataset. Heuristic 3 mentions several approaches: adding a tunable scalar to the diagonal? Using the identity matrix? Using a “pseudo-count”? \n  e. Table 2: How are the error estimates estimated relevant to all tables)? Why are the error estimates here +- 0? Are the quantities really measured to perfect accuracy? \n\nMinor comments\n\n1. Before equation (4): “where e is a one hot vector for the predicted category” – the description and notation there can be clarified: it was initially unclear to me which predicted category is referred to, and only after reading Wu et. al (2021) did I understand that these are calculated for each step I separately. \n\n2. After equation (4), it is stated that calculating the gradients is tricky. Why is it so? For self-containedess, the statement should be explained. Similarly, before equation (7) it is stated that FTH is more efficient than OGD – efficient in which sense? Compute time? Memory? Data complexity? \n\n3. Right before 5.1.3, it is mentioned that “test-sets are smaller”. Smaller than what? \n\n4. Typos: \n- Heuristic 1, line 2: shiftis -> shift is\n- Two lines below equation (19): minimum -> minima\n- Last line of page 6: there’s a superfluous ). \n- 5.2.2, last line of first paragraph, should read “in neither training nor validation sets for OOD test.”\n- The reference to Sun and Saenko (2016) is missing bibliographic info (journal name).\n- Appendix A: equations (23) and (24) seem to be the same\n\n\nReferences \n[1] Storkey, When training and test sets are different, in:Quinonero Candela et al., Dataset Shift in Machine Learning, 2009\n[2] Moreno-Torresa et al., A unifying view on dataset shift in classification  (2012)\n[3] Schoelkopf et al., On Causal and Anticausal Learning (2012)\n[4] Zhang et al., Domain adaptation under target and conditional shift (2012) \n[5] Kull and Flach, Patterns of dataset shift (2014)\n[6] Breeds: Benchmarks for subpopulation shift, Santurkar et al. (2020)\n[7] Cortes and Mohri, Domain Adaptation in Regression (2011)\n[8] Cortes and Mohri, Domain adaptation and sample bias correction theory and algorithm for regression (2014)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject