Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: It is well known that the performance of machine learning models is highly dependent on the distribution of the data on which it is evaluated: model performance deteriorates when tested on data generated from a distribution shifted with respect to the training data generating process. Identifying and mitigating the effects of distribution shifts is a major open challenge for machine learning practitioners, as distribution shifts are ubiquitous in an ever-changing world. In the supervised learning context, evaluating test performance and mitigating it usually require labelled testing data, which is often difficult or impossible to obtain. 

Little can be done about arbitrary distribution shifts – generalization from training to test data is only possible if the shift leaves some structure in the data unchanged. Label shift is a basic example of such a distribution shift, where the conditional probability P(X|Y) remains fixed, and only P(Y) changes. Here X are the covariates, Y the label, and P(X,Y) = P(X|Y)P(Y) is their joint distribution. Recent years saw much progress with the analysis of label shift, and methods have been developed to mitigate its impact on black box models – with deep learning a primary application -- in both offline and online settings. Essentially, these methods rely on re-weighting model predictions using the distribution of predicted (pseudo-)labels, and thus do not require true labels for the test data. 

The current paper follows three goals related to label shift adaptation:

1. The paper’s main effort focuses on examining how previously proposed label shift mitigation methods perform on shifted distributions do not satisfy the label shift condition – a scenario highly relevant to real-world applications, where often as pure label shifts are rare. In an online learning setting, albeit one in which the distribution does not shift continuously, the paper examines empirically how recently proposed algorithms for online adaptation to label shift perform on a few synthetic and realistic datasets that exemplify different kinds of “non-label” distribution shift. The empirical investigation also considers a couple of heuristically-motivated extensions to these algorithms, most notably performing model selection on an OOD validation set which is shifter with respect to both the training and test sets. The findings of these investigations are not clear cut, but suggest that in some cases, the proposed algorithms provide an improved adaptation to the distribution shift. The takeaway is that label shift adaptation methods (or some heuristic generalization thereof) might sometimes be useful to mitigate general distribution shifts, even if this practice has no known theoretical justification.

The paper considers two further issues related to label shift adaptation:

2. Past work on label shift adaptation has mostly focused on classification problems. The paper proposes an algorithm for label shift adaptation in regression settings, and studies it empirically.
3. Past algorithms for online label shift adaptation require the inversion of an empirically measured confusion matrix. The paper suggests a heuristic fix for the case when this matrix is non-invertible and studies it empirically. 

The latter two issues are discussed briefly (compared to the main topic of the paper), and here too the investigations do not provide clear cut conclusions on the efficacy of the proposed methods, but in some cases these methods perform better than the baseline.

strength_and_weaknesses: Major Strengths: 

1. The problem investigated is well motivated. Distribution shifts are indeed a big and relevant problem when machine learning models are deployed in the real world. Much of the work to date has focused on idealized types of shifts, like label or covariate shift. It is natural to wonder how much methods developed for idealized shifts might be useful in more realistic settings. Furthermore, if label shift adaptation methods generalize to realistic shift scenarios, they are attractive from a practical standpoint, as they do not require labelled test data. 
2. Empirical results are, for the most part (except for some comments below), clearly presented: I could understand what was done and believe I have enough information to attempt to reproduce the results. 
3. The paper is quite honest about the inconclusive nature of much of the results, and does not try to oversell the proposed methods. 

Major Weaknesses: 

1. A systematic or principled approach to the types of distribution shifts considered is missing. Distributions can shift in many ways and for many reasons. Adding conditional shift to label shift is tantamount to considering general distribution shifts. Indeed, the paper considers two examples with no label shift (P(Y) is not changed in the synthetic MNIST and COCO-on-Places datasets), an example with covariate shift (Mixture of Gaussians), and two general distribution shifts (from the WILDS dataset). Framing the issue as “label shift in the presence of conditional shift” might give a wrong impression that the conditional shift is a perturbation of the label shift condition. I find it clearer to state that general distribution shifts are considered. 
Little can be said about distribution shifts in general, without focusing on particular types or characteristics of the shifts, such as label/covariate shift, subpopulation shift [6], or shifts where the data generating process has a fixed known causal structure [3]-[5]. Since experiments in the paper do not belong to a particular type of shift, it is hard to compare results or to generalize from them to general shifts. 
The lack of a systematic approach to general label shifts is reflected also in the absence of discussion of relevant work on this issue, including refs [1]—[6].

2. Given the vast scope of possible distribution shifts, with no systemic understanding of how they relate to or differ from label shift, and with heuristic methods lacking a theoretical foundation – given these, a major and comprehensive empirical study is necessary in order to ascertain the usefulness of the proposed methods. The paper offers modest experiments, in terms of types and strengths of shift, types of data, and alternative baselines/methods. This severely limits the usefulness of the results, as it is unclear when the suggested methods can be expected to improve upon baselines, and how good such improvement are compared to alternative methods. As it stands, few generalizable insights can be drawn from the empirical scope of the paper. The paper itself is honest about the modest and tentative nature of the findings, when it concludes that the experiments are “suggestive” that the proposed methods show “promising trends for the most part” in the limited scope in which they were tested. 
Concretely, for the experiments performed, here are some suggestions of baselines/methods that might provide a wider context for obtained results: 
  a. An optimal fixed classifier, as considered by Wu et al. (2021).
  b. Results obtained from offline domain adaptation methods (Garg et al., 2020).
  c. Results obtained from known domain generalization methods such as those mentioned in the related works section of the paper, or the ones surveyed by Gulrajani & Lopez-Paz (2020). In particular, if I understand correctly, CORAL was used for the two WILDS datasets considered in the paper, but not the others. It might be more informative to test all datasets with and without CORALS (and/or other domain adaptation methods). 
  d. The paper emphasizes the importance of the use of an OOD validation set. It would thus be useful to test the effect of this OOD validation set on test performance by considering the effect of different validation sets, preferably with different characteristics. For example, for the synthetic colored MNIST dataset, one could use validation sets that are more or less correlated with the test sets. 

3. Goals 2+3 above are not explored in detail in the paper. No references are given to prior work on regression label shift / domain adaptation (e.g., [7]-[8] below), nor to the discussion in Lipton et al. (2018, section 7) about remedies to non-invertible empirical confusion matrices. The corresponding experiments provide only an initial investigation into them. The paper provides some interesting but embryonic discussion/exploration of both. Their inclusion in the current form of the paper 

4. Some key definitions and explanations are lacking in the paper, making it difficult to understand some sections of it.
  a. “Conditional shift” is not defined. While it is a term used in the literature and whose meaning might be intuitive, many other terms are used in the literature as well. To guarantee that there are no misunderstandings regarding this central concept, its definition should be provided. 
  b. Method FTH-H-B and FTH-H-B (R) are never clearly defined (what is the “pseudo-count hyper-parameter” mentioned? I did not understand).
  c. In equation (3), the definition of the expected error rate \ell^{\test{new}} is only given in words, not in a formula. 
  d. In section 4.1, what are a, b, kappa, and mu?   
  e. In appendix A, none of the notation is defined, and in fact no information is given about the context and goal of the derivation there. 

Further comments

1. Online vs offline methods. The scope of label shifts considered in this paper is more limited than those considered by Wu et al.: here only constant shifts are considered (test data is drawn from a fixed shifted distribution), whereas Wu et al considered distributions that keep changing throughout training. An important strength of online methods are their ability to deal with continual changes. Considering only constant changes reduces (but does not invalidate) the usefulness of online methods compared to offline ones. The decision to focus on online methods should be motivated in the paper. 

2. OOD validation: the concept of OOD validation is introduced in Heuristic 1 without being properly defined/explained. As this is a central tenet in the proposed methods, the idea and procedure should have a clear and detailed explanation. Furthermore, in Heuristic 1 it is written that OOD validation is a standard practice of model selection, with a reference to Gularjani & Lopez-Paz (2020). As far as I can tell, this reference (which emphasizes the importance of validation set details in the context of domain generalization) does not advocate the use of validation on a separate OOD set. Rather, it attributes this method to Krueger et al. (2020), who indeed mention it in an appendix. 
Regarding the method itself OOD validation itself: why should it work? I can understand that it might be useful when the shifts in the validation and test sets are somehow related (like the Skewed-MNIST example where test is a more severe shift of the same type as validation), but why would it help in examples like the mixture of Gaussians, or the WILDS datasets? Looking at the experiment results, it indeed seems to me that OOD validation is helpful only for the skewed-MNIST example. If my reading is correct, this should be stated clearly, and the appropriate qualifications should be made in the conclusions about the merits of OOD validation. Currently, section 5.3 states that “Using OOD validation sets … improves results on the whole” – but for S-COCO-on-Places and iWildCam (Avg) I do not see any improvement more significant than the noise level, and for iWildCam (F1) there is a small deterioration (which is also consistent with noise). 
From a practical perspective, performing OOD validation is not always possible as it requires more labelled data – it would be useful to emphasize this fact. Technically – what are all the optimization steps performed on this validation set? I.e., which hyper-parameters are calculated on this validation set, other than the confusion matrix? 

3. Non-invertible confusion matrices. The methods proposed in Heuristic 3 surely generate invertible matrices, but why would they be expected to work for label shift and general distribution shift adaptation? They seem to me ad-hoc and unmotivated. What would be their merit compared to using a pseudo inverse, or the soft probability matrix suggested by Lipton et al. (2018)? 

4. Section 4: The role of this Bayesian discussion is not clear to me. What insights are gained from this Bayesian perspective? Are these insights relevant also to cases of pure-label shift, or only general distribution shifts? I found the discussion around equations (11)-(14) confusing on first reading. The notation in equations (11)-(12) is confusing, perhaps Y|\phi ~ Cat(\alpha) and \phi ~ Dir(\alpha). The notation in equation (13)-(14) – P_t(\phi), P^{new}_{t+1} is not defined anywhere. 
I found the whole of section 4.1 confusing. How is the discussion related to label shifts in regression problems? What are the takeaways or results of this section? Are the results valid only for the Gaussian example with a conjugate prior, or more generally applicable? What kind of calibration is performed in this section, and why is it useful?

5. Experiment details. Right before section 5.1: 
  a. It would be worthwhile to provide the details of “the surrogate loss implementation of Wu et al.” 
  b. What are the details of the grid search used for the parameter of OGD? On which validation set is it taking place.
  c. Skewed-MNIST should reference the inspiration from color MNIST of Arjovsky et al. (2019). A table with the makeup (number of digits of each color) of each of the train/val/test datasets would be useful. It is stated that “Since the overall class frequencies are balanced … we drop the P(Y)”. Drop it from where (same comment for skewed COCO on Places)? Appendix C.1 describes how digits were split into two sets – was there a precise protocol for this? How is the “tend(ency) to be confused” measured context? What was the optimizer used for training  - SGD? 
  d. WILDS-iWildCam: it is stated that “We use Heuristic 3 for evaluating methods on this dataset. Heuristic 3 mentions several approaches: adding a tunable scalar to the diagonal? Using the identity matrix? Using a “pseudo-count”? 
  e. Table 2: How are the error estimates estimated relevant to all tables)? Why are the error estimates here +- 0? Are the quantities really measured to perfect accuracy? 

Minor comments

1. Before equation (4): “where e is a one hot vector for the predicted category” – the description and notation there can be clarified: it was initially unclear to me which predicted category is referred to, and only after reading Wu et. al (2021) did I understand that these are calculated for each step I separately. 

2. After equation (4), it is stated that calculating the gradients is tricky. Why is it so? For self-containedess, the statement should be explained. Similarly, before equation (7) it is stated that FTH is more efficient than OGD – efficient in which sense? Compute time? Memory? Data complexity? 

3. Right before 5.1.3, it is mentioned that “test-sets are smaller”. Smaller than what? 

4. Typos: 
- Heuristic 1, line 2: shiftis -> shift is
- Two lines below equation (19): minimum -> minima
- Last line of page 6: there’s a superfluous ). 
- 5.2.2, last line of first paragraph, should read “in neither training nor validation sets for OOD test.”
- The reference to Sun and Saenko (2016) is missing bibliographic info (journal name).
- Appendix A: equations (23) and (24) seem to be the same


References 
[1] Storkey, When training and test sets are different, in:Quinonero Candela et al., Dataset Shift in Machine Learning, 2009
[2] Moreno-Torresa et al., A unifying view on dataset shift in classification  (2012)
[3] Schoelkopf et al., On Causal and Anticausal Learning (2012)
[4] Zhang et al., Domain adaptation under target and conditional shift (2012) 
[5] Kull and Flach, Patterns of dataset shift (2014)
[6] Breeds: Benchmarks for subpopulation shift, Santurkar et al. (2020)
[7] Cortes and Mohri, Domain Adaptation in Regression (2011)
[8] Cortes and Mohri, Domain adaptation and sample bias correction theory and algorithm for regression (2014)

clarity,_quality,_novelty_and_reproducibility: The paper, while being short and concise, is for the most part easily readable. Some sections that I found to be more difficult to understand are listed above. 

Experiments are described clearly and seem reproducible. Some minor misunderstandings that I had regarding experimental protocols are listed above. 

As far as I can tell, the paper's examination of online labels+conditional shift adaptation of neural networks is novel, as are the experiments performed here. 

As detailed above, the quality of the paper can in my opinion be greatly improved if more context was provided about the distribution shifts considered, a more thorough empirical investigation was conducted, and the unclear/undefined terms and sections are clarified. 
summary_of_the_review: The work presented in this paper is novel, seems technically correct, and addresses a key problem to many real-world scenarios. I believe that the work in its current state with some corrections/improvements could and should merit publication in some venue. However, with the flaws described above, I do not believe this paper is ready for publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This draft considers the problem of online learning with label shift in the presence of additional conditional shift. In addition to the change in the class-priors $\Pr[y]$, the posterior probability $\Pr[x|y]$ can also change over time. Based on the previous work of Wu et al. (2021), the authors propose three heuristics to improve its empirical performance for both classification and regression tasks when the additional conditional shift appears. In their empirical studies, the authors suggest 1) using the OOD validation set instead of the ID validation set when estimating confusion matrices (a key component in the work of Wu et al. (2021)), and 2) adding scaling hyperparameters to the original loss function to improve the performance. 
strength_and_weaknesses: Strength:
+ This work studies a well-motivated problem, which covers many real-world applications.

Weakness:
- The authors introduce three heuristics to improve the empirical performance of the algorithm proposed by Wu et al. (2021) when an additional conditional shift appears. There are, however, no significant or distinguishable improvements over the original method based on their empirical studies. For example, in Table 1, the proposed algorithm's performance on S-COCO-ON-PLACES and IWILDCAM (Avg.) is nearly the same compared with the original algorithm, and even worse than the original one in IWILDCAM (F1).
- Heuristic 3 (adding a tunable scalar to the diagonal and renormalizing rows to the confusion matrix) is a common practice to avoid the non-invertible problem.
- A discussion of the intuition behind the heuristics is suggested.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality:
It is easy to follow this draft. The proposed idea is easy to understand.

Novelty:
The novelty of the proposed method is limited. There is a lack of discussion of the intuition behind the proposed heuristics. In the related works, heuristic 3 is a common practice.

Reproducibility:
The proposed heuristic is not difficult to implement. The reproducibility is acceptable.
summary_of_the_review: This draft considers the problem of online learning with label shift. The authors propose three heuristics to handle the additional conditional shift based on the previous work of Wu et al. (2021). However, the empirical studies do not show significant improvements over the original approach. The proposed approach lacks theoretical support, and its proposed heuristics lack intuitive support, as well.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies how to adapt a black box to the testing distribution in an online fashion under the label shift condition. The main contributions are to propose several heuristics to improve the algorithm proposed by Wu et al., (2021) when the label-shift assumption is broken, or the confusion matrix is non-invertible. Empirical studies are conducted to validate the effectiveness of the proposed methods.

strength_and_weaknesses: ### strength
+ This paper considers an interesting problem on how to relax the label shift assumption made in the previous work.
+ This paper extends the online label shift problem to the regression setting.
### weaknesses:
- About the written quality: Although this paper is well-organized, some parts of the presentations, particularly the algorithm design part, are not totally clear. The unclear parts are listed as follows.
	- about the background: the notation $f$ in Eq. (2) and Eq. (4) are very confusing. In Eq. (2), $f$ is used for an underlying base model, but Eq. (4)  uses $f$ for the model to be learned. 
	- about Heuristic 1: it is unclear to me how the validation set is collected. Does the learner collect the validation data once at the beginning of the testing online test stage, or do the data just appear in an online fashion? The former one seems less promising in the online adaption problem since the underlying distribution $P_t$ could be different for every iteration.
	- about the Bayesian methods: the notation $\hat{y}_\tau$ in Eq. (9) and $y_t$ in Eq.(13) is not defined, though I can guess they are the pseudo-label and the true label. It is unclear to me why Eq. (13) updates with the true label while Eq. (9) updates with the pseudo-label. It is a very strong requirement to obtain the true label for each iteration.

- About the soundness of the proposed heuristics:
	- about the validation set: the main difference between this work and Wu et al., 2020 is that the latter does not require the validation set. When a validate set sampled from the testing distribution is available, a strong baseline is that we can just estimate $\hat{\mathbf{q}}^{\mathrm{new}}$ by the labeled validate set. I think it would be necessary to compare the proposed method with such a baseline.
	- about Heuristics 1: Heuristics 1 aims to solve the problem when the label-shift condition is broken. But the proposed methods are still based on the reweighted classifier Eq (2). Such a kind of reweighting mechanism still crucially relies on the label shift assumption. In this sense, I believe only the adjustment on the estimation of the confusion matrix is not sufficient.
	- about Heuristic 3: it seems that the use of an identity matrix instead of the confusion matrix will sacrifice the unbiasedness of the gradient estimator. I am not sure whether such a method can perform well when the confusion matrix is invertible. 

clarity,_quality,_novelty_and_reproducibility: Quality: the paper lacks a comparison with the baseline, which learns directly with the validation dataset, and the proposed heuristics are somewhat unconvincing. (Please see the second point of the weaknesses for more details.)

Clarity: This paper is well-structured, but some notations are abused, which makes the background part hard to follow. Besides, the difference between this work and the previous one [Wu et al., 2021] on the problem setup is not clearly discussed. It is unclear to me how the validation dataset is collected, which plays an important role in the algorithms design of this paper. 

Originality: this paper is an extension of Wu et al., [2020]. Although some heuristics are proposed to improve the previous work, some of them are less convincing to me.

Reproducibility: codes for the experiments are not provided.

summary_of_the_review: This paper considers how to improve the previous work [Wu, et al., 2021] to learn beyond the label shift assumption. This is an interesting problem, but the proposed method is somewhat unconvincing to me as it requires an additional validation dataset and the reweighed classifier still implicitly relies on the label shift assumption. Besides, the background, problem setup, and method parts of the paper are not clearly written. Given the above concerns, I tend to reject this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Using standard label shift handling techniques/equations, it is proposed to handle additional manifestation-shift (p(x/y) shift) by simple heuristics based on additional hyperparameters (like in (8)) or using Bayesian models over labels etc. (19). Empirically it is shown that the proposed heuristics improve over those in Wu et.al.'21.

strength_and_weaknesses: Weakness:
1. The writeup does not seem to be self contained and the reader may have to refer to Wu et.al.'21 to understand some background.
2. Some parts may require more details. For example, how is p estimated in (8) ? Do the additional hyperparameters effect p's estimation?
3. While the Bayesian model equations are standard, the connection of it for the problem at hand in section 4 is not clear to me. Perhaps some re-writing might help here.
4. Apart for some cryptic justification, none of the propositions seem to have theoretical justifications. This makes it very hard for me to evaluate the work.
clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper does not seem to be self contained and some necessary details seem to be explained at a very high level. this makes it very hard to understand the methodology, motivation etc.

Novelty and QUality:
limited novelty and technical contribution - essentially the heuristics

Reproducibility:
Because of the high level intuitions, and lack of details, one may not be able to reproduce the results.
summary_of_the_review: This seems to be a work in preliminary form with many missing details and justifications. Hence I tend to not accept the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper discusses the problem of online learning with label shift in the presence of conditional shift.

Strength:
The paper is well written and the investigated problem is general and well-motivated.

Weakness:
The authors' proposal is heuristic and there is no theoretical justification.
justification_for_why_not_higher_score: I would like to thank the authors for their replies to the questions raised by the reviewers and for updating the manuscript, which improved the understanding of the authors' work to some extent. However, the overall contribution of this work is limited, in particular, lack of theoretical justification is critical. Therefore, I cannot recommend the acceptance of this paper.
Decision: Reject