Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a new experimental setting $Unsupervised\ Latent\ Label\ Shift$, where the data are available from multiple domains and the label distributions vary across different domains but the class conditionals do not. Furthermore, this paper proposes a DDFA framework to solve the problem unsupervisedly and theoretically demonstrates that the matrix factorization solution may be identified. The experimental results demonstrate the effectiveness of the proposed framework.
strengths_and_weaknesses: Strengths:
1. The writing is very clear and the paper is easy to follow.
2. The setting of this paper is very interesting and practical.
3.  I appreciate the notion of using matrix factorization to calculate the distribution $Q_{X|D}$, which can save the cost on training the generative model $Q_{X|Y}$, making this framework realistic and easy to use in real-world applications.
4.More crucially, this work presents the assumptions and theoretic analysis of the $Q_{X|D}$ identifiability.


Weakness:
1. In the section "Discretize", the number of clusters $m$ is larger than $k$. Can the authors perform an ablation study on the ratio between $m$ and $k$? If so, the results may be more convincing.







It would be nice to have a citation for the relevant paper [1].

[1] Guo J, Gong M, Liu T, Zhang K, Tao D. Ltf: A label transformation framework for correcting label shift. International Conference on Machine Learning 2020 Nov 21 (pp. 3843-3853). PMLR.
questions: Please refer to the above section
limitations: Please refer to the above section
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors introduce a novel general framework, referred to as Discriminate-Discretize-Factorize-Adjust (DDFA), for unsupervised learning under Latent Label Shift (LLS). With semi-syntheitc experiments, the authors show that the developed DDFA can leverage domain information to improve the performance of unsupervised classification.
strengths_and_weaknesses: Strengths:

1) The principle ``instances that shift together group together’’ is interesting, and the authors creatively utilize topic model to capture these shits across various domains

2) The paper is well-organized and the proof is very rigorous.

Weakness:

1) My only concern is whether the label shift assumption exists in real-world datasets. Refer to the question part for details.

questions: As far as I known, this work could be the first one to apply topic model to unsupervised learning under label shift. It does bring a lot of new ideas to me about how to make full use of topic models under multi-domain scenarios.

The whole work is built under the assumption of label shift, and all experiments are conducted with semi-syntheitc datasets. As the paper said, ``if we notice that whenever, ....we might conclude that the two were tied to the same underlying concept.’’, I absolutely agree with the point that topic model can be used to capture these co-occurrence patterns. 

However, what if the assumption does not hold in the dataset. For instance, in the first domain, all dogs are under sun, but in the second domain, all dogs are under the moonlight. In that case, I assume that the topic models can hardly summarize these dogs into the same topic, and the developed DDFA will also fail.

So, I think the authors had better to prove whether the assumption really hold in real-world dataset, rather than use semi-syntheitc experiments.Then, it will be better to discover or visualize the underlying principle capture by DDFA, e.g, the amount of butterfly and caterpillar will shift together.

Additional Question

In Algorithm 2, it is quite confused to me that how you can obtain q(y|x) with Q(D|Y) and f(x). In my consideration, for a topic model, given a word x in the document d, it is unable to infer which topic the word assigned to.  So, which assumption can be used to infer the topic index y for each word x. You had better reexplained it under the scenario of topic model.

limitations: The novelty is high but the assumption is too idealistic.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Traditional unsupervised learning methods involve clustering according to similarity in some representation space. However, the true labels may not align with such clusters, for instance, if the clusters learn a spurious feature instead (such as adult versus caterpillar for moth versus butterfly classification). Instead, this paper proposes an unsupervised learning technique based on label shifted domains; that is, given unlabeled datasets coming from different domains where $p(x|y)$ stays the same (e.g. the within-class structure) but $p(y)$ changes, such as across time and location, we can uniquely recover the labels of points. First, the paper considers the tabular setting and draws an equivalence to topic modeling, where points are words, domains are documents, and labels are topics. A simple set of assumptions is presented for recoverability through non-negative matrix factorization. The next challenge is extending such an approach to continuous data. This is done by clustering examples based on the vectors $[p(d|x)]_d$ approximated by a classifier over the domains, performing the topic modeling approach on the discretization, and converting recovered quantities back to $p(y|d, x)$ using Bayes rule and the classifier. They show that on real-world datasets where feature space similarity does not imply the same label, this approach is able to outperform other unsupervised learning methods given synthetically induced label shifts.

strengths_and_weaknesses: **Strengths**

_Originality_: I found this paper to be very novel in establishing a connection between label shift and topic modeling.

_Quality_: theoretical and empirical results were solid. The theory felt straightforward, drawing directly from topic modeling literature and applying Bayes rule, but I think it was sufficient for the scope of this paper.

_Presentation_: the paper was well-written and easy to follow.

_Significance_: this method seems to be potentially very useful for handling datasets with spurious features that naive clusters would overindex on, which is a pervasive challenge in areas like medical imaging (Oakden-Rayner et. al., 2019). 

Oakden-Rayner, L. et. al. Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging. ML4C at NeurIPS, 2019.

**Weaknesses**

_Quality_: there were no direct theoretical results on the DDFA method. It would have been nice to discuss further tradeoffs for this discretization, such as choice of m and estimation error.

_Significance_: I am not sure how practically relevant this sort of approach can be. My first concern is in general about label shift, since I am not aware of if there are real datasets that cleanly capture this phenomenon. Second, based on the recoverability assumptions, if we had a task with many classes, we would need just as many domains/sources of data, such as those across time and location. 


questions: **Q1.** Could this approach be applied to the Waterbirds dataset (https://github.com/kohpangwei/group_DRO)? Would we expect it to do well? This is similar to FieldGuide, and I imagine that when naive clustering is done, things will be clustered by the birds’ background, which tends to dominate the image. I’m also curious what the effect of subgroup imbalance is here, since there are few land birds in water and few water birds on land. Equivalently, in FieldGuide, are there roughly the same amount of immature and adult moths/butterflies? Is your approach robust to these imbalances? 

**Q2.** The importance of discretizing with f(x) could be made clearer by performing an ablation on some naive representation space. 

**Q3.** Presentation-wise, it is not clear to me how theorem 2 and DDFA are related. I believe the connection is that asymptotically, DDFA satisfies theorem 2, but this is not explicit enough. Furthermore, a quick glance at Theorem 2 does not highlight how the identifiability is dependent on the discretization approach; it currently sounds like something that is just simply true.

**Q4.** Alpha (line 325) should be defined in the text. There is a typo in algorithm 1, line 8: return $\hat{\textbf{Q}}_{c(X) | Y}$? But that is discarded, right?

limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work tackles the unsupervised learning problem under latent label shift. In this problem, each data point has a domain label, but the true classification label is not available. The key underlying assumption is that the class conditional distribution is domain invariant, while the distribution of labels in each domain affects the final data distribution. To resolve this problem, this paper relates it to the classical topic modeling problem and further extends its framework to the continuous domain. Empirically, the proposed method outperforms the SCAN method on CIFAR and FieldGuide datasets.
strengths_and_weaknesses: Pros:

1. While current unsupervised learning techniques are mostly driven by the similarity assumption among data points, this work helps understand the conditions in which unsupervised learning is possible. I believe this work brings new insights to the community.
2. The linkage between the LLS problem with the topic modeling is intuitive and convincing.
3. This work is original and well-written. 

Cons:

1. My main concern lies in the clustering step in their algorithm, which is mainly designed to discretize the input such that they act like words. This can be problematic in the high-dimensional case, where the identifiability assumption can also fail. I suppose this is the reason that DDFA relies on a well-trained backbone in experiments. 
2. The experiments are sort of weak; see my question below.


questions: 1. Note that the 'pre-training backbone' step still relies on the feature-space similarity assumption. I doubt whether the main goal of this work that 'unsupervised learning can work beyond feature similarity' still holds? Or maybe similarity learning is still an essential step for unsupervised learning, but this work takes a further step on a coarser granularity? For example, we still have to find the sub-clusters of caterpillars and butterflies, but they belong to the same super-cluster. 

2. How does DDFA (RI) perform on FieldGuide datasets? It's weird that different variants of DDFA are used on different datasets.

3. On the FieldGuide-2 dataset, the Q{Y|D} error seems to be high, why? Why is the final performance not affected by a bad estimation of Q_{Y|D}? 

4. Are the limitations of this work described anywhere? I notice that the authors answered Yes in the checklist. 
limitations: NA
review_title: null

Meta-review: metareview: This paper studies the problem of identifying classes in unlabeled data. It considers a little-to-not-studied setting, where data is available exhibiting latent class proportion shift. This means that p(y) changes, but by assumption the conditional feature distributions p(x|y) do not. The paper proposes a principle that features that shift together should group together in order to identify the latent classes. In both the tabular and continuous data settings, the paper shows theoretically and in practice that this principle with the associated assumptions is sufficient to label the data in an unsupervised setting.

The reviewers all agreed that the paper makes a strong contribution to a less studied area, and that the idea of exploiting label shift to identify classes is a significant advancement in this area. In particular, they liked the connection between the proposed setting and topic modeling. This leads to an approach for handling continuous data by clustering. The reviewers also generally felt the paper is clear and well written.

Much of the discussion related to the realism of the proposed setting and the assumptions. The authors are encouraged to include the mains points from these discussions in the final version.
confidence: Certain
Decision: Accept