Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: - Assuming that there is no bi-directional causation, identify the causal direction between X and Y
- Argue that SSL is less effective in the causal direction than in the anti-causal case
- Argue that modeling label noise is agnostic to the causal direction
    - suggest a method to detect causal structure based on modeling label noise
- Introduce the CDNL estimator, which is claimed to be SOTA for $P(\tilde{Y} \mid Y^*)$
    - end to end learning of both $P(Y^* \mid X), P(\tilde{Y} \mid Y^*)$
- Demonstrate that the label noise methods outperform the SSL methods on causal datasets, while mostly matching levels of SSL methods on anti-causal datasets
    - although it is mentioned that SSL should outperform label noise methods for datasets where the ‘complexity’ is high
strength_and_weaknesses: - There are not any experiments that demonstrate how to use the CDNL estimator in order to achieve better results than simply comparing outcomes for label noise and SSL methods
    - Why would I use CDNL instead of doing both SSL and label noise methods?
    - I am not convinced that the CDNL estimator is the best method for estimating causal direction when there is a vast body of literature about how to estimate causal direction which has been ignored by this work
- The comparison with the VolMinNet is too brief: the authors would do well to outline the method of VolMinNet, what its weaknesses are, and how CDNL improves upon it

clarity,_quality,_novelty_and_reproducibility: - Several spelling mistakes across the paper, notably the consistent misspelling of causal as 'casual'
- Presentation of tables is confusing
    - Unclear at first what the rows mean
        - label noise and SSL methods should be more clearly labeled as such
        - the captions on the tables do not fully explain the table nor the main observations
    - I would produce plots of the results instead of tables since we measure how one continuous value (noise) affects another (accuracy)

summary_of_the_review: This paper introduces a novel method to determine the causal direction of a data generating process, when label noise is present. However, there is not enough of a literature review for me to feel comfortable that this method has not been tried in some manner before. Further, the examination of how this method can be usefully applied is poorly explained and unclear to me at present, when i compare against simply running both SSL and label noise methods in my problem of choice. Finally, the argument that anticausal and causal learning leads to different outcomes for SSL methods is not new. I would like to see more examination of the previous work on this area including methods developed (or lack thereof) to identity whether SSL is appropriate.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors approach the challenge of huge datasets with label noise from two perspectives: a semi-supervised learning (SSL) strategy that leverages labelled and unlabeled samples, and a noise modelling (NM) approach. According to the authors, semi-supervised learning is dependent on the causal generation process, but noise modelling is not. The authors also provide a method for determining a dataset's causal structure and evaluate results between SSL and NM for different rates of causality on synthetic and real-life datasets
strength_and_weaknesses: I'd like to start by expressing that I'm not totally confident in my judgement, therefore I'm interested in the author's reponse to these statements/questions.

Strengths

-The work addresses a significant issue in dealing with training models on large datasets with noisy labels.
- The authors suggest an intriguing method termed CDNL, which computes the flip rate using a clustering method to assess causality structure.
- The proposed method appears to outperform existing methods in computing the average flip rate, which aids in detecting the dataset's causality structure.

Weaknesses
- It is not clear what the contributions and conclusions of this work are. It would help if they are explicitly and concisely listed at the end of the introduction and the abstract. Currently they seem scattered over the article and cluttered. For instance, if the proposed estimator outperforms VolMinNet in error estimation, then that should be mentioned earlier on as part of the contributions.

- I am missing how the works described in the related work related to the current work. A sentence or two about the limitations of existing methods and how the current method addresses them at the end of each paragraph would make things clearer. For example, the paragraph on "Method based on semi-supervised learning" could have an extra such sentence(s).  Right now it is not clear how this work contributes to the literature in comparison to existing works.

- I am having a hard time understanding Figure 2, could you clarify how that figure shows that semi-supervised learning does not work when P(X) does not contain information to learn clean label Y? Further, the caption in Figure 2 repeats the conditional "When Y causes X" twice with different conclusions, adding to the confusion further.

- It is not clear why the following quoted claim is true, is there a theorem that proves it? "when X causes Y , the flip rate P(Y˜ |Y ′ ) estimated by an unsupervised classification method usually has a large estimation error, where Y ′ is pseudo labels estimated by the unsupervised method. However, when Y causes X, the estimation error is small."

- this work misses large-scale real-life datasets like ImageNet which is important to reliably evaluate the methods in this work.

- More explanation and comparison needs to be done about why CDNL outperform VolMinNet. For example, is there inductive bias in CDNL that helps it learn better?

- The table is poorly presented, it is not clear which methods are semi-supervised and which are noise modeling. It would help if the authors add a column that says something like "SSL" for the semi-supervised and "NM" for the noise modeling methods

- It is not clear whether SSL is helping when there is a causal structure, so far it seems that Forward, Reweighting and T-Revision almost consistently achieve state-of-the-art except in MNIST and CIFAR10 and it is not clear why.

- It is not clear what the differences are in the rows of Table 1, in both major rows, the dataset is XYguassian. Are these differentiated by casual and anticasual?

Overall this work could use more clarification and more experiments and theorems before deriving any conclusion. The results seem a bit inconsistent to justify the use of SSL over noise-modeling and vice versa. It is not clear how this work can help us develop better methods for large datasets with noisy labels in practice
clarity,_quality,_novelty_and_reproducibility: Please see the Strengths and Weaknesses section above.
summary_of_the_review: Please see the Strengths and Weaknesses section above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies two methods (modeling label noise and semi-supervised methods) for handling datasets with noisy labels, and the causal structure strongly influences which of the methods perform better. The paper additionally proposes a method for finding the causal structure.
strength_and_weaknesses: The paper studies an interesting problem of comparing two major lines of work, but the conclusions are not strongly supported. There is little theoretical justification in the paper, and the experiments do not closely match the conclusions drawn in the paper.

Major Comments:

1) There are many probability distributions that can be modeled under either (a) or (b) of Figure 1; it's unclear how the causal structure be directly used to determine which set of methods would perform better

2) A core claim is that the estimation error of P(\tilde Y | Y') is more accurate, but this does not seem to have any justification.

3) There appears to be mismatches in the experimental results and the main conclusions of the paper. 

Even in the datasets where semi-supervised methods are supposed to generally perform well, it seems common that only one of the methods actually performs well, while the others still have poor performance.

The tables also have incorrect numbers bolded in a number of columns (at the very least, JoCor outperformed T-Revision in XYgaussian Sym 30%, Forward outperformed Dividemix in YXgaussian Sym 30%, and Dividemix outperformed T-revision in KrKp Instance 30%), which falsely suggests that the trends are stronger than they actually are.
clarity,_quality,_novelty_and_reproducibility: The paper does not introduce the influence of the data generative processes clearly (in particular, it is unclear how the outlines in Figure 2 actually represent probabilities).

Additionally, there are a relatively large number of typographical issues. Some examples:
  - inccorect on page 1
  - mixmatch on page 1 has a broken citation
  - Gaussian is written as guassian multiple times in the paper
summary_of_the_review: The paper studies an interesting problem, but there is no theoretical justification, and the experiments to not strongly support the conclusions drawn by the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper compares two families of methods for learning with noisy labels. One is to use semi-supervised learning methods, and the other is to model label noise and design statistically consistent classifiers. The paper wants to answer the question: which one is better? From the perspective of causal data generation, the authors found that the answer depends on the underlying data generating process. Specifically, when the instance X is a cause of the clean label Y, modeling noise tends to be better. When Y causes X however, semi-supervised learning methods tends to perform better. In real-world applications, such causal structures are generally unknown. The authors proposed a method to discover the causal structure from a given dataset containing label noise.
strength_and_weaknesses: **Strong points**

It is of significance to study advantages and disadvantages of the two popular families of methods dealing with noisy labels, and to understand conditions under which one is better than the other. The authors made a good observation from the perspective of causal data generation (namely, when the instance X is a cause of the clean label Y, modeling noise tends to be better; when Y causes X however, semi-supervised learning methods tends to perform better), and also proposed a method to discover such causal structure from noisy data. The finding is intuitive from the theoretical point of view. The proposed method was also demonstrated its effectiveness by extensive empirical experiments.

**Weak points**

The main problem is that, in the proposed method (CDNL), it is required to estimate Bayes labels $Y^*$. As Bayes labels are used to estimate $P(\tilde{Y}|Y')$ and $P(\tilde{Y}|Y^*)$, which is in turn used to calculate the metric (Eq. 3) that determines which family is better, the quality of $\hat{Y}^*$ needs to be good. However, if $\hat{Y}^*$ is estimated reasonably well from the noisy data, we will have obtained a good classifier. The why bother proceeding further? Also, it seems that some model-based or semi-supervised methods are needed to estimate Bayes labels. For example, in Page 6 Eq. 7, a diagonally dominant column stochastic matrix is used to model the label noise. The arguments seem somewhat circular.
clarity,_quality,_novelty_and_reproducibility: Overall the paper is clear and well-written. The main idea is novel at large. Reproducibility is good.

summary_of_the_review: The paper is marginally below the acceptance threshold. The decision was based on weighing the strong and weak points above.

**Major questions and comments.**

1. Figure 2 should be improved. Just to be precise, what are those contours, the shaded areas, and the straight dotted lines?

2. How do you know KrKp, Balancescale and Splice are causal datasets, while waveform, MNIST and CIFAR10 are anticausal datasets?

3. Just above Section 4.2, the paper says "When the complexity of anticausal datasets is high, with limited sample size, the semi-supervised method should have better performance than modeling label-noise methods." In Table 3, we see that the semi-supervised methods performed better in MNIST and CIFAR10. However, these two datasets have relatively large sample sizes. I don't see how the claim can be verified.


**Minor comments.**

1. Inconsistent notation. In Page 1, $T$ is row-stochastic, while in Eq. 2, it is column-stochastic.

2. Inconsistent notation. In Eq. 1, $C$ is the number of classes, while in Eq. 3, $L$ is used as the number of classes.

3. Page 4, second to the last line: "estimates" -> "estimate".

4. Page 6, in the paragraph just above Section 4: some citations should be in parentheses. 

5. Table 1: for sym 30%, the best performance should be 86.04.

6. Page 9, Section 4.2, second to the last line: "from X to Y" should be "from Y to X".
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the problem of learning with noisy labels. They broadly categorize the methods into two groups 1) that model the label noise 2) semi-supervised methods. The paper focuses on the problem that in practice it is hard to figure out which group of methods should be employed on the problem at hand. They offer a solution based on "causal data generation process". They argue that semi-supervised methods depend heavily on the data generation process while modeling label noise is independent of the data generation process. Based on this they claim that if the given dataset has a causal generative structure i.e. features causing the labels, then the semi-supervised methods would not helpful. They also give a method to estimate this causal structure from the data, as it is not available in practice.
strength_and_weaknesses: Strengths:
S1. The problem to find when a kind of methods are better than others for learning from noisy labels is interesting and of practical relevance. 
S2. The use of causal generative process to study the problem looks promising and interesting.

S3. Empirical comparison of various methods in the two groups on synthetic and real datasets.

Weaknesses:
W1. The paper makes very strong claims about two groups of methods and I have two main concerns with this: a) Looking at Table 1, I don't see significant differences b/w the two groups of the methods in XY causal and YX causal settings both. b) I don't quite see how the theory is supporting their claims either.

W2. I understand different data generation from Figure 1, however I could not understand how these differences affect different groups of methods.

W3. The algorithm 1, is too abstract and doesn't provide more details into how $h$ and $A$ are optimized and how the clusters are obtained.
clarity,_quality,_novelty_and_reproducibility: Clarity of the technical parts could be improved, with clear definitions of different methods (groups) they are studying and how do these definitions interact with the causal data generation process. The work seems novel but relies heavily on Scho ̈lkopf et al., 2012; Peters et al., 2017. I have not reproduced the results, however the code is shared and the experiments are on public datasets with existing baselines. 
summary_of_the_review: The paper studies an interesting and relevant problem. They make strong claims about semi-supervised methods and methods that model the label noise. I am not quite convinced by the claims and the details provided to support the claims.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The main thrust of the paper is to introduce an estimator to help determine the causal direction of the data generating process when the data labels are noisy. The paper contrasts SSL and explicit label noise modelling.  The reviewers I think found this an interesting paper, but struggled to identify clear conclusions and recommendations as to how the approach would be used in practice. The paper suffered from a lack of clarity in places and I think probably confuses the reader in places. Despite a reasonable discussion phase in which reviewers with some background in causality were present, the review scores didn't increase sufficiently to make the paper in its current form acceptable for ICLR. 
justification_for_why_not_higher_score: There is no collective enthusiasm amongst reviewers (or myself) to champion this paper. Whilst the work has some merit, the presentation, conclusions and practical recommendations are not yet clear enough.
Decision: Reject