Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper proposes a method to achieve good prediction accuracy under label noise, by estimating the transition matrix from clean conditional probabilities, to noisy conditional probabilities. The paper presents experiments on synthetic noise models with a known transition matrix and natural noise models.

strengths_and_weaknesses: Strengths:

- Some of the experimental results are promising, especially on the synthetic noise models (Tables 1-3 and Figure 2).

Weaknesses:

- The paper could benefit from some changes regarding its style, which is atypical and unfortunately hurts clarity quite significantly. Here are just a few examples of things that could be improved. The introduction contains heavy notation that is not necessary at this point. The introduction even contains the proof of the “theoretical result”, as the paper notes on line 188. The paper is also not self-contained, and important concepts are not defined at all, e.g. the sufficiently scattered assumption or the anchor point assumption, the sym, asym or pair noise models, the baselines etc. In addition, the writing is at times too colloquial (e.g. “we creatively propose”, “what’s more” etc).  There are also numerous typos and grammar mistakes (line 39, 53 etc).

- The paper does not explain how the authors arrived at the procedure described in section 2.2. (see also the Questions section). There is also very little discussion on the impact of the 3 terms in the loss, or why the loss in equation 5 is particularly well suited for this problem.

- It is not possible to review the soundness of the experimental evaluation since too many details are currently missing from the manuscript. 

questions: - What is the motivation for the method proposed in section 2.2? For instance, why is it necessary to learn the second matrix $T^b$? 

- What happens if one only minimizes the second term in equation 5? Some of the experiments imply good performance and it would be interesting to understand what is the reason behind the method’s success in those scenarios. A more thorough ablation study would likely reveal what makes the method work well and what is perhaps unnecessary and can be stripped away from the loss.
It is difficult to assess the performance of the method because crucial details about the experimental setup are missing. What are the exact noise models? How many classes are assumed to suffer from label noise in the symmetric/asymetric/pair model? What was the procedure used for hyperparameter tuning for the baselines and for the proposed method?

- The method seems to not perform particularly well on datasets with natural noise. Why are only some and not all baselines presented in tables 4 and 5? What are the confidence intervals for the values in tables 4 and 5? What are possible reasons why the proposed method performs worse on natural noise models compared to the synthetic noise instances? I suggest adding to the experimental comparison more settings with noise models other than the ones considered in Tables 1-3, e.g. random flips for all classes with equal probability, other natural label noise datasets (e.g. OpenImages, MS-COCO with the noisy annotations, the dataset of https://github.com/zhongyy/Unequal-Training-for-Deep-Face-Recognition-with-Long-Tailed-Noisy-Data etc).

- How does a simple baseline that uses early stopping regularization (see e.g. https://arxiv.org/pdf/1910.09055.pdf and http://proceedings.mlr.press/v108/li20j/li20j.pdf) compare to the proposed method?

limitations: It is unclear from the results reported in the tables (in particular Tables 2 and 3) when the proposed method breaks and what exactly are the assumptions that it requires to perform well. Instead of selecting the 3 fixed noise rates reported in the paper, I suggest a plot in which the noise rate is varied on the Ox axis, while the Oy axis indicates the test accuracy. See also the discussion regarding other noise models in the Questions section.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes the idea of using the forward-backward cycle-consistency regularization to estimate the transition matrix T. This method addresses a long-standing problem of estimating the transition matrix depending on the inaccurately estimated noisy class posterior probability. Theoretical analysis and experimental results both show that the proposed method is superior to the compared methods. Particularly, it seems that this paper firstly proposes to estimate the transition matrix T in a backward manner.
strengths_and_weaknesses: Strengths:
1) Novelty: The main contribution of this work is the idea of estimating the transition matrix T with the forward-backward cycle-consistency regularization, under the sufficiently scattered assumption. Maybe, this is the first to estimate the transition matrix T bidirectionally.
2) Writing: The paper looks good and well-written. The introduction and the algorithm part are easy to follow.
3) Experiments: The proposed method achieves better results in both the synthetic and real-world datasets, and provides detailed ablation study.

Weaknesses:
1) It needs some model complexity analysis, especially for the training process of the backward and forward transition matrix.
2) Some implementation details are missing. How do you combine the proposed transition matrix estimating approach with the traditional DivideMix[11] algorithm, as shown in Table 4 and Table 5 ?

questions: Besides answering the above listed weaknesses, I also curious about the following question: Why not directly use T^-1 instead of the learned T^b to optimize the transition matrix?
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a label-noise learning algorithm through estimating the transition matrix T, to build a statistically consistent classifier. Specifically, the proposed algorithm tries to estimate the transition matrix under the forward-backward cycle-consistency regularization, which helps to minimize the volume of the transition matrix indirectly without exploiting the estimated noisy class posterior. Theoretical analysis and experimental results justify the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:
1)	The proposed method is technical sound and novel. It develops an algorithm to estimate the transition matrix under the sufficiently scattered assumption, by incorporating the proposed cycle-consistency regularization. This method can reduce the dependency of estimating the transition matrix T on the inaccurately estimated noisy class posterior.
2)	Detailed theoretical analysis illustrates that the cycle-consistency regularization helps to minimize the volume of the transition matrix T, which encourages the estimated T to converge to its optimal solution.
3)	The experiment is sufficient. They illustrate the effectiveness of the proposed method from both of the classification performance and the T estimation error, on both the synthetic and real-world noisy datasets.
Weaknesses:
1)	Since this method needs a forward-backward training process, does it require more time to train the network?  How many extra parameters are introduced in the newly proposed method compared with the previously proposed method VolMinNet[13]?
2)	This paper states that it tries to estimate the transition matrix under the sufficiently scattered assumption, what’s the difference between this assumption and the previous anchor point assumption?

questions: see the weaknesses above
limitations: YES
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the class-dependent noisy labels. To solve this, the authors proposes a cycle-consistency regularization on the estimation of the transition matrix for learning with class-dependent noisy-labels. The proposed method could encourage the estimated transition matrix to converge to its optimal solution, without explicitly estimating the noisy class posterior probability. Therefore, it could help to build a better statistically consistent classifier. Experimental results on several datasets show the effectiveness of this method, on reducing the estimation error of T and boosting the classification performance.
strengths_and_weaknesses: Strengths:
1. Different from the previous representative work VolMinNet, this paper proposes a new strategy to optimize the transition matrix T. The proposed method successfully addresses the problem of how to reduce the side-effects of the inaccurate noisy class posterior on estimating the transition matrix T. 
2. The author conduct experiment on several datasets to show the effectiveness of the proposed method. Moreover, the theoretical analysis is also provided to demonstrate the effectiveness of the algorithm.

Weaknesses:
1. In the overall objective function shown in Eq. (4), I believe we could also obtain the experiment results by just optimizing the backward transition matrix T^b. That is to say, we could just minimize Eq. (3) to obtain one intermediate results. However, this paper does not list this result for experimental ablation study.
2. In line 43-44, why anchor point assumption is a special case of the sufficiently scattered assumption? 
3. The author claimed that the proposed method can be used as a plug-and-play module and integrate it into the DivideMix. The authors should clearly define the experiment settings and how they are combined. 
questions: See the weaknesses above.
limitations: The authors have discussed the limitations of this paper, and there is no negative societal impact.
review_title: null

Meta-review: metareview: This work addresses the problem of estimating the transition matrix by using forward-backward cycle-consistency, with class-dependent noisy labels. There is merit in this work, as the proposed method might encourage the estimated transition matrix to converge to its optimal solution, without explicitly estimating the noisy class posterior probability. Therefore, it could help to build better statistically consistent classifiers.  It is shown theoretically that the proposed method is superior compared to compared methods, and the effectiveness of the method is demonstrated on several different datasets. There was a lively discussion going on between the reviewers and the authors. Although there remain some open questions about how the hyperparameter values are chosen, I think this paper should be accepted. 
confidence: Less certain
Decision: Accept