Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses the problem of joint image and label noise. It proposes a unified framework for reliable learning to deal with the joint noise problem. It first filters out noisy data and then utilizes deep ensemble to decrease the model uncertainty of noisy data. The experiments reveal that the proposed model achieves superior performance on the dataset with real-world noise.
strengths_and_weaknesses: 1. This paper addresses a very important and interesting problem, which is noise exists in both images and labels. The paper is well written and easy to follow.
2. This paper proposes the first unified framework for joint noise by distinguishing noisy data from clean data based on the practical confidence sample filter. It is promising the intuitive to penalize the model uncertainty of noisy data.
3. There are plenty of experiments in the paper to clarify the effectiveness of the model design such as the uncertainty penalty strategy. The proposed method significantly outperforms state-of-the-art methods on the real-world noisy dataset. 

questions: 1. The relationship "noisy image v.s. value of w" and "noisy label v.s. value of w" should be demonstrated or visualized to validate the effectiveness of the filter.
2. The authors should discuss the limitations of the work.
limitations: The limitations and potential negative societal impact are not addressed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a unified framework to address the noise in the input images and annotated labels. To be more specific, the proposed method uses confidence scores to distinguish if the labels or the input images are noisy or not.
strengths_and_weaknesses: Strength: 
1. The paper is well written and clearly organized, and the paper is easy to follow.
2. The proposed method is easy to implement.
3. The proposed method is able to address the noise in the input images and labels at the same time.

Weaknesses:
1. The proposed method seems trivial. In the equation 6 and 7, the weights are only determined by using two thresholds, which are fine tuned by hand. This makes the proposed method trivial and incremental.
2. Also the reviewer has some concerns about the equation 6 and 7. The hyper-parameters epsilon 1 and epsilon 2 are tuned by hand and in the appendix B.2, the authors claimed that they are set to 0.02 and 5% respectively. The reviewer is curious about the selection of these two hyper-parameters. Do these parameters require heavy tunes or is it complicated to tune epsilon 1 and 2? If not, why do the authors conduct experiments on epsilon 1 from 0.015 to 0.035, and epsilon 2 from 1% to 6%? Further explanations might be needed on these parameters.
3. To the best of the reviewer's knowledge, a large-scale noisy dataset is not included in the paper - ImageNet-C, which contains different types of noises. The reviewer is wondering why the authors ignore this benchmark.

questions: 1. The authors might need to carefully design how to set confidence scores of noisy label and input images.
2. The authors might need to explain more on the selections of epsilons, which are a very important part of the proposed method.

limitations: The authors have addressed the potential societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work explores a novel learning task of supervised learning with both image noise and label noise. The focus on the learning under dual noises seems to assume a maximum of one type of noise in each data point (unclear to me). To address the proposed task, a unified framework with a confidence-based sample filter is proposed, where model uncertainty is penalized instead to avoid over-fitting with false information.
strengths_and_weaknesses: + This work advances the field by considering a new task that simultaneously handles both image noise (x-noise) and label noise (y-noise) presented in the training dataset. The considered problem is highly realistic and would be very useful for learning with real-world data.
+ This work proposes a confidence-based method to detect images that may have noise. Then, dedicated thresholds are used to determine if a given image is noisy or not.
+ A learning objective is derived to estimate model uncertainty, which is subsequently used as a learning objective to improve the discriminative and robustness of the trained model. 

- One concern of the weakness is that the filtering is a binary process (eqn 6 & 7). As the experiment is conducted on CIFAR-100 and Tiny ImageNet, it is unclear if such a naive approach is reliable for more challenging real-world datasets (e.g., ImageNet).

questions: - Please clarify if a given image label will only have x-noise or y-noise only? or both types of noise can be applied

- For the section in line 156-166, please discuss the challenge of detecting x-noise only in the easy and challenging datasets. I have concerns about this as on a hard dataset, the data uncertainty could come from either noise or the underlying complexity/bias of the visual data. 

- In line 188, it states that the dirty observation is unhelpful for model training. I would like the author to elaborate if x-noise I and x-noise II would have a similar impact on the model learning, and shall both types of noise be handled differently. 

- Please also discuss if the parameters (e.g., epsilon_1 and epsilon_2) are dataset-dependent. It is unclear if tuning is required for different datasets.

- I suggest naming the WebVision as WebVision-50 in the paper as readers may think it is validated on the full dataset. 

- In Fig 3, is the Best performance or Last reported results.

- For Table 1, please also show result with clean dataset (no noise augmented). This will give the reader a clear view of the challenge in present of noise. 
limitations: The paper discusses the technical limitation but not the societal impact. However, I don't see any potential societal impact that is unique in this work.
review_title: null

Meta-review: metareview: Novel work that addresses the somewhat realistic setting of noise in input (images) and annotated labels. The method uses confidence scores to decide whether the labels or the images are noisy. Reviewers agree that the work is clear and easy to follow + easy to implement. I disagree with the reviewer that the proposed method is trivial -- fundamentally, showing that these simple thresholds actually help in a variety of settings is in fact a useful contribution to the community. I believe the simplicity of the proposed approach will make it more likely that this work is adopted or studied in the community, so I recommend acceptance.
confidence: Certain
Decision: Accept