Title: Label Noise Detection and Correction via Ensemble of Siamese Networks
Venue: NeurIPS 2025
Accepted: no
Keywords: noisy label

contrastive learning

noisy label detection
Abstract: Deep neural networks can suffer severe performance degradation when trained on datasets with
      instance-dependent label noise—annotation errors that correlate with input features.
      To address this issue, we propose a lightweight, model-agnostic preprocessing
      framework based on an ensemble of contrastive Siamese networks.
      Our method detects and corrects noisy labels by measuring embedding consistency:
      clean samples yield stable representations across models, while noisy
      samples exhibit high variability and increased misclassification rates.
      Each Siamese model is trained on a subset of image pairs, and we demonstrate
      that noisy instances are significantly more likely to be misclassified under
      this subset-driven embedding process, with the ensemble’s false-positive
      rate decaying exponentially with the number of models. Ultimately, samples
      with high model disagreement are flagged and either relabeled by consensus or
      discarded. Empirically, on real-world CIFAR-10N (9.01\% natural noise), our method
      reduces label corruption to 4.45\% and achieves 88.51\% accuracy on the cleaned
      dataset—0.26 percentage points ahead of the nearest baseline.
      Under synthetic instance-dependent noise, label corruption on CIFAR-10 is reduced
      from 40\% to 25.9\% (yielding a 12.54 percentage point accuracy gain) and on
      Fashion-MNIST from 40\% to 4.6\% (a 2.23 percentage point accuracy gain).
      Our preprocessing step adds minimal overhead, produces interpretable
      uncertainty scores, and can be seamlessly integrated with any downstream
      learner to enhance robustness against label noise.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to enhance robustness against label noise by introducing a preprocessing method for noisy datasets. This method focuses on three key strategies: i) retaining correctly labelled examples, ii) re-labelling noisy examples when the true label can be confidently estimated, and iii) removing examples where the true label is uncertain. After refining the dataset, standard training is conducted using a classification neural network with cross-entropy loss.\n\nTo assess the certainty of labels in the training set, the authors propose measuring the consistency of class predictions from an ensemble of models that have not been trained on this specific example. This is achieved by dividing the training set into folds, training multiple ensemble members on different subfolds, and evaluating their predictions on the shared validation set of the fold. A prediction is considered consistent if a majority of the ensemble members agree on the class. If the ensemble consistently predicts the given label, it is deemed correct (i). If the ensemble predicts a different label consistently, the label is updated to this prediction (ii). If the predictions are inconsistent, the example is filtered out (iii).\n\nEach ensemble member is trained in a semi-supervised manner, utilising an existing unsupervised loss method that encourages embeddings of input pairs to be similar if they belong to the same class and different if they belong to different classes. The authors provide a theoretical analysis of the challenges in selecting pairs for the unsupervised loss in the presence of noisy labels, demonstrating that noisy examples are more likely to be misclassified under certain assumptions. They also explore how ensemble methods influence noise detection performance.\n\nThe proposed method is evaluated on CIFAR-10 and Fashion-MNIST datasets, incorporating various levels of input-dependent noise.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\nThe method is intuitive as it uses the consistency of several different networks on unseen data to estimate what the true label is. The evaluation of synthetic instance-dependent noise on CIFAR-10 and Fashion-MNIST were thorough. \n\n# Weaknesses\n\n## Quality\n\n- **Unsupervised loss relies on noisy labels.**  \n  It’s unclear why the authors chose a label‑dependent unsupervised objective rather than modern label‑agnostic self‑supervised methods (e.g., SimCLR, Barlow Twins). Those approaches would eliminate the need to select pairs based on labels completely, and thus completely remove any need for theoretically studying this.\n\n- **Per‑member subfolds.**  \n  The paper lacks motivation for assigning each ensemble member its own subfold. Could the same inner fold be reused with different random seeds (for initialization and augmentation) to induce diversity? Prior work shows this alone suffices for uncertainty estimation [1] and would simplify the method and implementation.\n\n---\n\n## Experiments\n\n1. **Missing Ablation Studies** \n   - Remove the unsupervised loss to isolate its contribution.  \n   - Disable early stopping and/or dropout—both known to mitigate label noise—to quantify their impact.  \n   - Revisit the “subfold” design: train all ensemble members on the same fold with different seeds.\n\n2. **Missing Hyperparameter Sensitivity Studies**  \n   - With at least six hyperparameters ($\\tau_d$, $\\tau_r$ , $m$, $k$, $\\gamma$, dropout rate), how robust is performance to their choice? An extensive sensitivity analysis is needed.\n\n3. **Missing Experiments Validating Theory.** As we have synthetic label noise, we know the true labels and can thus, e.g., measure the false-positive rate in practice, and compare it to theory.\n\n4. **Broader Benchmarks**  \n   - Only performing experiments on CIFAR-10, CIFAR-10N, and Fashion-MNIST is a bit limited. Why no IIN experiments or experiments on Clothing1M? Why not perform experiments on, e.g., CIFAR-100 with as large k and m as the computational resources allow? \n   - As this method corrects labels, it might be good to compare against label‑correction baselines such as ELR [2] to contextualise gains.\n\n5. **Performance Discrepancy**  \n   - Table 3’s error rates are significantly worse than those reported in the original CIFAR‑N paper—please clarify the source of this gap.\n\n---\n\n## Theory\n\n1. **Misclassification Probability (Thm. 2.2).**  \n   - How does one motivate/justify the assumption that model performance and feature quality are independent, or conditioning model failure on noise or cleanliness has no effect (Line 524-525)?\n   - How does one motivate/justify categorising latents only into \"poor\" and \"good\"?\n   - A limitation of the theory is that it only considers uniform noise.\n\n2. **Bernoulli Modelling of Ensemble Errors (Thm. 2.3).**  \n   - Treating each member’s error as i.i.d. Bernoulli is a strong simplification (L563). How can one justify this in practice? Seems overly simplistic.\n\n---\n\n## Clarity\n\n1. **“Lightweight” Preprocessing (Line 3).**  \n   - How is training \\(m\\times k = 100\\) networks considered lightweight? \n\n2. **Notation in Sec. 2.3.**  \n   - Variables **L** and **MP** are introduced as “latent quality” and “performance indicators” but left undefined until the appendix. All essential definitions should appear in the main text.\n\n---\n\n## Originality\n\n1. Several closely related works are neither discussed nor compared against [2-7]\n\n\n---\n## References\n\n[1] Lakshminarayanan et al., Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles.  \n\n[2] Liu et al., Early‑Learning Regularization Prevents Memorization of Noisy Labels.  \n\n[3] Nashaat et al., Semi‑Supervised Ensemble Learning for Dealing with Inaccurate and Incomplete Supervision \n\n[4] Lu et al., An Ensemble Model for Combating Label Noise  \n\n[5] Nguyen et al., SELF: Learning to Filter Noisy Labels with Self‑Ensembling\n\n[6] Song et al., SELFIE: Refurbishing Unclean Samples for Robust Deep Learning\n\n[7] Chen et al., Detecting Noisy Labels with Repeated Cross‑Validations\n\n[8] An Ensemble Noise‑Robust K‑fold Cross‑Validation Selection Method for Noisy Labels\n\nQUESTIONS:\n1. **Theory Assumptions & Guarantees**  \n   - **Question:** Several simplifying assumptions are made in the theory, e.g., each ensemble member’s error is modelled as an i.i.d. Bernoulli trial. Can you (a) empirically validate these simplifying assumptions on realistic noise patterns?\n   - **Guidance:** Report the observed correlation of errors across members and show whether the Bernoulli model holds; provide at least one theoretical extension beyond uniform noise.  \n\n2. **Experimental Ablations & Robustness**  \n   - **Question:** Your method has many moving parts (unsupervised loss on noisy labels, per‑member subfolds, early stopping, dropout, and six hyperparameters). Could you provide (a) an ablation study isolating each component’s contribution, and (b) a sensitivity analysis over key hyperparameters (τ_d, τ_r, m, k, γ, dropout rate)?  \n   - **Guidance:** Include tables or plots showing performance when (i) the unsupervised loss is removed, (ii) subfolds are replaced by shared folds + different seeds, (iii) dropout/early stopping are disabled, and (iv) some or all hyperparameters are varied.  \n\n\n3. **Related Work & Positioning**  \n   - **Question:** Several ensemble‑based and self‑supervised noise‑robust methods ([2-7] above) are omitted. Can you (a) discuss how your approach differs from and improves upon these, and (b) include direct empirical comparisons on one additional real‑world dataset (e.g., Clothing1M)?  \n   - **Guidance:** Add a paragraph in “Related Work” contrasting your label‑dependent unsupervised loss with self‑supervised methods and how your work differs from the references I provided, and include a mini‑benchmark table with the closest work.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-28"}

comments: SUMMARY:
The paper proposes a pre-processing pipeline that cleans instance-dependent label noise before any downstream model is applied.

Some key ingredients of the proposed method:

- An ensemble of contrastive Siamese networks trained on different data folds; each predicts whether a sample’s label is incorrect via embedding inconsistency.

- Samples with high ensemble disagreement are either re-labelled by majority vote or discarded.

- Experiments on synthetic IDN (CIFAR-10, Fashion-MNIST) and real noise (CIFAR-10N) show sizeable accuracy gains over strong baselines.

STRENGTHS AND WEAKNESSES:
Strengths

- The problem the paper studies is of great importance. The idea is simple and intuitive to understand. It makes sense to me.

- The paper gives some preliminary theoretical analysis of noise detection.

- Experimental results show performance gain of the proposed method.

Weaknesses

- The major concern is its applicablity to practical scenarios. What if the datset size gets much larger? How does the noise detection works for out-of-distribution data?

- The training overhead (e.g. time) should also be carefully evaluated.

- There also lacks a comprehensive study of hyperparameter, say the inner/outer iteration number m and k.

- The experiments are still quite toy-ish. The results on CIFAR-10 and Fashion-MNIST may not be very informative and cannot indicate the generalizability.

- Theorems rely on latent–performance independence and bounded error correlations; real ensembles may violate these, weakening guarantees.

- Some hyperparameters (eg, $\tau$) are chosen by grid-search may be dataset-specific; guidance for unseen noise rates or fully unlabeled data is limited.

QUESTIONS:
See the weakness section.

LIMITATIONS:
n/a
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework that utilizes an ensemble of Siamese networks with contrastive learning to detect and correct instance-dependent label noise. The study found that contrastive training produces compact, well-separated clusters for cleanly labeled samples, while noisily labeled data remain ambiguous and are more likely to be misclassified by individual models. Experimental results on two public datasets demonstrate that this approach outperforms other compared methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The framework introduces an ensemble of Siamese networks trained on different subsets to detect and correct noisy labeled data. Noise detection is achieved by computing the disagreement counts among the predictions of the ensemble, while label correction is performed by checking for agreement on the new label.\n\n2. It offers a theoretical analysis demonstrating that noisy examples have a higher probability of misclassification.\n\n3. Experimental results on two public datasets show that this approach outperforms other methods.\n\nWeaknesses\n1. The settings for the hyperparameters ( r_d ) in line 88 and ( \\tau_r ) in line 89 are not clearly explained, despite their importance in detecting and correcting noisy labels.\n\n2. In the experimental section, the paper presents results for different noise levels (20%-40%). It would be interesting to see performance results for higher noise levels, such as 60%.\n\n3. The proposed method is evaluated only on multi-class classification problems, where a single label is predicted for a given image. It is unclear whether the method can be applied to multi-label classification problems, where multiple labels are predicted for a single image.\n\n4. The proposed method requires training ( m \\times k ) models for nested cross-validation. As the number of classes in the dataset increases, the number of models needed also increases to maintain detection power. It is not clear how this method can be applied to datasets with more than 10 classes, such as the ImageNet dataset.\n\nQUESTIONS:\nPlease see my questions in the weakness section. My main concerns are the constraint on the noise levels, whether the proposed method can be used in multi-class classification tasks, and computational requirement for dataset with more classes.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}