Kati8/ICLR_2023_2025 huggingface dataset:
authors: Liqin Ye
authors: Agam Shah
authors: Sudheer Chava
reviews: 
Comments: This paper presents a work on learning with noisy labels in datasets annotated by LLMs. The proposed framework, SiDyP, is based on a noisy-supervised model called PLC and a simplex diffusion model. In this framework, the PLC model, fine-tuned with the noisy labeled data, is used to obtain dynamic training trajectories and induce a prior distribution, while the diffusion model serves as a posterior to iteratively calibrate PLC’s prior. Specifically, this paper introduces a true label candidates retrieval algorithm and a prior dynamic distillation
algorithm to mitigate the negative impact brought by noisy labels. Extensive experiments demonstrate that the proposed framework achieves competitive classification performance on multiple tasks.
Strengths: 1. Denoising LLM-generated labels may be a promising direction and is worth further investigation.
2. Experimental results are good, which shows that the proposed method is a state-of-the-art one.
Weaknesses: 1.The novelty of this paper is limited, capturing the label prior distribution based on training dynamics and using the simplex model for iterative refinement are the technical contributions of DyGen and TESS, respectively. The contributions of this paper seem like engineering innocations.

2.The proposed method introduces a few more hyperparameters. In practice, when we don't have a clean validation set, these hyperparameters can be difficult to set.

3.Writing needs to be improved. the dynamic prior needs to be clearly explained in the abstract and introduction, which can easily be confused with training dynamics. Additionally, the organization of Section 3 is not reasonable.

4.A more detailed ablation study would have been desirable, such as the number of candidate labels in Algorithm 1 and the number of iterations in Algorithm 2.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: This paper proposes a method called SiDyP to address the learning-from-noisy-labels problem.  
In the experiments, the authors validated the effectiveness of the proposed method using noisy data generated from the zero-shot and few-shot inference of Llama-3.
Strengths: The authors conducted experiments using noisy labels generated by LLM Llama and demonstrated the effectiveness of the proposed method, which is one of the contributions of this paper.
Weaknesses: 1)	On the novelty of introducing LLMs for generating noisy data for learn-from-noisy-labels methods.  
The authors claimed in the Introduction that, this is the first time that LLMs-generated datasets have been introduced in the realm of learning from noisy labels.
Actually, this statement is not rigorous. 
There are already some works that focus on learning from noisy labels generated by LLMs [1,2,3].
These works concern the topic of Weak Supervision, which can be seen as a subset of the learning-from-noisy-labels problem; even though Weak Supervision considers the context of each one sample with multiple noisy labels, the learning-from-noisy-labels problem often refers to the context of one sample with one label.
Anyway, the authors should take these prior works into account while writing.

[1] Alfred: A System for Prompted Weak Supervision. ACL-2023 (demo).  
[2] Combining prompt-based language models and weak supervision for labeling named entity recognition on legal documents. Artificial Intelligence and Law, 2024.  
[3] https://snorkel.ai/blog/few-shot-learning-large-language-models/. Blog from Snorkel.


2) On the core research motivation and contributions.  
Related to the first issue mentioned above, and more importantly, the core research motivation and contributions of this paper are not sufficiently clear:  

    &emsp;2.1) If the designed method is more specifically tailored to noisy labels generated by LLMs (rather than noisy labels in the general learning-from-noisy-labels works), or in other words, if the proposed method is specifically designed for noisy labels generated by LLMs, then:
    The authors should analyze from theoretical or experimental perspectives whether noisy labels generated by LLMs have different characteristics compared to noisy labels in the traditional sense, and why the designed method is well-suited to handle these characteristics.

    &emsp;2.2) If the designed method is not specifically aimed at noisy labels generated by LLMs, then this paper essentially presents a new typical learning-from-noisy-labels method.    

    &emsp;2.3) In summary, the authors do not clarify in the paper whether noisy labels generated by LLMs have different characteristics compared to traditional noisy labels, nor do they specify whether their contributions are specifically designed for noisy labels generated by LLMs.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper proposes applying simplex diffusion to denoise annotations generated by large language models (LLMs). The core idea is to first obtain noisy annotations from LLMs and then iteratively refine these labels with a simplex diffusion model, which maps labels to a continuous probability space. The main experiments are conducted across four common text classification datasets using Llama 3.1-70b. Overall the results show that the proposed method achieves competitive performance in both zero-shot and few-shot settings. Additionally, the authors explore the method’s robustness by testing with different 4 LLM backbones and different types of label noises.
Strengths: Overall, I think this paper is well-written and relatively easy to follow. The results are very competitive and clearly show the advantage of the proposed method. I think the construction of experiments is correct.  Additionally, the authors clearly communicate the problem and the potential impact of their approach on improving LLM-generated annotations, which would be a valuable contribution to the field.
Weaknesses: = The comparisons with Mixtral models might not be entirely fair. If the authors used the base Mixtral-8x22b model, which is not instruction/chat-finetuned, it may not perform comparably to other models that have been finetuned on instruction data. For a fairer comparison, using Mixtral-8x22b-Instruct-v* or similar instruction-tuned models would better align with the capabilities of the other models tested.

= Insufficient literature coverage: Overall, this paper aims to solve the problem of LLM for data annotation and reduce annotation noise. I think it would be justifiable to have a better coverage of related work especially on LLM for data annotations. Give the limitation in space for the initial submission, I expect the authors to conduct a more rigorous literature survey of related work.

= Although the approach is novel for this specific problem of denoising llm annotaions, the method largely adapts simplex diffusion for signal denoising directly to noisy label refinement. This may raise questions regarding the technical novelty, as the adaptation itself does not appear to involve significant theoretical modifications.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3