Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aims to address the challenging problem of Instance-Dependent Partial Label Learning, which is caused by overfitting on candidate labels, by rethinking the generation process of instance-dependent partial labels. In detail, the authors consider a two-step decomposition-based process. Categorical Distribution generates the correct label for a given instance, and Bernoulli Distribution samples the incorrect labels. In this way, they explicitly model the generation process. Then, to form their optimization objective, Maximum A Posterior (MAP) is performed, with Dirichlet Distribution and Beta Distribution introduced as prior distributions.
strength_and_weaknesses: Pros: 
1.	The proposed generation process is inspiring and offers a new perspective for me on Instance-Dependent Partial Label Learning. The generation process is distinct from traditional Multi-class Learning and Multi-Label Learning, but it also shares a few similarities, which is significant for understanding this problem.
2.	The paper is well-organized and logical. It begins by explicitly modeling the process with decoupled probabilistic distributions, then forms the optimization objective using a MAP technique, and finally theoretically analyzes the estimation error of the optimization objective.
3.	The experimental results are superior to the baselines, demonstrating the effectiveness of the proposed approach.

Cons and Qs:
1. Since the optimization is based on the proposed generation model, the log-likelihood
loss function in Eq.(4), which is derived from Eq.(1)~(3), could be more detailed. On the other hand, it is also used for ablation studies. Hence, more explanations need to be made to convince me.
2. Could you explain whether the proposed process is adaptable to a uniform generation process of candidate labels or the relationship between them?
3. I am curious about why Eq. (6) holds. Because the derivation is not in the appendix, a detailed explanation would be greatly appreciated.
4. I'm not sure how to estimate the parameters $\lambda$, $\alpha$, $\beta$, $\theta$ and $z$. Eq.(8) and Eq.(9) use $\lambda$, $\alpha$, $\beta$ to calculate $\hat{\alpha}$, $\hat{\beta}$, while Eq.(10) and Eq.(11) replace $\lambda$, $\alpha$, $\beta$ with $\hat{\lambda}$, $\hat{\alpha}$, $\hat{\beta}$.

clarity,_quality,_novelty_and_reproducibility: The paper is well organized and theoretically sound, indicating its good clarity and high quality. It is novel to decompose the generation process and explicitly model it using two probabilistic distributions. The overall proposed approach is easy to understand, so I believe that its reproducibility is assured.
summary_of_the_review: Overall, the paper proposes an insightful and novel approach to dealing with the instance-dependent PLL, and the experimental results demonstrate its effectiveness. As a result, I vote for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an instance-dependent PLL approach by decomposing the generation process of candidate labels into two processes and explicitly modeling the processes using different probability distributions, which the risk estimator is built upon. The MAP technique is employed to create the final empirical risk estimator. Then, the generation error bound of it is proposed for it in the theoretical analysis. Finally, experiments on corrupted benchmark datasets and real-world datasets validate the effectiveness of the proposed approach.
strength_and_weaknesses: Strength:
1.  It makes sense that the paper decomposes and models the generation process of candidate labels with two separate processes. The generation process is reasonable and interesting, and it has the potential to benefit future research.

2.  This definition of the data generation process is mathematically clear. The reasoning that follows is standard and rigorous, including the derivation of the risk estimator and the statistical analysis of the generalization error bound, is standard and rigorous.  All of these contribute to the soundness of the paper.

3.  The proposed approach IDGP indicates the superiority when it is compared to the existing PLL methods in benchmark and real-world datasets.

Weakness:
1. In the paper, the authors respectively utilize Categorical and Bernoulli distributions to capture these two processes. More explanations can be provided on it to further clarify why the instance-dependent partial labels could be modeled by these two distributions.

2. The presentation is good, but it might be better to use visuals and figures to provide an additional explanation of the learning procedure. 

3. The approach in Proposed Method includes two branch models, the main branch $f$ and the auxiliary branch $g$. The manually corrupted benchmark datasets in Experiments also use two sub-models $f$ and $g$. Overall, the former corresponds to the latter. However, the former $g$ appears to output two variables, $alpha$ and $beta$, whereas the latter $g$ appears to output only one. The difference inside should be explained in detail. 

4. Some related works [1, 2] are suggested to be cited and discussed. 
[1] Lyu, Gengyu, Yanan Wu, and Songhe Feng. "Deep Graph Matching for Partial Label Learning."
[2] He, Shuo, et al. "Partial Label Learning with Semantic Label Representations."

clarity,_quality,_novelty_and_reproducibility: The paper is simple to read, and the results are well presented. In this paper, the idea of decomposing and modeling the generation process is novel in PLL. The optimization procedure and objective are provided, and replicating the approach appears to be simple.
summary_of_the_review: This paper is intriguing and makes good contributions to Instance-Dependent Partial Label Learning.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the problem of partial label learning (PLL) on the instance-dependent scenario, which is a more realistic and practical setting. A new generative model is proposed that decomposes the generative process into two parts related to the ground-truth labels and other candidate labels, respectively. The distribution of the correct label and the incorrect label in the candidate label set of each training example are modeled explicitly by decoupled probability distributions Categorical distribution and Bernoulli distribution. MAP estimation is employed on the PLL training dataset to deduce a risk minimizer. Experiments on benchmark and real-world datasets validate the effectiveness of the proposed method. 
strength_and_weaknesses: Strengths:
1. This paper provides a new perspective on instance-dependent partial label learning. 
2. This paper provides a theoretical analysis for the estimation error bound of the proposed empirical risk. 
3. As far as I know, it is the first attempt to explicitly models the generation process of the correct labels and incorrect positive labels from different distributions. 
4. The theoretical justifications and empirical validations are solid.

Weaknesses:
1. The authors should give more discussion about the results on real-world PLL datasets. 
2. The authors should give more details the MAP optimization problem in Eq. (7)
3. “Then we perform Maximum A Posterior(MAP) estimation” should be revised as “Then we perform Maximum A Posterior (MAP) estimation”.


clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The extensive empirical results on both synthetic and real-world datasets validate the effectiveness of the proposed method.  Since the approaches are not difficult, the reproducibility could be ensured.


summary_of_the_review: Overall, this is a well-written paper and proposes a novel approach for instance-dependent partial label learning.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors extend VALEN with an instance-dependent incorrect label process.
strength_and_weaknesses: Strengths:
  * Clearly a good idea to improve an already strong baseline.

Weaknesses:
  * Clarity is lacking.
    * In equation (1), is the generation of candidiate labels conditioned on the correct label?  or perhaps conditionally independent of the correct label given the instance?  the notation in equation (1) is not clear and the surrounding text is silent.  (later in equation (3), there's a latent variable that is jointly conditioned with the instance, but not the correct label; and strangely equation (3) apparently attempts to model the probability that the correct label is [not] in the candidate set, whereas in PLL this is assumed always in the candidate set.) 
    * I find the discussion around equation (10) completely unintelligible, but possibly related to my previous concern listed above.
  * A theoretical correctness concern:
    * In Lemma 1 and Theorem 1, boundedness of $\mathcal{L}_{MAP}$ does not superficially appear to be a reasonable assumption, as it has a negative log in it, but this is presented without comment.
  * Some experimental concerns:
    * The transformation of classification datasets to PPL datasets uses the assumed generative model, which degrades from all the conclusions drawn from it.  Consequently, I'm not drawing any of the conclusions you are hoping for:
      * Table 1 does not convince me the proposed approach is better.
      * Table 3 does not convince me regularization is beneficial.
      * Figure 1 does not convince me the technique is not overly sensitive to choice of hyperparameter, except for Figure 1b.
      * **Note**: Table 2 *is* persuasive.
      * **Constructive feedback**: 
        * Drop Table 1 entirely.
        * Populate Figure 1 with 3 datasets from Table 2.
        * Do the comparison in Table 3 on datasets from Table 2.
clarity,_quality,_novelty_and_reproducibility: Clarity issues discussed above.

Originality ... well, let's be honest, it's a obvious incremental next step given VALEN, but it's still cool.

Quality issues discussed above.  
summary_of_the_review: **Don't panic.**  This paper wants to be accepted you just have to address my concerns and my score will change.  

Factors that are creating a low score right now are:
1.  a confusing exposition in section 3.2 [have someone with strong writing skills read this section and help you edit it]
2. an apparently disqualifying precondition in lemma 1 and theorem 1 [justify it or change it]
3. use of your own generative model for evaluation [drop all use of these datasets and only use datasets from Table 2].


(Updated score from 3->6)
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper targets a typical weakly-supervised learning problem named partial label learning. A novel method is proposed by rethinking the generation process of instance-dependent partial labels. Specifically, a two-step decomposition-based process is considered, where two steps are related to the ground-truth labels and other candidate labels respectively. The MAP technique is used to create the empirical risk estimator. Both theoretical analysis and empirical verifications are provided to confirm the effectiveness of the proposed method. 

The strengths of this paper include: (1) It is insightful to handle the problem of partial label learning from the generation process. The rationality of the proposed generation process in this paper is well justified. Besides, this idea can inspire follow-up research in partial label learning. (2) Both theoretical justifications and empirical validations are solid. This paper gives a detailed analysis of the estimation error bound of the predictive model. Additionally, comprehensive experiments indicate the superiority of the proposed method when it is in various benchmarks. (3) The presentation of the paper is logical. The weaknesses of this paper mainly lie in that it needs more clarifications of method descriptions and theoretical analysis.

In the rebuttal process, the authors provide detailed responses to the concerns of reviewers. All reviewers then vote for accepting this paper. AC checks all reviewing activities and agrees with reviewers that this paper makes solid contributions to partial label learning. 
justification_for_why_not_higher_score: The score of this paper is "Accept (spotlight)" now. The concern that makes it not be selected for oral presentation is that this paper's contributions are focused on partial label learning. Although the core idea is interesting, it may be limited in other weakly-supervised learning settings since it relies on specific prior distributions. Besides, more clarifications of method descriptions and theoretical analysis are still needed. 

justification_for_why_not_lower_score: This paper makes solid contributions to partial label learning. The core idea of rethinking the generation process is interesting and can inspire follow-up research in partial label learning. Therefore, AC and reviewers unanimously think that this paper should be accepted and highlighted.
Decision: Accept: notable-top-25%