Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a the adversary-aware partial label learning problem to protect the data privacy. The novel adversary-aware loss function, together with an immature teacher within momentum disambiguation algorithm, has achieved state of-the-art performance and proven to be a provable classifier.
strength_and_weaknesses: Pros:
* This paper proposes to take the data privacy into account in partial label learning, which is novel and significant in practice.

* The proposed method is smart and easy to follow, referring the idea of self-supervised learning. 

Cons:
* The theoretical analysis is based on the fully rank transition matrix. I have a concern about the existence of this condition especially when the noise is very regular.

* I think the rival is essentially a noisy label for PLL. Thus, incorporating the label noise learning methods into the existing PLL methods will be an important baseline.

* Some typos. For example, in the caption of Figure 2, "the Method inWang et al. (2022)".


clarity,_quality,_novelty_and_reproducibility: The clarity, quality and originality is fine.
summary_of_the_review: Please refer to the Strength And Weaknesses.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed a novel problem setting of adversary aware partial label learning (PLL) and a novel solution including an adversary aware loss and immature teacher within momentum (ITWM) to solve it. Theoretical analysis are presented and some empirical results are reported.
strength_and_weaknesses: Strengths:

1. The problem setting is a new interesting direction, and a solution is also presented to pave way to a new research area, with potentially profound implications in privacy enhancing PLL.

Weaknesses / Major Concerns:

1. My primary concern is the presentation of the paper. The current writing is very convoluted with too many concepts, without being proper introduced first. For example, the authors talk about the problem setting in Sec 1 (Page 2), but many notations were not introduced until Sec 1.2 (Page 3). Moving on, some challenges with respect to the proposed problem setting are discussed on Page 2, but it is very unclear and hard to follow (see point 2 below). With all the doubts (as a reader who hasn't read every single reference), the problem is set in Sec 1.2 (Page 3) without proper justifications (see point 3 below).
2. Given the current writing, I find it very hard to understand the motivation of the work when reading through Sec 1 (Page 2). To give a few examples, why is the assumption that the rival is generated depending only on Y but instance X (e.g. simplification for theoretical analysis, or intractability, or practicality)? Why does the inclusion of rival imply an inconsistent classifier according to Eq 2? Then why one cannot obtain consistent classifier due to intractability of \bar{Q}? How does the proposed ITWM help approximate \bar{Q} hence the consistency issue? Are there other alternatives one may consider, and why ITWM in particular?
3. The two main parts of the paper adversary aware PLL (Sec 1.2) and the ITWM algorithm (Sec 2.1) both lack clear explanation. For example, why the choice of the particular form Eq 4 for this problem (e.g. additive instance dependent noise but class dependent transition of rival)? How does the proposed setting extend and differ from Wen et al (2021) (e.g. a dedicated and elaborate section to such related works, or at least in appendix, is much appreciated)? How does instance embedding ("prototype") and the introduction of sparse matrix (A) help instead of obtaining true transition matrix (Page 4)? Why the positive sample set which appeared all of a sudden in Sec 1.3 without any mention throughout? In Sec 2.1, ITWM introduced a bunch of new notations that seem very disconnected from the previous texts? Then why does the adversary aware loss combine two terms (Eq 20) and how is the contrastive loss needed?
4. With all the doubts above, I cannot properly judge the theoretical analysis (Sec 3) or the experiments (Sec 4).
clarity,_quality,_novelty_and_reproducibility: Though the paper seems to introduce a novel and somewhat significant research problem, the paper in its current form has severe issues with the presentation of the proposed problem/method.
summary_of_the_review: My major concern for the paper in its current form is the writing. Without properly rewriting the entire paper, I can hardly recommend a higher score. If the authors are willing to polish the writing significantly during rebuttal, I'm happy to have another read and adjust my recommendation accordingly.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper attempts to proposed a new method for so-called adversary-aware partial label learning (but the definition of this setting is not clear in the whole paper). The proposed method is an incremental improvement on a SOTA partial label learning method.
strength_and_weaknesses: Strengths:
It is difficult for me to tell the strengths about this paper because the extremely confusing notation system and language makes it difficult for readers to understand.

Weaknesses:
1. The paper is poorly written. 
Firstly, the paper does not have a formal definition of the adversary-aware partial label learning problem. From Eq2, the paper seems to introduce a hidden variable $Y'$ to generate the set of candidate labels $\vec{Y}$, but in the definition on page 4, the generation of $\vec{Y}$ is only related to the real label $Y$, which is contradictory. I am really confused about how the setting of the paper differs from the classic partial label learning.
Secondly, most of the notations are not explained. For example, the notions that first appear throughout Sec1.3, $F, \boldsymbol{w}, \boldsymbol{v}, \boldsymbol{u}$ in Sec2.1, etc.
In addition, the methodological and theoretical parts of the article are also very vague. Assumption2 and Lemma2 appear out of thin air on page 7.
2. The methodology section appears to be almost an incremental improvement on PiCo, the only difference being the use of prototypes instead of queues. My bigger confusion is that the proposed method seems to be disconnected from the previous setting, as it does not use $Q$ and it is unclear how to get $T$.
3. The experiment is not convincing. There is no explanation of how to generate the settings studied in the paper, i.e., adversary-aware partial labels, and the reproducibility of the algorithm is poor.
clarity,_quality,_novelty_and_reproducibility: The paper is poorly written, lacks novelty, and the experiments are unconvincing.
summary_of_the_review: In its current state, the paper is not ready for publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a novel problem setting named adversary-aware partial label learning. The authors propose an Adversary-Aware loss function and the immature teacher within momentum (ITWM) disambiguation algorithm to tackle this problem.
strength_and_weaknesses: (+) The proposed new setting is potentially interesting.

(+) The experimental results show that the proposed method is effective.


(-) Although the setting, namely adversary-aware partial label learning, is new, I’m not fully convinced by its practical significance and necessity. In other words, do real-world applications really have such problem in partial label learning? Is it possible to provide a *real* case that falls into the scope of this research? 

(-) The definitions of “adversary” and “rival labels” need more explanations. To be specific, the name of the setting is “adversary-aware partial label learning”. However, when I read the paper, I notice that the authors only use a transition matrix to describe its relationship with true label. Therefore, I’m not clear what is the meaning of “adversary” here. In fact, even for the random label noise, one may also use a transition matrix to model such relationship. Note that, for adversary, it means one intends to take some measures to degrade the model performance. However, in the developed model, I cannot see how to resist such attack or the so-called “rival label noise”.

(-) The writing needs significant improvement. Many typos can be found. For example, “accordingly to the Eq.equation 17”. Moreover, the writing quality in the sections of model, theoretical analyses, and experiments, is not as good as that in the introduction. 

clarity,_quality,_novelty_and_reproducibility: The paper is generally readable. It seems that the results are reproducible, but I did not check it completely.
summary_of_the_review: This paper is potentially interesting. However, it contains some immature factors, especially the practical meaning and the handling of adversary, so I think it is not suitable to be accepted currently.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Unfortunately, the reviewers were not enthusiastic enough about this paper for it to be considered for acceptance at ICLR 2023. There are just too many papers that reviewers were much more enthusiastic about this year, and ICLR has a very low acceptance rate. The authors are encouraged to take the reviewer comments very seriously, even if there are things you disagree with, and make sure that all issues are addressed, and any potential sources of confusion are completely eliminated, in the next version of the paper. Also, please ensure that these are addressed in the initial paper submission for that next conference. 
justification_for_why_not_higher_score: see above
justification_for_why_not_lower_score: see above
Decision: Reject