Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper tackles the partial multi-label learning problem from a probabilistic formulation with theoretical interpretation and derives a novel objective function based on the concept of Mutual Information (MI). To optimize the objective function, the authors propose an optimization algorithm to identify ground-truth labels and update classifier alternatively.
main_review: The idea of studying the PML problem with tools from information theory is novel. The training objective function derived by the authors is reasonable and can be well explained theoretically. The technique and theoretical parts are solid and seem sound to me, which can benefit the following research in PML. The contribution is highly related and important to the community.

The overall structure and writing are clear and easy to follow.

I have some questions: 
- Why there are no related references following the statement “the state-of-the-art PML methods” in the first section?
- In the paper, the authors claim that y, y_v are the vectors, but describe that “y_v is selected from y”. I believe there are some notation abuses, and the authors should modify it to make the mathematical expression more strict.
- The notation v is used as vector in section 3, but used as the identified ground-truth label set in section 4. Different notations are suggested to use.

limitations_and_societal_impact: - Why there are no related references following the statement “the state-of-the-art PML methods” in the first section?
- In the paper, the authors claim that y, y_v are the vectors, but describe that “y_v is selected from y”. I believe there are some notation abuses, and the authors should modify it to make the mathematical expression more strict.
- The notation v is used as vector in section 3, but used as the identified ground-truth label set in section 4. Different notations are suggested to use.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents two fundamental questions about the assumption of existing PML research. The authors derive a novel PML method from a probabilistic formulation which meets the $\epsilon$-identifiable definition. They provide theoretical analysis and algorithms to verify the proposed two questions. Thorough comparative experiments validate the superiorities of the proposed method.
main_review: Pros: 
This paper provides a new insight into partial multi-label learning problem from mutual information perspective, which is novel and highly related to the NeurIPS community. The presentation of the paper is clear and the relation with prior work is well-explained. A new objective function for PML is derived from a probabilistic formulation and a novel definition to evaluate whether a PML method can identify the ground-truth labels precisely is presented. The theoretical analysis is sound and solid. Thorough comparative experiments have been conducted to validate the superiorities of the proposed method, which has good academic value. 

Cons:
1.	From Table 3 to 5, we can see that the proposed method does not always perform the best, such as, onYeastBP and Eurlex-ed datasets. How do the authors view this?
2.	Some details need elaboration. How to calculate the probabilities in Eq. (18) and Eq. (21)? Besides, it is pointed out that LIBLINEAR with L2-regularized square hinge loss is employed to train the binary classifiers. How is the L2-regularized square hinge loss incorporated into the optimization algorithm designed in section 5?
3.	In Algorithm 2, the while loop of the condition should be explicitly explained.
4.	There are some typos, such as “fundermental”, “ojective”. The authors should carefully proofread this paper and correct all the typos.

limitations_and_societal_impact: 1.	From Table 3 to 5, we can see that the proposed method does not always perform the best, such as, onYeastBP and Eurlex-ed datasets. How do the authors view this?
2.	How to calculate the probabilities in Eq. (18) and Eq. (21)? 
3.	How is the L2-regularized square hinge loss incorporated into the optimization algorithm designed in section 5?
4.	In Algorithm 2, the while loop of the condition should be explicitly explained.
5.	There are some typos, such as “fundermental”, “ojective”. The authors should carefully proofread this paper and correct all the typos.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses the problem setting of partial multi-label learning (PML). The task is to select the ground-truth labels from a set of label candidates and disregard false-positive or noisy labels that may occur in the labelling process. The authors propose MILI-PML that exploits dependencies between labels and features for selection and relies on the assumption that ground-truth labels have a higher overall dependency between labels and features than false-positive ones. The paper provides theoretical analysis to show that the model can identify ground-truth labels from a candidate set correctly and propose a corresponding optimization procedure. Further, the authors conduct experiments to demonstrate the effectiveness of MILI-PML and claim that the method achieves superior performance compared to other SOTA methods.

** Thanks for the response to this review. 
main_review: The paper considers the problem setting of PML from an information-theoretic perspective and justifies the proposed method MILI-PML accordingly. With this perspective, the authors provide a contribution to the field of PML. The authors include recent methods and compare the performance of the proposed method against those in the experimental part.

The quality of the paper is not sufficient yet and needs improvement. Some details of the theoretical analysis need to be clarified, and the experimental part needs to be extended by additional assessments, especially concerning the limitations of the proposed method.

The paper makes two assumptions on which MILI-PML is based: (1) "that the overall correlation between a ground-truth label $y_k$ and features should be stronger than the overall correlation between a false-positive label $y_{\bar{k}}$ and features" and (2) "It is naturally hold that the KL-divergence value should be smaller when the ground-truth label $y_k$ is added to the existing identified label set $v$ than that of the false-positive label $y_{\bar{k}}$".

The first assumption seems plausible on an intuitive level. However, it is unclear to what extent this assumption is fulfilled in the datasets used. The reported results give little information about the assumption, as only the performance on different evaluation metrics is reported. Whether the assumption tends to be fulfilled within a dataset could be shown by providing measures of the dependency between individual features and labels, such as a suitable correlation coefficient. Further, it is unclear to what extent MILI-PML exploits dependencies between features and labels. Therefore, it would strengthen the paper to assess how the performance of MILI-PML to select or reject a label correctly is related to the level of dependency between the features and a ground-truth or false-positive label. This influence could be analyzed by generating artificial data in which the dependency between a label and the features is controllable and evaluating the performance of MILI-PML under different configurations. This might also indicate potentials limitations of the proposed method.

Regarding the second assumption, I suppose that the relation between $D_{KL}(P(Y_v \cup Y_{\bar{k}} | X) || Q(Y_v \cup Y_{\bar{k}} | X,w))$ and $D_{KL}(P(Y_v \cup Y_{k} | X) || Q(Y_v \cup Y_{k} | X,w))$ depends on both the choice of model $Q$ and its parameterization $w$. I expect that depending on the choice of $Q$ and $w$, the relation of the two KL-divergences can take on any value. Please clarify why your assumption that in general $Q$ approximates the true probability distribution $P$ better when a ground-truth label is added to $v$ rather than a false-positive one holds.

In Chapter 6.2 (and Chapter 3 of the Supplementary Material), it is stated that the hyper-parameters of the baselines were set to the corresponding values of the original papers. Optimize the hyper-parameters for the baselines for a fair comparison with MILI-PML and state the range of hyper-parameters considered and methods used for selection.

The submission is generally well written and organized. Mathematical statements are presented in (great) detail and are mostly easy to follow. Please see section „Questions and minor remarks“ for minor issues. However, I would recommend a language review before publication.

With the provided information about the implementation and the experiments, it is not yet possible to reproduce the results. Therefore, please provide the following details to improve reproducibility:

Number of evaluation runs and calculation of error bars in the results tables.
Detailed descriptions of how the first and third quantities are calculated in Eq. 21.
Detailed descriptions of how the real world datasets are used in the experimental part. For example, the MIRFlickr dataset [1] contains Flickr tags (raw form and processed form) and annotations. It is unclear which of these are used as labels, as the relevant statistics do not match the information on the website.
The experimental section indicates promising results for MILI-PML and could be of significance for the area of PML due to both the performance improvement and the theoretical analysis. However, the experimental part does not yet evaluate the method thoroughly enough, and therefore the significance cannot yet be assessed finally.

Questions and minor remarks:

Can you explain the change of distribution in Eq. 7 in which the joint probability distribution of features and labels $P(x,y)$ is exchanged with the joint probability distribution of disregarded labels and selected labels $P(y_{\bar{v}, y_v})$?

Minus is missing at the last line of Eq. 1 in the supplementary material.

Typos or missing words in line: 56, 79, 136, 147, 151, 175, 184, 258, 278, 283.

[1] http://press.liacs.nl/mirflickr/
limitations_and_societal_impact: The paper partly addresses the limitations of the proposed method by stating assumptions that need to be fulfilled. It does not address the potential negative social impacts of the application of the proposed method. The authors state in the theoretical analysis that the algorithm only works under given conditions and certain assumptions. Limitations due to configurations (for example, which predictive model $Q$ is used) are not described. The experimental part does not examine whether the central assumptions about dependencies in the data hold and to what extent MILI-PML exploits them. I would appreciate it if the authors would comment in the paper on the potential impacts of their method. For example, I would think the method could be of interest in the context of automatic filtering of "noisy" or "false" labels and could lead to bias or discrimination depending on the domain of the data (e.g. personal data). Moreover, there might be a risk considering the quality of the data used. For example, if a large amount of structurally misannotated data is included in the underlying data set, the central assumption could be affected.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Partial multi-label learning has appeared to be more useful in many applications. This paper first investigates some basic problems existed in the current research, and proposes a new method based on the mutual information, which appears to be interesting to the community.
main_review: - This paper first investigates two basic issues existed in the current research of partial multi-label learning and provides the solid technical solutions with theoretical guarantee.

- A new method based on mutual information is proposed in this paper. The motivation and derivation are clear. The idea is interesting, which could motivate the community to bring some new tools from other domains to address partial multi-label learning tasks.

- The extensive experiments are conducted on various data sets to verify the superior performance of the proposed method.

limitations_and_societal_impact: - Some state-of-the-art partial multi-label references are missing, such as 1) Partial Multi-Label Learning with Label Distribution 2) Noisy label tolerance: A new perspective of Partial Multi-Label Learning 3) Partial multi-label learning with mutual teaching.

- The explanation of Theorem 1 is weak; the author should provide more explanations.

- Can the author do the experiments on the image data set?
review_title: null

Meta-review: The author response clarified the main concerns regarding the paper, notably regarding the assumptions and the presentation. The reviewers agree that the paper makes a strong contribution to the PML setting, and that the experiments convincingly support the approach.
Decision: Accept (Poster)