Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper deals with how to estimate the 'noise transition matrix' under a multi-label setup. Like a single-label setup, the authors attempted to make an idea without using 'anchor points' for matrix estimation. They leverage the 'sample selection' method (ie., GMM) for extra clean supervision and provide a detailed mathematical derivation for the matrix estimation. The experimental section includes VOC and MS-COCO datasets; they injected a synthetic label noise with two factors, controlling the flip ratio of '1'->'0' and '0'->'1'. Under the proposed synthetic noise, the method works well and outperforms other simple extensions of similar works (but developed under the single-label setup).

strengths_and_weaknesses: ### Strengths
1. This paper is the first approach to estimating the noisy transition matrix for handling noisy labels with multi-labels. 
2. Leveraging mismatched label correlation is useful.
3. The method shows higher performance than other simple baselines.

### Weaknesses
I agree that estimating the noise transition matrix helps make a statistically consistent classifier. However, I felt some major weaknesses of this paper, as below:
1. **Assumption of Instance-independent Noise.** As the authors said, many previous studies make the same assumption for matrix estimations. But, there have been numerous recent studies to overcome this in the same direction [1-3]. I agree that dealing with 'instance-dependent' label noise in a multi-label setup is very difficult to address. At least, the author should mention what is more challenging in this setup compared to the single-label case. And, why making this assumption is reasonable with a more specific reason. Just saying 'the vast majority of current algorithms mainly focus' is not enough as a research paper.
2. **Noise Injection Protocol.** According to the paper, the authors assume the class-dependent label noise and aim to estimate an accurate noise transition matrix under this assumption. However, the noise injection protocol in Line 217 looks like 'class-independent label noise. Specifically, the $\rho just means the probability of '0' (or '1') being flipped to '1' (or '0') without any consideration of class pairs like <i, j>; only class j associates with the flipping and no connection between classes. Therefore, this is not very realistic and not class-dependent label noise. Did I get it wrong?
3. **Unrealistic Evaluation.** Related to the second weakness, the author should have included at least one realistic real noisy data. There are benchmark data with noisy multi-label instances, called OpenImages database. The author can find other better datasets if possible. Without realistic evaluation, I am not convinced of the robust results in the paper since the injection protocol is very unrealistic.
4. **Class Imbalance Problem & Low mAP.** Unlike single-label classification, the biggest difference in multi-label setup is 'class imbalance'. The authors also mentioned that this problem is very severe in Lines 44 & 102. However, there is no detailed mention of how to resolve this issue with the proposed idea throughout the paper. Next, when I am looking at the results on MS-COCO, the mAP value is too low compared with other multi-label papers using the same ResNet-50 pre-trained models. As an example, [4] shows ResNet-50 with GAP head achieves around 80mAP on MS COCO dataset (refer to Figure 6). However, in this paper, the mAP was less than 70 with Standard at the easiest noise setup (0.0, 0.2); this is a 20% missing label scenario. There are 20% missing labels but I think 20% of missing labels do not make 10mAP drops. Could you report the performance of your method and others under (0, 0) setup? This is a very good reference to check whether your implementation is correct and whether your method is still comparable with a zero-noise setup.

**[1]** Approximating instance-dependent noise via instance-confidence embedding, arXiv 2021

**[2]** A Second-Order Approach to Learning With Instance-Dependent Label Noise, CVPR 2021

**[3]** An Information Fusion Approach to Learning with Instance-Dependent Label Noise, ICLR 2022

**[4]** ML-Decoder: Scalable and Versatile Classification Head, arXiv 2021

questions: Please see the weakness section above.
limitations: Please see the weakness section above. In addition, the paper writing should be improved, particularly the theoretical section. Too many results are omitted and borrowed from [16, 24, 25, 31].
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new estimator to estimate the transition matrix in noisy multi-label learning. The main idea of the estimator is to utilize the clean label co-occurrence as a constraint to help the estimation of the transition matrix. Specifically, authors derive a two-stage method to estimate then transition probabilities. At the first stage, the paper utilizes an existing technique to select clean labels for estimating the label occurrence. Although the estimated label occurrence is biased due to the selection bias, authors claims that the selection biased cannot lead to large estimation error. At the section stage, the paper obtains the label co-occurrence with frequency counting and use the mismatch of label correlation to estimate the transition matrix.
strengths_and_weaknesses: Strength

1. The motivation of the proposed method is reasonable. The information of label co-occurrence is useful for the transition matrix estimation.

2. The paper reports extensive experiments to validate the effectiveness of the proposed method.

3. The paper conducts comprehensive analyses for noisy multi-label learning.
 

Weakness

1. In section 3.2, authors claim that the label co-occurrence estimated by using selected clean labels is unbiased. This is based on the assumption that “given Y^j”, the features about class j is biased, while the features about another class i is unbiased. The assumption is unreasonable, since the features are biased with respect to class i and class j co-occur in an image.

2. Authors select a small number of clean labels for estimating the label co-occurrence. However, the estimated label co-occurrence may be imprecise due to the insufficient labels used for frequency counting, especially for positive labels.

3. There are many typos, such as page 4 line 168 Ys; Algorithm 1 line 4 Dt; page 2 line 48 “while “bird” and “sky” are always co-occurrence”.

4. The references are not cited properly. In experiments, the citations for multi-label with missing labels and partial multi-label are missed.

questions: How to determine the threshold \tau, especially without the clean validation set?
limitations: None
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: 
This paper presents a way of estimating the noise transition matrix for noisy multi-label learning, which makes use of label correlations through a bilinear decomposition based on frequency counting. It first theoretically studies the identifiable problem of estimating the noise transition matrix under the multilevel setting,  which then leads to the development of the bilinear decomposition-based method for estimating the noise transition matrix.  Experiments results derived from several image datasets show that the proposed method can better estimate the transition matrix.
strengths_and_weaknesses: Strengths

Label-nosing learning is an important and challenging problem not just in multi-class classification and also in the multi-label setting.  This paper presents a bilinear decomposition-based approach based a simple frequency counting with a clearly described algorithm. 

The proposed estimation method is well motivated with a set of theorems that study the identifiable problem in the multi-label learning setting, which seems to be thorough with proper proof in the Appendix. The proof is also easy to follow. 

Overall, the paper is written well, and most of the notations were clear and well-organized to guide the readers through the text. 


Weaknesses

My major concern is around the performance of the proposed estimation method.  In terms of estimation of the transition matrices, it outperforms the competitors in most of the settings, which is good. However, when coming to the classification performance, the proposed method is not a clear winner, particularly compared with those without considering noise labels across the datasets. For instance, for varying levels of noise, it is hard to conclude the behaviour of the proposed method. 

Ablation studies showing how the proposed model benefit from label correlation and deal with imbalance issue are missing. For example, there is a running parameter $\tau$ used in Stage 1 to select the sample set $\mathcal{D}^j_t$, which I believe will have an impact on the performance of the proposed method. 
questions: 1. Label correlation plays an important role in the proposed estimation. Even though assumption 2 is validated in the appendix. Could the authors empirically show this role with some examples from the experiments? For example, labels can be strongly correlated and or weakly correlated. Is it possible to study the association between the strength of label correlation and the model performance? 
2. The server imbalance issue is indeed a challenging problem in multi-label learning, which I agree with the authors. I am wondering if the proposed method can deal with this issue? If so, how?
3. Other common performance metrics used in multiple-label learning are the ranking-based metrics, like P@K, R@K, NDCG@k, etc. Could the authors consider one of the metrics, say NDCG@K? 
4. How will the threshold $\tau$ affect the model performance?
5. Given the mixed performance of the proposed method,  Is there any consistent takeaway message here? I.e., In which circumstance the proposed method can be applied and perform better?
6. What is $n_a$ in computing $\rho{-}$ in line 222?
limitations: The authors have addressed the limitations and border impact in the appendix.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper discusses the estimation problem of the transition matrices in the noisy multi-label setting. They study the identifiability problem of the class-dependent transition matrix in noisy multi-label learning and propose a new estimator by exploiting label correlations without both anchor points and accurate fitting of noisy class posterior inspired by the identifiability results. Experimentally, the effectiveness of the proposed method is also illustrated.
strengths_and_weaknesses: **Strenghths:** 
1. They utilize the mismatch of label correlations to identify the transition matrix without both anchor points and accurate fitting of noisy class posterior in noisy multi-label learning.
2. The method is effective to the issue with both theoretical analyses and empirical results support this method.
3. Clear writing and structure.

**Weaknesses:** 
1. Some formulas like in line 192 are recommended to be written in more simplified way.
2. The experimental results are recommended to be presented in different forms but not just tables.
questions: 1. Can the authors clarify the originality of the proof of Theorem 1-4? 
2. Some dots should be replaced by $\cd$, e.g. the dots in line 190 and line 204.
limitations: They address it adequately in Appendix.
review_title: null

Meta-review: metareview: Estimating the noisy transition matrix for handling noisy labels with multi-labels.  Good experimental work illustrating estimating transition matrices.  reviewers liked theory and the writeup.   Paper has had improved citations and writing.

There was some discussion about the assumptions.  Nuances of this should be addressed in the revised paper, for instance your comments about class imbalance.

Regarding reviewer KMFh's Q5:  Note retrieval metrics (e.g., R@K) have been widely used in multi-label classification, although versions of F1 are probably more common.  They give alternative looks at the errors.

Regarding Reviewer n5cR's weakness 2: would be nice to do summary plots and/or win/loss tables and put some of the big tables in appendices.


confidence: Certain
Decision: Accept