Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper addresses the low-label regime of multi-label recognition by introducing an image-text framework. A light learnable overhead is added to pre-trained vision-language models which can quickly adapt to MLR tasks with unseen classes. On two selected benchmarks the proposed method demonstrated promising results.
strengths_and_weaknesses: The MLR task with a low-label regime is a valuable problem in practice.

Introducing prompts improves the feasibility of the solution.

Extensive experimental results.

Good presentation.

- Lack of theoretical analysis of the logits.
questions: What are the key theoretical contributions of this paper?

What are the most challenging technical problems addressed in this paper?

When handle the positive and negative prompts, how is it compared to contrastive learning?
limitations: The theoretical contribution is limited despite a fancy task with a novel and practical framework design.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper solves the multi-label recognition tasks with limited annotations via pre-trained vision-language models (e.g., CLIP). The approach involves the learning of dual prompts, which significantly reduces the amount of learnable parameters. The experiments were conducted on partial-label and zero-shot MLR benchmarks and showed state-of-the-art performance.
 
strengths_and_weaknesses: This paper unifies partial-labeled and zero-shot multi-label recognition problems and presents a solution to address both challenging problems. The approach takes advantage of powerful vision-language models and learns a pair of positive and negative prompts as the textual input, which is different to previous MLR methods. Results are promising in that the proposed method outperformed state-of-the-arts in partial-labeled and zero-shot MLR benchmarks. Ablation studies on design choices are also provided.
 
questions: 1. Although this paper aims to address limited-annotation MLR, the proposed method should also work for typical (supervised) MLR, in which all labels are available for optimization. I would like to know authors' thoughts on this point.
2. I am not quite sure how the method works for zero-shot MLR. I can understand that a pair of positive and negative prompts for each class is learned when solving MLR with partial labels. In that case inference can be made by directly comparing the positive and negative logits to make a decision for each label. For zero-shot MLR, only a pair of prompts are learned (they are shared for all classes in zero-shot MLR). In that case, how does the inference proceed?

limitations: Yes, authors have provided two paragraphs in the conclusion section to describe the limitations and potential negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a method for the partial- and zero-label multi-label image classification problems. Specifically, the paper extends on CoOp [70], where a learnable prompt is used combined with the image to tune CLIP-like architecture to downstream tasks; and proposes to use two learnable prompts, one for the positive label (+1) and the other for the negative label (-1) of the same class, dubbed DualCoOp. Moreover, the paper also proposes to calculate the similarity scores between the feature from each image region and the text features, then aggregates all the logits. The proposed method is evaluated on VOC, COCO, and NUS-Wide in both the partial-label setting between 10-90% labels and the zero-label(shot) setting.
strengths_and_weaknesses: The main strengths of the paper are:

1.  The paper proposes a simple yet effective idea to extend CoOp [70] to DualCoOp, with two learnable prompts for negative and positive labels. This idea intuitively is suitable for multi-label learning and performs well based on the experimental results shown in the paper. However,  I do have some questions and doubts about the explanation of the effectiveness of the proposed method and ablation study.
2. The paper is well written and easy to follow. The proposed idea should be easily reproducible. Moreover, the experiments are quite thorough in general, albeit I would like to see more ablation of the baseline methods, which I will explain in the weaknesses and questions.

The main weaknesses of the paper are:

1. As the idea is based on CoOp [70], I would like to see a thorough ablation compared with CoOp, especially in Table 1 and for the partial-label setting. I understand that M3 in Table 8 is basically the same as CoOp, but I think the threshold choice of 0.5 is not fair to methods that rely on thresholds. I would like to see the mAP number of Table 8, which does not rely on threshold selection. 
2. The paper overempharized the idea of learnable parameters and ignored what will this actually translates into real-world effect. Based on my understanding, although the learnable parameters are small, the actual inference latency would be greater than a straightforward multi-label classification network. I would like to see actually training/inference time and memory consumption in the same hardware environment rather than just the parameters.
3. Missing info on other baseline methods and possible unfair comparisons. I appreciate the papers' efforts in using CLIP initialization for  SST [7] and SARB [45], but for researchers that are not absolutely certain about the details of the compared method, I would suggest listing the training/inference resolution, backbone architecture, and pre-training data for each method. Pre-training on 400M obviously would bring benefits to downstream tasks than just IN-1K (as shown in the comparison in SST and SARB), so I would say most of the comparisons in the paper do not have much value in understanding which method is better in what situation, besides adding one more datapoint confirming CLIP is good. I would rather see more comparisons with CLIP-based methods, such as CoOp.
questions: Please refer to the weaknesses section for details.  I would lean more towards accepting if the authors could answer my questions and provide the data points during rebuttal.
limitations: NA
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles the multi-label recognition (MLR) task in the low-label regime. The authors propose DualCoOp in order to adapt powerful vision-language models (such as CLIP) to solve MLR, by using dual (positive and negative) prompts. The paper presents results on partial label MLR and zero-shot MLR on datasets such as MSCOCO, PASCAL VOC etc.
strengths_and_weaknesses: The paper is well-written and it has a strong related work section. The experimental comparison with other state of the arts methods is very good the authors compare their method with the most recent works.
One main advantage of the method is that it only trains a light-weight head in order to adapt CLIP for MLR, which translates in having just a few learnable parameters.

The proposed method is similar to CoOP, so a more in-depth comparison would be beneficial.

Some ablations of the Class-Specific Region Feature Aggregation module are missing.

Overall the presented work is well-written and is easy to understand the overall idea. Also the method achieves very good results as compared to other works.
questions: Would fine-tunning CLIP in conjunction with DualCoOp produce better results?

Would the method work on a dataset that is from a different domain as compared to what was CLIP trained on?

How would the results of training with 100% of data compare against some fully supervised methods?

How would using different amount of N+ and N- affect the performance? Would it be more useful to have more N+ than N-?
limitations: The authors discussed the limitations and the potential negative societal impact.
review_title: null

Meta-review: metareview: This paper’s DualCoOp extends the previous CoOp prompt learning framework to multi-label and multi-label zero-shot recognition. Reviewers were broadly positive, appreciating the writing, good results, and overall idea of exploiting pre-trained CLIP models for MLR. Questions focused on comparison to vanilla CoOp, evaluation in the fully supervised regime, inference cost and impact of fine-tuning. These were all generally resolved during author feedback phase. Since all reviewers are positive and questions are resolved, I recommend accept. 

confidence: Certain
Decision: Accept