Contrastively Enforcing Distinctiveness  for Multi-Label Classification
Authors: Son Duy Dao
He Zhao
Dinh Phung
Jianfei Cai
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-label Classification
Contrastive learning

Abstract: Recently, as an effective way of learning latent representations, contrastive learning has been increasingly popular and successful in various domains. The success of constrastive learning in single-label classifications motivates us to leverage this learning framework to enhance distinctiveness for better performance in multi-label image classification. In this paper, we show that a direct application of contrastive learning can hardly improve in multi-label cases. Accordingly, we propose a novel framework for multi-label classification with contrastive learning in a fully supervised setting, which learns multiple representations of an image under the context of different labels. This facilities a simple yet intuitive adaption of contrastive learning into our model to boost its performance in multi-label image classification.
Extensive experiments on two benchmark datasets show that the proposed framework achieves state-of-the-art performance in the comparison with the advanced methods in multi-label classification.

Under review as a conference paper at ICLR 2022
CONTRASTIVELY ENFORCING DISTINCTIVENESS FOR MULTI-LABEL CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, as an effective way of learning latent representations, contrastive learn- ing has been increasingly popular and successful in various domains. The success of contrastive learning in single-lahel classifications motivates us to leverage this learning framework to enhance distinctiveness for better performance multi- label image classification. In this paper, we show that direct application of contrastive learning can hardly improve in multi-label cases. Accordingly, we propose novel framework for multi-label classification with contrastive learning in a fully supervised setting, which learns multiple representations of an image under the context of different labels. This facilities simple yet intuitive adaption of contrastive learning into our model to boost its performance in multi-label im- age classification. Extensive experiments on two benchmark datasets show that the proposed framework achieves state-of-the-au performance in the comparison with the advanced methods in multi-label classification.
INTRODUCTION
Multi-label image classification is fundamental and practical computer vision task, where the goal is to predict set of labels (e.g., objects or attributes) associated with an input image. It is an essential component in many applications such as recommendation systems (Jain et al. 2016 Yang et al. 2015), medical image diagnosis (Ge et al., 2018b), and human attribute recognition (Li et al. 2016b). Compared single-label cases, labti-label classification is usually more complex and challenging.
Recently, contrastive learning (CL. (Chen al., 2020 He al. 2020 al. 2020 Caron al.
2020 Bachman et al., 2019) has been shown as an effective pretext approach to learn latent rep- resentations an unsupervised way, which can be further used for supervised tasks. general, CL aims pull together an anchor and similar (or positive) sample in embedding space and push apart the anchor from many dissimilar (or negative) samples. Therefore, the choice of the positive and negative samples of anchor is key to achieving good performance with CL. In self- self-supervised CL (Chen et al., 2020), the positive sample is defined as those augmented from the same image with the anchor, while the negative samples supervised CL. (Khosla et al.] 2020) Sun et al., 2202 Yuan et al.) 2021 Huynh, (021) has been pro- posed, where all the images with the same label the anchor are considered the positive samples and vice versa for the negative ones. Supervised CL has shown improvements in single-label image classifications than the self-supervised counterpart. With the above successful examples, CL has drawn significant research attention and has been applied in other tasks including image segmenta- tion (Wang et al. 202I), adversarial training (Kim et al., 2020). and text to image learning (Radford) et al. 2021)
all the other images in the minibatch. More recently,
Given the appealing properties and promising results of CL in single-label classification, it natural to adapt it into multi-label cases to boost performance. However, this adaptation is non-trivial. In single-label cases, an image usually contains one salient object, thus, the label of the object can also be viewed as the unique label of the image. Therefore, it is reasonable to use one image-level representation of an image and to push the representation of the anchor close to its positive samples (e.g, augmentations of the anchor images with the same label the anchor), supervised and supervised CL. However, with a single image-level representation for an image, is hard define the positive negative samples for anchor image by its multiple labels in the
done in self-

---

Under review as a conference paper at ICLR 2022
multi-label classification. For example, it would not be reasonable to assume that the image-level representations of images containing apples must always be close to each other, as apple is just one of many objects in those images and an apple may only take small area of an image. As result, this setting hinders the application of existing CL methods to multi-label classification.
To bridge this gap, in this paper propose novel end-to-end framework for multi-label image classification that leverages the contrastive learning principle, termed MulCon or Multi-label Clas- sification with Contrastive Loss. Instead of using image-level representations as in previous CL methods, we introduce a new module that learns multiple label-level representations of an image, which are generated with the attention from the globally Dlass-specific embeddings to the image features learned by convolutional neural network (CNN). Each label-level embedding of an image corresponds to the image's 's representation the context of specific label. With these definitions, the supervised CL loss can be applied. Specifically, if we look at one specific label and view label- level embedding of an image as the anchor our proposed CL. framework, is straightforward to define the positive samples of anchor, which are the label-level embeddings of the other images in the minibatch with the same label and vice versa for the negative samples. For example, instead of sharing image-level representation with other objects, the apple object of an image has its own embedding and the embeddings of the apples of all the images in a minibatch are pushed close to each other. Therefore, our framework is intuitive in the multi-label setting. In this way, the CL loss can enforce the coherence and consistency of the label-level representations of images, which further provides more discriminative power of the prediction procedure based on these representations.
The main contributions can be summarized as follows: 1) Contrastive learning has been shown successful in many single-label classification problems, however, it is non-trivial to apply it in multi- label classification. We propose an intuitive and conceptually simple framework that encompasses contrastive learning. 2) Along with the proposed framework, we also introduce a practical training scheme of contrastive learning for multi-label classification. 3) We conduct extensive experiments on large-scale benchmark datasets, showing that the proposed framework achieves the state-of-the- art performance in multi-label image classification.
PROPOSED METHOD
In this section, we first discuss about the background knowledge and relevant notations and then elaborate on the details of the proposed framework, MulCon.
2.1 BACKGROUND AND NOTATIONS
Multi-label Classification Following the standard setting of multi-label image classification, we denote a minibatch of input images by X E NNN x Hx3 where N is the batch size, H and W are the height and width of the images. Each image ii X is associated with multiple labels selected from a set of L labels in total, which are denoted by multi-hot binary vector yı E {0, 13 For an active label of x4, Vij = 1 and vice versa. Our task build an end-to-end model that takes xy to predict its labels yi-
The Attention mechanism (uong et al., 2015 Xu et al. 2015) has been widely used in various areas of computer vision and natural language processing, which enhances the important parts of the data of interest and fades out the rest. Assume that ny query vectors of size d, denoted as Q E R xd,, and n. key-value pairs denoted as K E RRd.dd, V xd.. The attention function maps the query vectors Q to outputs using the key-value pairs as follows:
Attention
Au(Q, =
where the dot product (QKT) X and w(-) is softmax function. The dot product returns the similarity of each query and key value. The output (QKK")V E xd,, is the weighted sum over V. where larger weight corresponds to larger similarity between query and key.
A powerful extension to the above (single-) attention mechanism is the multi-head attention in- troduced in (Vaswani et al., 2017). which allows the model jointly attend to information from different representation subspaces at different positions. Instead of computing a single attention function, this method first projects Q, K, V onto h different vectors, respectively. An attention
2

---

Under review as a conference paper at ICLR 2022
function Att(-) is applied individually to these h projections. The output is linear transformation of the concatenation of all attention outputs:
MultiAut(, K, V) = concat(O1, O2
OO,)W" "',
O₂ Au(Qw3, KW VWF) for E e,,,..,
(2)
where W W gfa W wy are learnable parameters of some linear layers. QW? E R KW RT xd VW R xd are vectors projected from Q, K, respectively. dy = dg/h and dh d./h.
Following the architecture of the transformer Vaswani et al. 2017 Lee et al. 2019), we define the following multi-head attention block:
MultiAuBlockI K, V) = Layer + Q'w"),
Layer Norm(concat
QW7) + MultiAn K,V)
where WV' E Rdd xd a learnable linear layer. Base on the multi-head attention block, we further define self-attention block follows:
SA(X)= MultiAAtBlock(X,XX
(4)
Contrastive Learning Contrastive learning (CL) has been an increasingly popular and effective representation learning approach (Chen et al. 2020 He et al. 2020 Khosla et al., 2020). As the first proposed CL approach, self-supervised CL (Chen et al. 2020) is proposed to learn presentations in an unsupervised manner. Specifically, a mini-batch is constructed from the original input images and their augmented versions. Given minibatch of 2.V instances I {1...2N] and an anchor instance E I, the augmented version of i, denoted as ia E is considered the positive sample, and the other 2(N 1) instances within the mini-batch considered negative examples. The loss function of self-supervised CL. is defined as follows:
Lieeff = - -E log exp(zi /T,/T) SEI
(5)
Σ exp(2: 2a/T]
ZaE((1)
where 2; is the image embedding, 2 2i, denotes the inner dot product between two embeddings, R+ is scalar temperature parameter, and A(1) P22 It can be seen that self self-supervised CL pushes the embeddings of the samples augmented from the same image close to each other. More re- cently, the supervised CL (Khosla et al. 2020) adapts CL. into the supervised settings, which utilizes the label information to select positive and negative samples. For supervised CL, the embeddings of the samples with the same labels are pushed close each other, which achieves better performance in classification tasks. For more details of CL. we refer to Liu et al. (2021b). To our knowledge, CL. has not been applied in solving multi-label classification yet.
2.2 PROPOSED METHOD
Unlike previous CL methods for single-label classification that use image-level representations, we propose to learn multiple label-level representations for each image, which facilitates the application of CL in multi-label classification. We introduce MulCon, which consists of three main neural network modules: the label-level embedding network, the contrastive learning projection network, and the classification networks as shown Figure
Label-Level Embedding Network As the key module of our framework, the label-level embedding network takes an image I; as input and outputs its label-level representations, denoted by di RLXD where each row of 93 corresponds to the embedding of the image under the context of a specific label. Specifically, the label-level embedding network consists of two components: 1) The encoder block. We first adopt encoder network that learns the image-level embedding as the backbone model as the backbone model: r; = Enc(x;) E Cxx Xx where C, H, W are the number of channels, height, and width of the output. The backbone model can be implemented with an arbitrary model that learns good image features, e.g.. a convolutional neural network (CNN) such ResNet (He et al. 2016) visual transformer (Dosovitskly al., 2020). We then reshape
3

---

Under review as a conference paper at ICLR 2022
Label-Ievel embedding network
Label-wise attention block
Label-wise attention block
Enc(-)
MultiAtt Block
SA
LabeH embedding
person
f
Figure I: Left: Overall structure of our MulCon framework. It consists of three main components: label-Ievel embedding network that uses label-wise attention block with an Encoder (Enc(-) for extracting label-level embeddings Di from input images, set of independent classifiers fa for multi- label prediction, and projector (Proj(-)) to map label-level embeddings to a latent space for con- trastive learning. Right: Detail of the label-wise attention block. takes image features T2 from (Enc(-)) input and returns label-level embedding 9i output. The module contains several self- attention blocks, a
attention block and a set of learnable label embedding.
E RWH for the next step's processing. 2) The label-wise attention block. In this block, we introduce set of vectors U E R each row of which is be learned as the global embedding for a specific label. To generate an image's 's label-level embeddings Jt. we propose cascade of several self-attention blocks and multi-head attention block to capture the interactions between r; and U as follows:
T: SA(rr: fi MultiAAtBBlock(U,ry.
9: SA(g:)-
To summarise, given an image we first learn its representation E RW by the encoder, each row of which captures the feature of a location in the image. We then pass r; to a self-attention block that captures the interactions between the features of the image. Next, we feed to the multi- head attention block, where the global label embeddings U the query and r; is the key and value. As the sizes of the input and output channels of the multi-head attention are C and D, respectively, we have ng = T4 = and d, = ny = for Eq, 2 With the multi-head attention block, our model can learn the "importance" (attention weight) of the image feature to a specific label. For example, an image contains an apple, the corresponding embedding is expected to be associated with a large attention weight from the apple label. The multi-headed attention its attention to multiple objects in an image, i.e., each the attention heads can generate attention scores for class-specific embedding over all the lmaee-level embeddings. In this case, if image consists of multiple apples, each of the apples receives specific attention score from the apple label. Following up the multi-head attention a self-attention block that implicitly helps improve label correlation. With the attention from all the labels, we can derive the label-level embeddings gi of the image from its image-label embedding r;-
also help label pay
R..DD we use 9ty RR. to denote the label-level embedding of the input image under the context of a specific label (1..... I.)). Following Chen et al. (2020); Khosla et al. (2020), our framework includes a projection network (-oj(-) that maps 9ı to vector in another embedding space: Zij = Proj(gy) E RRd where the contrastive learning is performed.
Contrastive Learning Projection Network After obtaining
Classification Network Recall that the label-level embedding giz captures the input image i's fea- ture under the context of the label Thus, it can be used to predict whether j is active in i. Accord- ingly, we introduce a fully connected layer as a classifier f! to predict the probability of the label j being active. Specifically, for each label j e L the prediction score is siz (9x)) (0, We further denote E (0, 4)
2.3 LEARNING MUL.CON WITH CONTRASTIVE LOSS
After introducing the framework, we describe the learning process of MulCon by showing the loss function first. Based on the label-level embeddings of the image i.e., Gi. introduce loss
4

---

Under review as a conference paper at ICLR 2022
function with two terms: the classification loss and contrastive loss. The overall training process is illustrated in Figure 2
Classification Loss For the classification loss, given the predictive probabilities output from the classification network 8; and the ground-trum -rutt multi-hot label vector yı, we apply the binary cross- entropy (BCE) loss, which has been widely used for multi-label classification:
LBCE
Mig log Sij + (1 - Yis [og)log[1= -8t))
noteworthy that other multi-label classification losses than BCE can also be used our frame- work, e.g., in Benn-Baruch et al. (2020): Lin et al. (2017).
Label-Jevel Contrastive Loss In addition to the classification loss, we introduce the label-level contrastive loss (LLCL) for multi-label classification, which is one of the key contributions of this paper. Because an image is associated with multiple labels, it is hard to directly apply CL as we analyzed before. However, in MulCon, after learning the label-level embeddings for image, show that the multi-label problem can be transformed into a single-label one, where CL can be adapted in straightforward way.
As LLCL works in the projected space, hereafter, we also call 2; the label-level embeddings for image i, which is projected from Pi. Given minibatch of N images, first forward-pass them through the label-level embedding network and contrastive learning projection network and then aggregate the label-level embeddings of all the images into Z {zis E Rd E {1,-.. ,.,,,N,, .N:i (I...., LJ}. Similarly, we define the set of the ground-truth labels of the minibatch: Y = -Mi E (0. 131i E f1..... .N::j E {1,... L}}. If we view an image's label-level embedding 2g as an instance instead of the image itself, 21j is associated with a single ground-truth label Vay. We further define = {=y E Z 1yg = }} as the set that contains the label-level embeddings with active ground-truth labels and A(i,j) j) = / Fij as the set contains the embeddings in I with ziy excluded.
In the minibatch, we now consider Zig E as the anchor, which presents the feature of im- age i under active label j. With LLCL, we aim to push 24j closer to the embeddings un- der the same active label j of other images in the minibatch, i.c., the positive set, defined as P(i,j) {=k E A(i,
Yij = =}}. With above notations and inspired by the supervised CL, we define the contrastive loss for the anchor Ziy as:
exp(zig Zp/T) log E E.nE(())) exp(zij za/ za/T))²
Σ
P((ij))
spE((i)))
The loss for the whole minibatch is: LLLCL = Enser CiLor. Together with the classification loss, we show the overall training loss of MulCon:
L LACE + +.I.ICI
where the parameter y controls the trade-off between the two losses.
Why and How does Contrastive Loss Help? Now would like to answer why contrastive loss helps in multi-label classification, by showing that it serves as an important complementary to the classification loss. Specifically, with the BCE classification loss applied in many multi-label classi- fication problems, each label can be viewed to be classified independently with a specific classifier as discussed in Section 2.2 That is say, each classifier focuses on the classification of spe- cific label and cares less about the distinctiveness of the features of different labels. Distinctiveness in classification means that we expect features of the instances with the same label to be close to each other, which has been known as an important factor for achieving good classification accuracy. enforce
Similar to single-label problems, the label-level contrastive loss in MulCon is designed distinctiveness of the label-level embeddings. To demonstrate this, we show the t-SNE (Van der Maaten & Hinton| 2008) visualization of the (active) label-level embeddings of 1,000 randomly sampled images of the COCO dataset (Lin et al. 2014) in Figure 3 These embeddings are from MulCon trained by the BCE loss (i.e., 7) and the combined loss of BCE and LLCL (i.c., Eq. D. respectively. Each dot represents one label-level embedding under the context of specific label and each color represents one class. It can be seen that compared with the loss with BCE only, the ad- ditional contrastive loss makes the label-level embeddings of the same label fall into more compact
5

---

Under review as a conference paper at ICLR 2022
Euz
embedding
ProojC
1:
f2
f
fc
31
92
$1
SW
Lyer
Figure 2: MulCon has two steps during training: pretrain- ing and contrastive finetuning. The first step is to train the label-Ievel embedding network with binary cross-entropy loss (LBCE) to effectively decompose an input image into several semantic components that the first component corresponds to the first label, etc. The second step is to finetune the previously trained network with contrastive loss (LLLCL) and LEC to improve the quality of label-level embedding.
Figure 3: t-SNE visualization for image components trained with only LBCE (top) and with the combination of LBCE and LLLCL (down).
clusters, which are better separated from the clusters of other labels. This clearly shows the enforced distinctiveness of the embeddings with the contrastive loss.
However, unlike single-label classification, being over distinct in the embedding space is not always good thing in multi-label classification. Specifically, label correlations important in multi-label classification (Chen et al., [2199]] You et al.) (2020). For example, when we see a cup in an image, the probability of a table existing below the cup in the image can be assumed to high. In this case, we actually do not want the embeddings of cups and tables are too far or too separated from each other. Therefore, the enforced distinctiveness with the contrastive loss needs to be judiciously controlled. We propose a simple training strategy to tackle this task, which consists of two steps: the pre-training step and the contrastive learning step. In the pre-training step (Step 1), we pretrain the backbone and label-level embedding network and the classification network of MulCon with the BCE loss only. the contrastive learning step (Step 2), we then plug the contrastive projection network with LLCL but also keep the BCE loss. In the first step, the BCE loss learns the label- level embeddings freely and implicitly obtains semantic structure by the effect of several attention blocks. After the embeddings are learned, of the embeddings. The network fintuned with small learning rate which helps improve feature distinctiveness without breaking the label semantic structure. We empirically find that the propose training policy works well in practice.
finetune them with LLCL to enforee distinctiveness
3 RELATED WORK ON MULTI-LABEL IMAGE CLASSIFICATION
Our paper mainly focuses on two things: contrastive learning and multi-label classification. To knowledge, most of the works in contrastive learning for single-label classification. Thus, although there is an emerging literature of contrastive learning, the details are omitted as our paper solves different problem. Multi-label classification has been challenging and important problem in computer vision, where many methods have been proposed. We mainly consider two lines of multi-label methods
close works.
Methods capturing label/class correlations For example, (Wang et al. 2016 Yang et al. 2016 Liu et al. 2017 Chen et al.) 2018a) explicitly capture the class correlation by CNN-based model

---

Under review as a conference paper at ICLR 2022
followed by Recurrent Neural Network (RNN) (Medsker & Jain 2001 Moreover, there also approaches based on on probabilistic graphical model (Li et al. 2016a 2014) which model label dependencies in the covariance of probabilistic distributions. Finally, as label correlations can be formulated into graphs, some approaches apply Graph Convolutional Network (GCN) (Zhou et al.) 2020) leverage graph structures learn label presentations such in Yang et al. (2016); Chen et al. (2019c); You et al. (2020).(L <Li et al.) (2019) proposes to align image feature with label feature. Compared with these methods, ours is different approach that controls label distinctiveness and correlation via contrastive loss.
Another research line close ours using visual attention for multi- label classification, such as in Guo et al. (2019) Chen et al. (2018b WWan et al. (2017); [a et al. (2015); Huynh Elhamifar (2020): Yazici et al. (2020). In addition, attention mechanisms are also introduced in Chen et al. (2019a); Ye et al. (2020) to capture label correlation. Recently, vision transformers (Dosovitskiy et al. 2020) has been a new paradigm of computer vision tasks and there emerges new works that use vision transformers for multi-label classification (Liu et al. 202 Lanchantin et al, 202T; Cheng et al. 202T), where attention mechanisms are naturally used. In our method, we use the attention mechanism for different purpose, i.e., to assist contrastive learning. Specifically, the attention is used to derive an image's ' label-level embeddings from its image-leved embeddings and the global-label embeddings. Moreover, our framework is flexible to also use vision transformers as the backbone model.
Methods with attentions
4 EXPERIMENTS
In this section, we compare our proposed model with the state-of-the-an multi-label classification methods. More results are shown in the appendix. we use the two most popular benchmark datasets: MS-COCO (Lin et al., 2014) and NUS-WIDE (Chua et al., 2009).. To evaluate the classification performance,
the standard metrics for multi-label classification including the mean average precision (mAP) mean average precision (mAP), overall precision (OP), recall (OR), Fl-measure (OFI) and per-category precision (CP), recall (CR), and Fl-measure (CF1), computed over all pre- diction scores and t highest prediction scores. As F1 is a comprehensive metric computed by precision and recall, we consider mAP, CF1, and OF1 as more important metrics. The formulas of the metrics are shown in the appendix. For a fair comparison, we adopt ResNet-101 (He et al.) 2016) as the backbone for our method to extract image-level features, which the same for the other com- pared methods, unless otherwise specified. The implementation and training details of our methods are shown in the appendix.
4.1 EXPERIMENTS ON MS-COCO
In the standard setting for multi-label classification, MS-COCO contains 122,218 images with 80 different categories, where every image is associated with 2.9 labels on the average. The dataset is divided into 82,081 images for training and 40.137 images for validation. For fair comparison, we choose state-of-the-art methods that use Resnet101 as backbone network, and further divide them into two groups Group I uses input image size 448 448 such as Multi Evidence (Ge et al., 201 20)8a), CADM (Chen et al. 2019b), ML-GCN (Chen et al., 2019c). KSSNet (Liu et al., 2018). MS-CMA You et al. 2020) and MCAR (Gao & Zhou 2021). Group uses larger input image size such as SSGRL (Chen et al. 2019a). C-Taa (Lanchantin et al. 2021) and ADD-GCN (Ye et al. 2020).
The results on MS-COCO are reported in Table T The numbers of the compared methods are taken from the best reported results in their papers. It can be seen that our approach achieves the state- of-the-art results especially on mAP, CFI and OFI. For example, in the 448 448 input resolution with all prediction scores, ours outperforms the second best (MS-CMA and MCAR) by 1.1% on mAP and 0.8% on CF1. With the resolution increased to 576 x 576, ours surpasses the second best (ADD-GCN) by 1.1% on mAP and 0.7% on CF1. Although our method may not always achieve better results on precision (CR and OR), it consistently obtains better performance on Fl, which is more comprehensive metric covering both precision and recall.
7

---

Under review as a conference paper at ICLR 2022
All
Top-3
Method
Resolution
mAp
CP
CR
CFI
OP
OR
OF1
CP
CR
CF1
OP
OR
OFI
Multi Evidence
448 x 448
80.4
70.2
74.9
85.2
72.5
78.4
84.5
62.2
70.6
89.1
64.3
74.7
CADM
148 x 418
82.3
82.5
72.2
77.0
84.0
75.6
79.6
87.1
63.6
73.5
89.4
66.0
76.0
ML-GCN
448 x 448
83.0
85.1
72.0
78.0
85.8
75.4
80.3
89.2
64.1
74.6
90.5
66.5
76.7
KSSNet
448 x 418
83.7
84.6
73.2
77.2
87.8
76.2
81.5
MS-CMA
448 x 448
83.8
82.9
74.4
78.4
84.4
77.9
81.0
86.7
64.9
74.3
90.9
67.2
77.2
MCAR
448 x 448
83.8
85.0
72.1
78.0
88.0
73.9
80.3
88.1
65.5
75.1
91.0
66.3
76.7
MulCon (Ours)
448 x 448
84.9
84.0
74.8
79.2
85.6
78.0
81.6
87.8
65.9
75.3
90.5
67.9
77.6
SSGRL
376 x 5YG
83.8
89.9
68.5
76.8
91.3
70.8
79.7
91.9
62.5
72.7
93.8
64.1
76.2
C-Trans
576 576
85.1
86.3
74.3
79.9
87.7
76.5
81.7
65.7
76.0
92.1
71.4
77.6
ADD-GE GCN
376 x 576
85.2
84.7
75.9
80.1
84.9
79.4
82.0
88.8
66.2
75.8
90.3
68.5
77.9
MulCon (Ours)
376 x 576
86.3
84.7
77.3
80.8
85.9
79.9
82.8
88.6
67.2
76.5
91.0
68.8
78.4
Table 1: Results on the COCO dataset. The best scores are highlighted in boldface. More important metrics including mAP, CFI, and OFI are highlighted in grey.
Method
mAP
CF1
OF1
FitsNet
57.4
54.9
70.4
Method
mAP
CF1
OF1
attention-tramsse
57.6
55.2
70.3
RIO1 + BCH
80.8
76.2
79.2
s-CLs
60.1
58.7
73.3
RIOI + BCE SCL
80.8
76.0
79.1
MS-CMA
61.4
60.5
73.8
LLEN + BCE
83.8
78.8
81.1
SRN
62.0
58.5
73.4
LLEN BCE + LLCL
83.7
78.8
81.1
MulCon (Ours)
63.9
61.8
74.8
MulCon
84.9
79.2
81.6
Table 2: Results on NUS- WIDE dataset. The best scores high- lighted in boldface.
Table 3: Ablation study of different variants and train- ing policies of MulCon.
4.2 RESULTS ON
The NUS-WIDE dataset originally contained 269,648 images from Flicker and has been manually annotated with 81 visual concepts. Since some URLs have been deleted, we follow (Ben-Baruch al. 2020) to obtain 200,000 images with 2.4 labels per image on average. We use the same hyperparameter setting as MS-COCO for training and testing on this dataset. We also select the state-of-the-art methods that report the results on NUS- WIDE, including FitsNet (Romero et al.) 2014), attention-transfer (Zagoruyko Ko & Komodakis) 2016). s-CLs (Liu et al., (018). CMA (You jet al. 2020) and SRN (Zhu al., (017). Following the standard and convention of many other works, we report mAP, CF1, OFI computed with all prediction scores. The results on NUS-WIDD are shown in Table 2 It can be seen that MulCon achieves the best results all the three metrics. For example, it our performs SRN (the second best mAP) by a significant margin, i.e., nearly 2%.
4.3 ABLATION STUDY
To fully understand the modules and training policies of MulCon, provide comprehensive ablation study in this section. Specifically, we are interested in the following variants of our method. 1) R101 + BCE: The backbone model, Resnet101, trained with the BCE loss (i.e., Eq. 7 This is a standard multi-label classification baseline. 2) R101 + BCE + SCL: Resnet101 trained with the BCE and supervised-CL (Khosla et al. (020) (SCL) losses (i.e., Eq. D.. This is variant where the supervised CL loss is directly applicd on the image-level features. Note that for this variant, each image is an anchor, and thus we need to define the positive/negative sets for each anchor. Following the spirit of (Khosla et al. (2020), we consider other images in the minibatch that have at least one common active label with the anchor as its positive samples and the others as the negative ones. 3) LLEN + BCE: The label-level embedding network (LLEN) with R101 as the backbone trained with BCE, which corresponds the MulCon framework trained in Step described in Section 2.3 4) LLEN BCE LLCL: The variant where train MulCon with the BCE and LLCL losses from the beginning, i.c., the second contrastive learning step described in Section 2.3 5) MulCon: The
complete model of MulCon trained with the two-step policy described in Section 2.3
8

---

Under review as a conference paper at ICLR 2022
Table 3 shows the results for the ablation study for method in MS-COCO. We have the follow- ing remarks: 1) By comparing between R101+BCE and RIOI+BCE+SCL. we can see that unlike in single-label classification, directly applying SCL the image-level features cannot improve the performance in multi-label cases (80.8 vs 80.8 mAP). P). 2) It can be observed that by using the pro- posed label-level embedding network instead of R101 (R101+BCE V.S. ILEN+BCE)) we can significantly improve the mAP score (80.8 vs 83.8 mAP). 3) Note that the difference between LLEN BCE + LLCL and MulCon is on the training policies: The former trained with BCE and LLCL from the beginning while the later is first pretrained with BCE (Step 1) and then futhur trained by BCE + LLCL (Step 2). We observe that the LLCL loss is applied from the beginning of the training, it can barely improve the performance. This is because LLCL may excessively enforce distinctiveness which makes the model care less about label correlations, as analyzed in Section 2.3 It can be seen that our proposed two-step training policy improves significantly.
4.4 QUALITATIVE ANALYSIS
To qualitatively study our method, we first exam the semantics captured by the label-level embed- dings learned with MulCon by conducting an image retrieval experiment. Specifically, given an input image Ti, we do forward-past to get its label-level embeddings 9t- With the ground-truth labels of dy we can specify a specific active label j. pick embedding Piy from Ji and use as the query for image retrieval. By comparing the Euclidean distance between the query embedding with the label-level embeddings and label. Figure(4showe the image retrieval results of MulCon and its variant with BCE only. More qualitative analysis is provided in the appendix.
other images, we can retrieve the closest images to our query image
Query
Model with contrastive
Model loss
apple
apple
Apple
apple
apple
56c
teday bear
book
dining table
book
book
book
D00k
book
book
booo
dining table
book
feddy bear
Ieddy
book
Teddy bear
10J0y DEOT
teddy DOOT
5200y
Icddy boat
boddy
Figure 4: Top-4 related images retrieved given an query image and label on COCO dataset. The results for our full model (MulCon) are on the left, and the results for our model without contrastive loss (MulCon with BCE only) are on the right. The label under each retrieved image is the one corresponding to the embedding closest to the picked query embedding.
CONCLUSION
In this paper, we have introduced an end-to-end multi-label image classification framework, Mul- Con, which leverages contrastive learning in multi-label classification. It has been shown that CL is not directly applicable in this domain, due to the fact that it is hard to define the positive/negative samples for anchor with multiple labels. To tackle this issue, we have introduced to lear label- level embeddings for an image with the multi-head attention mechanism. With label-level embed- dings, we transform the multi-label classification into single case for each label-level embedding.

---

Under review as a conference paper at ICLR 2022
which facilities straightforward adaption of supervised CL. We have provided analytical and em- pirical study of why CL helps in multi-label learning and proposed a simple training policy to control the distinctiveness enforced by CL. Extensive experimental results and visualization show the effee- tiveness of our approach and its ability to achieve the state-of-the-art performance in multi-label image classification.
REFERENCES
Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention. In ICLR (Poster), 2015.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximiz- ing mutual information across views. Advances in Neural Information Processing Systems, 32: 15535-15545, 2019.
Emanuel Ben-Baruch, Tal Ridnik, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zennik-Manor. Asymmetric loss for multi-label classification. arXiv preprint arXiv:2009. 14119, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS), 2020.
Shang-Fi Fu Chen, Yi-Chen Chen, Chih-Kuan Yeh, and Yu-Chiang Wang. Order-free mn with vi- sual attention for multi-label classification. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018a
Tianshui Chen, Zhouxia Wang, Guanbin Li, and Liang Lin. Recurrent attentional reinforcement learning for multi-label image recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018b.
Tianshui Chen, Muxin Xu, Xiaolu Hui, Hefeng Wu, and Liang Lin. Learning semantic-speciff graph representation for multi-label image recognition. In Proceedings of the IEEE/CVF Interna- fional Conference on Computer Vision, pp. 522-531, 2019a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Zhao-Min Chen, Xiu-Shen Wei, Xin Jin, and Yanwen Guo. Multi-label image recognition with joint class-aware map disentangling and label correlation embedding. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 622-627. IEEE, 2019b.
Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5177-5186, 2019c.
Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, Dong Shen, Zhongyuan Wang, Nian Shi, and Honglin Liu. Mltr: Multi-label classification with transformer. arXiv preprint urXiv:2106.0619 2021.
Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus-wide: real-world web image database from national university of singapore. In Proceedings of the ACM international conference on image and video retrieval, pp. 2009.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Terrance Vries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: /008. 04552, 2017.
10

---

Under review as a conference paper at ICLR 2022
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 11 (11(1))98-138)))) 1:9,-136, 2015.
Bin-Bin Gao and Hong- Yu Zhou. Learning discover multi-class attentional regions for multi labtl image recognition. IEEE Transactions on Image Processing, 30:5920-5932, 2021.
Weifeng Ge, Sibei Yang, and Yizhou Yu. Multi-evidence filtering and fusion for multi-label clas- sification, object detection and semantic segmentation based on weakly supervised learning. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1277-1286. IEEE, 2018a.
Zongyuan Ge, Dwarikanath Mahapatra, Suman Sedai, Rahil Garnavi, and Rajib Chakravorty, Chest X-rays classification: A multi-label and fine-grained problem. arXiv preprint arXiv:1807.0722: 180..07247, 2018b.
Hao Guo, Kang Zheng, Xiaochuan Fan, Hongkai Yu, and Song Wang. Visual attention consistency under image transforms for multi-label image classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 729-739, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2020.
Dat Huynh and Ehsan Elhamifar. A shared multi-attention framework for multi-label zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 8776-87866 2020.
Su V Huynh. A strong baseline for vehicle re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4147-4154, 2021.
Himanshu Jain, Yashoteja Prabhu, and Manik Varma. Extreme multi-label loss functions for rec- ommendation, lagging, ranking & other missing label applications. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 935- 944, 2016.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33, 2020.
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning. Advances in Neural Information Processing Systems, 33, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR (Poster), 2015.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen. Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsoureed dense image annotations. International Jnternationaaluural of computer vision, 123(1):32-73, 2017.
Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi. General multi-label image classi- fication with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 16478-16488 2021.

---

Under review as a conference paper at ICLR 2022
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-inven invarrian neural networks. In Interna- tional Conference on Machine Learning, pp. 375443753.. PMLR, 2019.
Changsheng Li, Chong Liu, Lixin Duan, Peng Gao, and Kai Zheng. Reconstruction regularized deep metric learning for multi-label image classification. IEEE transactions on neural networks and learning systems, (1(7):2294-22 -2003, 2019.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint 2Xii:2005504466 2020.
Qiang Li, Maoying Qiao, Wei Bian, and Dacheng Tao. Conditional graphical lasso for multi-label image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2977-2986, 2016a.
Xin Li, Feipeng Zhao, and Yuhong Guo. Multi-label image classification with a probabilistic label enhancement model. In Proceedings of the Thirtieth Conference Uncertainty in Artificial Intelligence, pp. 430-439, 2014.
Yining Li, Chen Huang, Chen Change Loy, and Xiaoou Tang. Human attribute recognition by deep hierarchical contexts. In European Conference on Computer Vision, pp. 684-700. Springer, 2016b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2999-3007. IEEE, 2017.
Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, and Changyin Sun. Semantic regu- larisation for recurrent image annotation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2872-2880, 2017.
Shilong Liu, Lei Zhang. Xiao Yang, Hang Su, and Jun Zhu. Query2label: A simple transformer way to multi-label classification. arXiv preprint urXiv:2107100344 2021a.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang. Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE Transactions Knowledge and Data En- gineering, 2021b.
Yongcheng Liu, Lu Sheng. Jing Shao, Junjie Yan, Shiming Xiang, and Chunhong Pan. Multi-label image classification via knowledge distillation from weakly-supervised detection. In Proceedings of the 26th ACM international conference on Multimediaa pp. 700-708, 2018.
Minh-Than Minh-TThag Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421, 2015.
Larry R Medsker and LC Jain. Recurrent neural networks. Design and Applications, 5, 2001.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6553 2014.
Jennifer Sun, Ann Kennedy, Eric Zhan, David J Anderson, Yisong Yue, and Pietro Perona. Task programming: Learning data efficient behavior representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2876-2885, 2021.
12

---

Under review as a conference paper at ICLR 2022
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998 5998-6008, 2017.
Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. Cnn-rnn: A unified framework for multi-label image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2285-2294, 2016.
Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Ex- ploring cross-image pixel contrast for semantic segmentation. arXiv preprint urXiv:2101.11920 2021.
Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, and Liang Lin. Multi-label image recog- nition by recurrently discovering attentional regions. In Proceedings of the IEEE international conference on computer vision, pp. 464-472, 2017.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp. 0048-2057... PMLR, 2015.
Hao Yang, Joey Tianyi Zhou, and Jianfei Cai. Improving multi-label learning with missing labels by structured semantic correlations. European conference on computer vision, pp. 835-851. Springer, 2016.
Xitong Yang, Yuncheng Li, and Jiebo Luo. Pinterest board recommendation for twitter users. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 963-966, 2015.
Vacit Oguz Yazici, Abel Gonzalez- Garciia Arnau Ramisa, Bartlomicj Twardowski, and Joost van de Weijer. Orderless recurrent models for multi-label classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13440-13449, 2020.
Jin Ye, Junjun He, Xiaojiang Peng, Wenhao Wu, and Yu Qiao. Attention-driven dynamic graph convolutional network for multi-label image recognition. In European Conference on Computer Vision, pp. 649-665. Springer, 2020.
Renchun You, Zhiyao Guo, Lei Cui, Xiang Long, Yingze Bao, and Shilei Wen. Cross-modality attention with semantic graph embedding for multi-label classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12709-12716, 2020.
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang. Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6995-7004, 2021.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 162.0099,,
2016.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applica- tions. Open, 1:57-81, 2020.
Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, and Xiaogang Wang. Learning spatial reg- ularization with image-level supervisions for multi-label image classification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5513-5522, 2017.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
In this section, we provide more details on the following subjects:
Implementation Details of MulCon
Evaluation Metrics
More Qualitative Analysis
A.1 IMPLEMENTATION DETAILS OF MULCON
Projector Following other contrastive learning methods (Chen et al. 2020 Khosla et al. 2020), the projector Proj(-) has two linear layers with ReLU(--) activation.
Pretraining with BCE We use Adam optimizer (Kingma & Ba 2015 and L-cycle policy with the maximum learning rate is 2c-4. The training batch size is set to 64, and the dimension of the label-level feature, i..., D, is 1024.
Step 1:
Step 2: Finetuning with LLCL We use Stochastic Gradient Descent (SGD) optimizer with mo- mentum 0.9 and weight decay is le-4. The learning rate is initialized as 0.01 and then reduced by a factor of 10 for every 20 epochs. The batch size for this step is 32. The temperature T is 0.2, and y is 0.1. The number of head in multi-head attention is empirically set to 4, which gives us the best results. Following [Ben- Barn-Barcch et al. (2020), we also employ Cutout (DeVries & Taylor) 2017) with factor 0.5 reguralization. During training. exponential moving average (EMA) applied to model's parameters with decay factor 0.9997.
Augmentations in Step 2 It has been known that augmentations are important for contrastive learn- ing. Following previous methods (Chen et al. 2020 Khosla et al. 2020). we have augmentation module transform each training image before feeding them to the network. Specifically, for the augmentation, we first resize the input image to 448 x 448, and then apply random horizontal flip, and a Random Augmentation module (Cubuk et al. 2020). Note that for the second step training with batch size 32, the actual batch size becomes 64 after data augmentation.
Note that we apply contrastive loss on label-level embeddings, not on images. For a batch size of 32 images, average have 94 label-level embeddings (with the active labels for each image). After augmentation,te the batch size of the label-level embeddings is 188 already, which is the maximum batch size that we future.
with computational resource. We will consider larger batch size in
A.2 EVALUATION METRICS
Beyond mean average precision (mAP), the standard metrics reported in the experimental section are: overall precision (OP), recall (OR), F1-measure (OF1) and per-category precision (CP), recall (CR), Fl-measure (CF1). These metrics are computed as follows:
OP T, Σ. TP, +
OR Σ.T. Σ TP, + FN; TR. 1 Σ TP, FN; C
FP, TP,
1 Σ TP; +P.
CP = C
CR
X OP X OR
CF1 CP CR
OF1 OP + OR
CP+CR
where TP; is true positive of class i, F.P: is false positive of class i, FN, is false negative of class é.
A.3 MORE RESULTS ON VOC AND VG500
Experiments with VOC 2007.
We compare our method with RDAL(Wang et al. 2017). RARL(Chen et al. 201 20188),
MCAR et al.] 2020), SSGRL,(Chen et al. 2019a) and ASI. (Ben-Baruch al. 2020)
14

---

Under review as a conference paper at ICLR 2022
al. 2015) dataset. Following other works, also use pretrained back- bone on COCO and fine-tune on VOC 2007. Overall, our method has a comparable results with other methods. We also try use TResnet and ASL loss combine with our Contrastive framework, but we do not observe any improvement. Figure
VOC2007
Bike
KDXL
97.4
96.3
96.2
9X4
87.7
31.
98.6
89.2
91.9
$1.6
92.0
MCAA
9930
3SCRL
98.4
9800
420
97.0
9900
90.
95.0
893
95.
993
990
955
Table 4: Results on VOCO7, Best results are highlighted in boldface.
Experiments with VG 500.
We compare the performance on VG 500 Krishna et al. 2017) with other methods such as R101 (He et al. 2016), R101-SRN(Zhu al., (017). SSGRL(Chen et al. 2019a) and C-Trans(Lanchantin
et al. 2021 in Table Our method also achieve a better results compare to other methods.
Method
mAP
RIOI
30.9
R101-SRN
33.5
SSGRL
36.6
C-Trans
38.4
MulCon
38.5
Table 5: Results on VG 500.
A.4 CLASS EMBEDDING RETRIEVAL
We compute the precision @k (P k) and recall @k (R@k) of class embedding retrieval between our method and Q2L. (Liu et al. 2021a) in Table We use the first 1000 images in MS-COCO for this experiment. We extract all of embeddings and d is the embedding's 's dimension. Then for an embedding 2;, we construct groundtruth label vector yi such that Vij if x; and æj has the same label, 1/j = 0 otherwise.
class embedding to create the set E Rxxd N is the total number
Then we have the ground truth set Y RNNN Given X, we can compute the pair-wise cosine similarity C RNXN Ciz cossim(r,, 2y)) Given C and Y, the P@k and R @k is computed as follows:
TPak
P@k
TPåK + FP@k
TPOK
R@k min(k,TPQk + FNOK)
where TPOK, TP@K,PPO. FPOk, FN@k are true positive, false positive and false negative at top predic- tions.
Method
P@3
P@5
P@ 10
P@ 15
P@20
R@3
R@5
R@ 10
R@15 15
R@20
MulCon
87.6
84.7
80.6
76.9
73.1
87.7
84.7
80.7
77.4
74.8
Q2L
77.7
73.7
69.5
66.3
62.6
77.8
73.8
69.6
66.7
64.1
Table 6: Quanntitative results of class embedding retrieval subset of MS-COCO.
15

---

Under review as a conference paper at ICLR 2022
Query
Closest labels
person
handbag
cellphone
chair
fork
oven
knife
sink
donut
hotdog
pizza
sandwich
Table 7: Label correlations on COCO.
Query
Results
bicycle, person, skateboard
bicycle, chair, person, skateboard
person, skateboard
bottle, person, skateboard
person, skateboard
bottle, bowl, person, sink, oven, dining table, fork, spoon
bottle, person, microwave, oven, dining table
bottle, bawl, sink, oven
bottle, book, oven, dining table
bowl, person, fork, spoon, pizza
Figure 5: Top-4 related images retrieved given query image and multiple labels on COCO dataset.
A.5 MORE QUALITATIVE ANALYSIS
Multi-label Image Retrieval To further demonstrate the intuitive meanings of the learned label- level embeddings in the multi-label setting, we provide an additional retrieval experiment where instead of using single query label, we select multiple ones and concatenate their label-level em- beddings as the query vector. That is to say, given image, we can retrieve the images with multi-label labels of interest. 5
the multi-label retrieval results.
Visualisation of Attention Maps the active labels of image produced by the multi-head attention used in the label-level embedding network. As illustrated in the top row of Figure 6 the attention maps can precisely highlight the regions of the image in terms of each of its ground-truth labels. the bottom row of Figure B we show the attention maps of the heads of one label "person". It can be observed that the heads tend to capture the multiple instances of "person" " in the image. Figure 7 presents more attention maps visualization. Here visualize the mean attention map computed from 4 heads.
We now show a visualization for the attention maps in terms of
Analysis on the Global Label Embeddings row U is the embedding vector specific label, encoding the semantic information of the label. Although randomly initialised, the embedding vectors are learned by minimising the loss function via gradient backpropogation. In Table 7 we have computed the pairwise Euclidean distances between label embeddings and the closest 3 labels retrieved with minimum distances. It can be seen that the label correlations are quite meaningful.
16

---

Under review as a conference paper at ICLR 2022
input
COW
umbrella
person
person 1
person
person
person 4
Figure 6: Visualization of attention maps. The top includes the input image and the selected attention map for the ground-truth labels. The bottom row includes the multi-heade attention maps for class "person".
17

---

Under review as a conference paper at ICLR 2022
car
person
suitcase
banara
book
remote
carrot
cup
cining table
fork
spo
car
potted plant
toilet
car
parking meter
apole
bowl
cup
dining caole
fork
vose
Figure 7: Visualization of attention maps. In each row, the left most image is input image, and the remains are the mean attention maps (from 4 head) for its ground truth classes.
18


Reviews:
[{"review_id": "ln6bOD_uPM", "reviewer": "Reviewer_ToSu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Multi-label classification is one of the fundamental problems in computer vision and has a large amount of studies in the literature. This paper  addressed the problem by contrastive learning, which has not been explored for multi-label classification. The proposed solution that learns label-level embeddings is intuitive and reasonable, and the results are promising.\n\nThe paper is well organized and written. It also provides sufficient amount of background to understand the proposed method.\n\nHowever, I have two questions for further clarification:\n\n1. The proposed label-level embedding network is considered one of the main contributions in this work. This network receives an image and returns a set of label-level embeddings. The architecture of this network involves attention blocks, similar to visual transformer and its variants. In fact, Query2Label [1] adopts transformer (taking advantage of cross-attention modules as well) and achieves a similar performance level. What is exactly the difference between the proposed design and those works? \n\n[1] Liu et al., Query2Label: A Simple Transformer Way to Multi-Label Classification, arXiv:2107.10834.\n\n2. The proposed method achieved state-of-the-art performance on both MS COCO and NUS-WIDE datasets; however, these two datasets have different characteristics. For example, COCO contains some \"small objects\" (e.g., spoon, cell phone, etc.) and NUS-WIDE has a few \"concepts\" (e.g., map). Image embeddings learned by contrastive embedding and attention mechanism may help recognition of small objects, but why is it helpful for recognizing concepts that are global, and perhaps somewhat vague? Does the performance gain against existing methods come mainly from the attention module?\n  ", "summary_of_the_paper": "This paper presents a contrastive learning based method for multi-label classification. In particular, authors propose to learn \"label-level embeddings\" for an image, thereby the multi-label classification problem can be transformed into a single-label one where contrastive learning can be naturally adapted. The proposed label-level embedding network involves self-attention and multi-head attention blocks that learn label-specific embeddings. Results were demonstrated by the experiments on two benchmark datasets (MS COCO and NUS-WIDE).\n\n", "summary_of_the_review": "This paper presents new state-of-the-art results on MS COCO and NUS-WIDE for multi-label classification. It leverages contrastive learning to enhance distinctiveness for great performance in multi-label image classification. It is also well organized and written. \n\nThe proposed label-level embedding network is conceptually similar to visual transformer or other attention variants. I am not sure about the novelty at current point. The results were demonstrated on MS COCO and NUS-WIDE only. I am not sure if it is scalable to larger datasets (like open images dataset).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ekf7IopMtqK", "reviewer": "Reviewer_rsBB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is written very well and easy to follow.\n2. The proposed approach is technically sound and achieves the new state-of-the-art.\n3. Interesting qualitative analysis for the properties of learned representations.\n\nWeaknesses:\n1. It is an incremental work and the main focus, i.e. the adaptation of contrastive learning for multi-label classification, is a trivial generalization of existing contrastive learning method.\n2. There are some exist works which introduce the idea of contrastive learning into multi-label classification. Please talk about the relationships between these existing works and the approach in this paper. Experimental comparisons with some of these works would be desirable.\n3. According to the results reported in the ablation study (LLEN+BEN+LLCL vs. LLEN+BEN), it seems that the contrastive learning term may not be a good regularization for multi-label learning. While the proposed MulCon trained with a two-step policy has a much better performance, I am really confused about whether the performance gain comes from the contrastive learning term or the improved training procedure, since many additional training tricks are utilized. More analyses are suggested to uncover the true sources of the performance gain.\n\n\nThere are some related works which introduce the idea of contrastive learning into multi-label classification:\n\n[1] Li, C., Liu, C., Duan, L., Gao, P., & Zheng, K. Reconstruction regularized deep metric learning for multi-label image classification. TNNLS, 2019.\n\n[2] Chen, C., Wang, H., Liu, W., Zhao, X., Hu, T., & Chen, G. Two-stage label embedding via neural factorization machine for multi-label classification. AAAI, 2019.\n\n[3] Liu, W., & Tsang, I. W. Large margin metric learning for multi-label prediction. AAAI, 2015.\n", "summary_of_the_paper": "In this paper, the authors introduce the contrastive learning into multi-label classification. Specifically, the multi-label classification problem is first decomposed into a series of binary classification problems with label-level features extracted by the attention mechanism. Then, label-wise contrastive learning is performed on these binary classification problems respectively. Comparative experiment shows the proposed approach achieves the new state-of-the-art performance in multi-label image classification.\n\nHowever, the proposed adaption of contrastive learning for multi-label classification is a trivial generalization of existing contrastive learning for single-label classification, since contrastive learning is simply performed on the binary classification problem of each class label. And it is not a new idea to introduce the contrastive learning into multi-label classification. Besides, extracting label-level features via attention mechanism is a well-established technique in many existing works. Thus, if I had not missed something, the contribution of this paper is very limited.\n", "summary_of_the_review": "Considering the limited contributions and unclear sources of the performance gain, I think this paper is marginally below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PBmrPommGnF", "reviewer": "Reviewer_2oF2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Built upon the transformer encoder-decoder structure, authors manage to obtain the label-level embedding for further contrastive learning, of which the label-level embedding among one mini-batch construct a natural training set for contrastive constrains;\n- The in-depth investigation have been made to analyze why and how dose contrastive loss help, including a vivid t-sne illustration;\n\nWeaknesses:\nThe idea of conducting the contrastive learning within the mini-batch among the label-level embedding is interesting. However, the major concern is the overall novelty of proposed method.\n- The attention-based label-level embedding is a straightforward application of transformer-like structure, which have been shown in the recent multi-label classification models;\n- Apart from the scenario of training with network, the bag-of-words kind training policy is also quite common in multi-label classification; holding the contrastive learning as the main contribution of the paper is quite limited;\n- The experimental results are still less advanced compared to the recent sota models, such as query2label, TResNet, MlTr etc., which can be referred at paper with codes (https://paperswithcode.com/sota/multi-label-classification-on-ms-coco). It is known that the multi-label performances are sensitive to input size, backbone, and training strategies etc., authors still need to expand their experiments to fully validate the effectiveness of the contrastive learning;\n- Also only two datasets are investigated, what about VOC07/12, Visual Genome? As the proposed framework is highly related to the label-level embedding, it should be verified on different label-set size to investigate its limitations.", "summary_of_the_paper": "In this paper, authors propose a framework for utilizing the contrastive learning to improve the feature distinctiveness for multi-label classification. The main purpose is to address that the direct application of contrastive learning (similar to the single-label classification case) is unhelpful for improving the multi-label performance.", "summary_of_the_review": "Overall, the idea of utilizing the contrastive loss during the training of multi-label image classification network is appealing, however the limited novelty and experiments are the main concerns. Authors should carefully address above concerns (weaknesses) during their feedback, the final recommendation will be made upon the feedback.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NPnHeuRrA6G", "reviewer": "Reviewer_3XCq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The organization of this paper is good, and it's easy to read.\n+ The proposed method seems to be an intuitive adaptation of the contrastive learning formula to multi-label classification.\n+ Experimental evaluation proves the attractiveness of the proposed method. ResNet-101-based MulCon beats other baselines on most of the presented measures. I believe that using only ResNet-101 models is a fair way of conducting such a comparison and four datasets are enough.\n+ The appendix seems to consist of enough details to reproduce these results.\n+ Ablation study (including results from discussion), qualitative analysis and pretty t-SNE visualization.\n\nWeaknesses and doubts:\n- The scope of the contribution is a bit limited, it seems to me that the main novelty in the area of multi-label classification is proposed Label-level Contrastive Loss. The rest is pretty much a simple arrangement of known blocks and techniques.\n- The motivation for the usage of contrastive learning is shallow.\nThe major performance boost comes from the attention block, which, as I understand, has almost the same number of parameters as ResNet-101 used as base-encoder. I lack a comment on a number of parameters of the proposed method and baselines in the paper.\n- There is no information about the maximum numbers of epochs for each MulCon training step. I suspect that two-step training of MulCon may take more time than other benchmarks since finetuning with LLCL that use SGD may require a large number of epochs + MulCon has a lot of additional parameters.\n- Is augmentation described in the appendix is also used with R101 + BCE + SCL, and LLEN + BCE + LLCL variants in the ablation study presented in Table 3? It is not clear from the paper. One of the authors' responses suggests that it was not. It is also not clear which other techniques from the finetune phase are necessary for this framework to work.\n\nNits:\n- It's not difficult to guess (based on results), but for clarity, it would be nice to add resolution used for ablation study.\n- Different ways of writing ResNet-101/Resnet101, it would be nice to unify the form.", "summary_of_the_paper": "The authors introduce a new framework for multi-label classification that leverages supervise contrastive learning. The framework adds an attention mechanism on top of the image encoder. Thanks to that, per-label features can be obtained to perform label-wise contrastive learning. The proposed method (MulCon) is trained in two steps. The first step performs pre-training by optimizing only binary-cross entropy (BCE) loss. In the second step, the model is finetuned using a combination of BCE and proposed by the authors Label-Lebel Contrastive Loss. The experimental study on four datasets shows that the proposed approach achieves the new state-of-the-art results among other methods based on ResNet-101, on most from many performance measures.", "summary_of_the_review": "The review was written after the rebuttal phase, and authors' responses to other reviewers' comments were taken into account. I find this paper to be a sold work, a simple idea with good empirical evaluation. In my opinion, the authors also did a good job answering the reviewers' comments. However, I still have some other doubts, so I believe that right now, this paper is marginally above the acceptance threshold - I recommend it for acceptance, but I will not be upset if the paper is rejected.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an interesting paper trying to answer how contrastive learning can be used in multi-label classification. The reviewers however had raised several doubts about motivations, novelty, or the impact of contrastive module on final results. For many of them the authors had delivered satisfying responses, but after a long discussion, we decided that the paper needs revision to improve in these aspects. For example, the authors should make it clear whether the image retrieval application from Section 4.4 is the main motivation of the method. If so, what are the competitive approaches to solve such problem? How to measure the performance of such methods? Answers for the above questions are crucial to find the right motivations for the contrastive module used by the authors. 

We hope that the authors will follow the recommendations and resubmit the paper to another top conference.
Decision: Reject