Contrastively Enforcing Distinctiveness  for Multi-Label Classification

1 INTRODUCTION . Multi-label image classification is a fundamental and practical computer vision task , where the goal is to predict a set of labels ( e.g. , objects or attributes ) associated with an input image . It is an essential component in many applications such as recommendation systems ( Jain et al. , 2016 ; Yang et al. , 2015 ) , medical image diagnosis ( Ge et al. , 2018b ) , and human attribute recognition ( Li et al. , 2016b ) . Compared to single-label cases , multi-label classification is usually more complex and challenging . Recently , contrastive learning ( CL ) ( Chen et al. , 2020 ; He et al. , 2020 ; Li et al. , 2020 ; Caron et al. , 2020 ; Bachman et al. , 2019 ) has been shown as an effective pretext approach to learn latent representations an unsupervised way , which can be further used for supervised tasks . In general , CL aims to pull together an anchor and a similar ( or positive ) sample in embedding space and push apart the anchor from many dissimilar ( or negative ) samples . Therefore , the choice of the positive and negative samples of an anchor is a key to achieving good performance with CL . In self-supervised CL ( Chen et al. , 2020 ) , the positive sample is defined as those augmented from the same image with the anchor , while the negative samples are all the other images in the minibatch . More recently , supervised CL ( Khosla et al. , 2020 ; Sun et al. , 2021 ; Yuan et al. , 2021 ; Huynh , 2021 ) has been proposed , where all the images with the same label as the anchor are considered as the positive samples and vice versa for the negative ones . Supervised CL has shown improvements in single-label image classifications than the self-supervised counterpart . With the above successful examples , CL has drawn significant research attention and has been applied in other tasks including image segmentation ( Wang et al. , 2021 ) , adversarial training ( Kim et al. , 2020 ) , and text to image learning ( Radford et al. , 2021 ) . Given the appealing properties and promising results of CL in single-label classification , it is natural to adapt it into multi-label cases to boost performance . However , this adaptation is non-trivial . In single-label cases , an image usually contains one salient object , thus , the label of the object can also be viewed as the unique label of the image . Therefore , it is reasonable to use one image-level representation of an image and to push the representation of the anchor close to its positive samples ( e.g , augmentations of the anchor or images with the same label as the anchor ) , as done in selfsupervised and supervised CL . However , with a single image-level representation for an image , it is hard to define the positive or negative samples for an anchor image by its multiple labels in the multi-label classification . For example , it would not be reasonable to assume that the image-level representations of images containing apples must always be close to each other , as apple is just one of many objects in those images and an apple may only take a small area of an image . As a result , this setting hinders the application of existing CL methods to multi-label classification . To bridge this gap , we in this paper propose a novel end-to-end framework for multi-label image classification that leverages the contrastive learning principle , termed MulCon or Multi-label Classification with Contrastive Loss . Instead of using image-level representations as in previous CL methods , we introduce a new module that learns multiple label-level representations of an image , which are generated with the attention from the globally class-specific embeddings to the image features learned by a convolutional neural network ( CNN ) . Each label-level embedding of an image corresponds to the image ’ s representation in the context of a specific label . With these definitions , the supervised CL loss can be applied . Specifically , if we look at one specific label and view a labellevel embedding of an image as the anchor in our proposed CL framework , it is straightforward to define the positive samples of an anchor , which are the label-level embeddings of the other images in the minibatch with the same label and vice versa for the negative samples . For example , instead of sharing an image-level representation with other objects , the apple object of an image has its own embedding and the embeddings of the apples of all the images in a minibatch are pushed close to each other . Therefore , our framework is intuitive in the multi-label setting . In this way , the CL loss can enforce the coherence and consistency of the label-level representations of images , which further provides more discriminative power of the prediction procedure based on these representations . The main contributions can be summarized as follows : 1 ) Contrastive learning has been shown successful in many single-label classification problems , however , it is non-trivial to apply it in multilabel classification . We propose an intuitive and conceptually simple framework that encompasses contrastive learning . 2 ) Along with the proposed framework , we also introduce a practical training scheme of contrastive learning for multi-label classification . 3 ) We conduct extensive experiments on large-scale benchmark datasets , showing that the proposed framework achieves the state-of-theart performance in multi-label image classification . 2 PROPOSED METHOD . In this section , we first discuss about the background knowledge and relevant notations and then elaborate on the details of the proposed framework , MulCon . 2.1 BACKGROUND AND NOTATIONS . Multi-label Classification Following the standard setting of multi-label image classification , we denote a minibatch of input images by X ∈ RN×W×H×3 , where N is the batch size , H and W are the height and width of the images . Each image xi ∈ X is associated with multiple labels selected from a set of L labels in total , which are denoted by a multi-hot binary vector yi ∈ { 0 , 1 } L. For an active label j of xi , yij = 1 and vice versa . Our task is to build an end-to-end model that takes xi to predict its labels yi . Attention The Attention mechanism ( Luong et al. , 2015 ; Xu et al. , 2015 ) has been widely used in various areas of computer vision and natural language processing , which enhances the important parts of the data of interest and fades out the rest . Assume that nq query vectors of size dq denoted as Q ∈ Rnq×dq , and nv key-value pairs denoted as K ∈ Rnv×dq , V ∈ Rnv×dv . The attention function maps the query vectors Q to outputs using the key-value pairs as follows : Att ( Q , K , V ) = ω ( QKT ) V ( 1 ) where the dot product ( QKT ) ∈ Rnq×nv and ω ( · ) is softmax function . The dot product returns the similarity of each query and key value . The output ω ( QKT ) V ∈ Rnq×dv is the weighted sum over V , where larger weight corresponds to larger similarity between query and key . A powerful extension to the above ( single- ) attention mechanism is the multi-head attention introduced in ( Vaswani et al. , 2017 ) , which allows the model to jointly attend to information from different representation subspaces at different positions . Instead of computing a single attention function , this method first projects Q , K , V onto h different vectors , respectively . An attention function Att ( · ) is applied individually to these h projections . The output is a linear transformation of the concatenation of all attention outputs : MultiAtt ( Q , K , V ) = concat ( O1 , O2 , · · · , Oh ) W o , Oi′ = Att ( QW q i′ , KW k i′ , V W v i′ ) for i ′ ∈ 1 , · · · , h , ( 2 ) whereW o , W qi′ , W k i′ , W v i′ are learnable parameters of some linear layers . QW q i′ ∈ R nq×dhq , KW ki′ ∈ Rnv×d h q , VW vi′ ∈ Rnv×d h v are vectors projected from Q , K , V respectively . dhq = dq/h and d h v = dv/h . Following the architecture of the transformer ( Vaswani et al. , 2017 ; Lee et al. , 2019 ) , we define the following multi-head attention block : MultiAttBlock ( Q , K , V ) = LayerNorm ( Q′ +Q′W q ′ ) , ( 3 ) Q′ = LayerNorm ( concat ( QW q1 , · · · , QW q h ) + MultiAtt ( Q , K , V ) ) where W q ′ ∈ Rdq×dq is a learnable linear layer . Base on the multi-head attention block , we further define a self-attention block as follows : SA ( X ) = MultiAttBlock ( X , X , X ) ( 4 ) Contrastive Learning Contrastive learning ( CL ) has been an increasingly popular and effective representation learning approach ( Chen et al. , 2020 ; He et al. , 2020 ; Khosla et al. , 2020 ) . As the first proposed CL approach , self-supervised CL ( Chen et al. , 2020 ) is proposed to learn presentations in an unsupervised manner . Specifically , a mini-batch is constructed from the original input images and their augmented versions . Given a minibatch of 2N instances I = { 1 ... 2N } and an anchor instance i ∈ I , the augmented version of i , denoted as ia ∈ I is considered the positive sample , and the other 2 ( N − 1 ) instances within the mini-batch are considered negative examples . The loss function of self-supervised CL is defined as follows : Lself = − ∑ i∈I log exp ( zi · zia/τ ) ∑ za∈A ( i ) exp ( zi · za/τ ) ( 5 ) where zi is the image embedding , zi · zia denotes the inner dot product between two embeddings , τ ∈ R+ is a scalar temperature parameter , and A ( i ) = I\zi . It can be seen that self-supervised CL pushes the embeddings of the samples augmented from the same image close to each other . More recently , the supervised CL ( Khosla et al. , 2020 ) adapts CL into the supervised settings , which utilizes the label information to select positive and negative samples . For supervised CL , the embeddings of the samples with the same labels are pushed close to each other , which achieves better performance in classification tasks . For more details of CL , we refer to Liu et al . ( 2021b ) . To our knowledge , CL has not been applied in solving multi-label classification yet . 2.2 PROPOSED METHOD . Unlike previous CL methods for single-label classification that use image-level representations , we propose to learn multiple label-level representations for each image , which facilitates the application of CL in multi-label classification . We introduce MulCon , which consists of three main neural network modules : the label-level embedding network , the contrastive learning projection network , and the classification networks as shown in Figure 1 . Label-Level Embedding Network As the key module of our framework , the label-level embedding network takes an image xi as input and outputs its label-level representations , denoted by gi ∈ RL×D , where each row of gi corresponds to the embedding of the image under the context of a specific label . Specifically , the label-level embedding network consists of two components : 1 ) The encoder block . We first adopt an encoder network that learns the image-level embedding as the backbone model as the backbone model : ri = Enc ( xi ) ∈ RC×H×W , where C , H , W are the number of channels , height , and width of the output . The backbone model can be implemented with an arbitrary model that learns good image features , e.g. , a convolutional neural network ( CNN ) such as ResNet ( He et al. , 2016 ) or visual transformer ( Dosovitskiy et al. , 2020 ) . We then reshape ri ∈ RWH×C for the next step ’ s processing . 2 ) The label-wise attention block . In this block , we introduce a set of vectors U ∈ RL×C , each row of which is to be learned as the global embedding for a specific label . To generate an image ’ s label-level embeddings gi , we propose a cascade of several self-attention blocks and a multi-head attention block to capture the interactions between ri and U as follows : ri = SA ( ri ) ; gi = MultiAttBlock ( U , ri , ri ) ; gi = SA ( gi ) . ( 6 ) To summarise , given an image xi , we first learn its representation ri ∈ RWH×C by the encoder , each row of which captures the feature of a location in the image . We then pass ri to a self-attention block that captures the interactions between the features of the image . Next , we feed ri to the multihead attention block , where the global label embeddings U is the query and ri is the key and value . As the sizes of the input and output channels of the multi-head attention are C and D , respectively , we have nq = nv = C and dq = nq = D for Eq . 2 . With the multi-head attention block , our model can learn the “ importance ” ( attention weight ) of the image feature to a specific label . For example , if an image contains an apple , the corresponding embedding is expected to be associated with a large attention weight from the apple label . The multi-headed attention can also help a label pay its attention to multiple objects in an image , i.e. , each of the attention heads can generate attention scores for a class-specific embedding over all the image-level embeddings . In this case , if an image consists of multiple apples , each of the apples receives a specific attention score from the apple label . Following up the multi-head attention is a self-attention block that implicitly helps improve label correlation . With the attention from all the labels , we can derive the label-level embeddings gi of the image from its image-label embedding ri . Contrastive Learning Projection Network After obtaining gi ∈ RL×D , we use gij ∈ RD to denote the label-level embedding of the input image i under the context of a specific label j ( j ∈ { 1 , · · · , L } ) . Following Chen et al . ( 2020 ) ; Khosla et al . ( 2020 ) , our framework includes a projection network Proj ( · ) that maps gij to a vector in another embedding space : zij = Proj ( gij ) ∈ Rdz , where the contrastive learning is performed . Classification Network Recall that the label-level embedding gij captures the input image i ’ s feature under the context of the label j . Thus , it can be used to predict whether j is active in i . Accordingly , we introduce a fully connected layer as a classifier f jc to predict the probability of the label j being active . Specifically , for each label j ∈ L the prediction score is sij = σ ( f jc ( gij ) ) ∈ ( 0 , 1 ) . We further denote si ∈ ( 0 , 1 ) L .

Summary: The authors introduce a new framework for multi-label classification that leverages supervise contrastive learning. The framework adds an attention mechanism on top of the image encoder. Thanks to that, per-label features can be obtained to perform label-wise contrastive learning. The proposed method (MulCon) is trained in two steps. The first step performs pre-training by optimizing only binary-cross entropy (BCE) loss. In the second step, the model is finetuned using a combination of BCE and proposed by the authors Label-Lebel Contrastive Loss. The experimental study on four datasets shows that the proposed approach achieves the new state-of-the-art results among other methods based on ResNet-101, on most from many performance measures.