C+1 Loss: Learn to Classify C Classes of Interest and the Background Class Differentially

1 INTRODUCTION . In machine learning , Softmax is one of the most widely used classifiers for classification , especially in CV tasks . During training , the Softmax classifier is often supervised by cross-entropy loss which treats each class without difference . However , there is a type of problem present all around the classification area , for which it is unreasonable to treat each class the same . We called this type of problem as C+1 classification . C+1 classification means classifying samples from C semantically deterministic classes and the ( C + 1 ) th semantically underterministic class . We can semantically uniquely define each of the C classes by its inherent characteristics , so we can say they are semantically deterministic . Generally , we are interested in the C classes , so we call them C CoIs and one of them as a class of interest ( CoI ) hereafter . The ( C +1 ) th class includes all the other stuff beyond C classes . Because it doesn ’ t have uniform inherent characteristics and can ’ t be described uniquely in semantics . That is why we say it is semantically undeterministic . In most cases , we regard things belonging to the ( C + 1 ) th class as background , and we will call it background class hereinafter . Based on the above description , we consider that it is reasonable to treat the C CoIs and the background class differentially during training . On one hand , for the C CoIs , it ’ s reasonable to drive a C + 1 classifier to learn a compact and independent representation space for each of them because of their inherent characteristics . Then the C + 1 classifier can embed samples from each CoI into its own representation space . On the other hand , it ’ s also reasonable to drive the C + 1 classifier to map any samples from the background class into somewhere in feature space far away from all representation spaces of the C classes , considering that samples from background class doesn ’ t have any inherent characteristics different from C CoIs . For example in figure 1 , if we are interested in cat , we can recognize a cat from an image at the first glance by the knowledge in the subconscious which uniquely defines cat . On the contrary , we can also recognize a not-cat instantly because it doesn ’ t have any inherent characteristics of cat . According to the inherent characteristics of C+1 classification , we figure out that a C+1 classifier will be preferable if it has a compact and independent representation space for each of the C CoIs in addition to its separability for all classes . This can guarantee it behaves well while encountering a sample which have different styles from the samples of corresponding CoI in the training set . Furthermore , it will be much better if there is large enough margin between the background class and the C CoIs . This will make the classifier more robust and generalizable while encountering a sample from any new classes belonging to the super background class , especially those that never appear in the training set . Above on these , we conclude three properties which a good C + 1 classifier should have—separability , compactness and background margin . Based on the three properties , we define a uniform general C + 1 loss which includes three parts corresponding to the three properties , driving the C+1 classifier to satisfy those properties . At last , considering semantic segmentation and object detection are two of the most typical C+1 classification problems , and are widely used in AI systems , we instantialize a C + 1 loss and practice it in semantic segmentation and object detection tasks , proving its superiority over the traditional cross-entropy loss . Specifically , in semantic segmentation , Softmax is widely used to classify each pixel of an image to one label of a predefined class set . Typically the predefined class set includes C CoIs and a background class . For instance , PASCAL VOC 2012 segmentation Everingham et al . ( 2010 ) contains 20 CoIs and a background class . The background class contains all other stuff . In object detection , Softmax is usually used to classify proposal boxes of an image to one of a predefined class set . For example , MS COCO Lin et al . ( 2014 ) contains 80 CoIs . A detector trained on it should classify all proposal boxes as one of the 80 CoIs or as background class . In these two typical tasks , the C + 1 classifiers need to recognize all the CoIs , and classify a variety of other things as background . It ’ s difficult for cross-entropy loss to drive the classifiers to learn well , which treats all samples without difference during training . This paper contains three contributions summarized as follows : 1.We define the C + 1 classification problem present all around the classification area . 2.We propose three properties that a good C + 1 classifier should have , and define a uniform C + 1 loss , which includes three parts driving the classifier to satisfy these properties . 3.We instantialize a C + 1 loss consisting of three terms , and practice it on semantic segmentation and object detection tasks , proving its superiority over the popular cross-entropy loss . 2 RELATED WORK . 2.1 SOFTMAX . Softmax is one of the most widely used classifier for a variety of pattern recognition tasks . Nowadays there are a plenty of variants for Softmax , such as L2-Softmax Ranjan et al . ( 2017 ) , Largemargin Softmax Liu et al . ( 2017 ) , Angular Softmax Liu et al . ( 2016 ) , Normface Wang et al . ( 2017 ) , AM-Softmax Wang et al . ( 2018a ) , CosFace Wang et al . ( 2018b ) and ArcFace Deng et al . ( 2019 ) . Large-margin Softmax is the first attempt to add parameter m into the original Softmax to control the margin and the larger m is , the larger the decision margin between the classes . Angular Softmax also known as SphereFace is an improvement to Large-margin Softmax with additional constraints on W and b , introducing the hypersphere manifold which makes the features suitable for open-set FR problem . L2-Softmax and Normface share similar ideas where L2-Softmax normalizes only the features and Normface normalizes both classifier weights and features and applys a scale parameter after that . The normaling and scaling steps push the learning progress focusing on optimizing angles among the classes , making the features not only separable but also discriminable . AM-Softmax ( additive margin Softmax loss ) and CosFace ’ s works are inspired from SphereFace by moving the parameter m that controls the margin from angular space to cosine space by addition . This also makes the implementation easier . ArcFace ( additive angular margin loss ) moves the parameter m from scaling to addition to expand the optimization boundary . To sum up , thse variants of Softmax improve Softmax from these aspects : normalization of weights or features , margin in angular space or cosine space , setting of margin m. They use the parameter m in different ways resulting in different decision boundary . However , they treat all classes equally during training as Softmax . And we propose to give some special care for the background class . So Softmax and its variants are not the best choice for C + 1 classification problem . 2.2 METRIC LEARNING . Metric learning aims to maximize inter-class variation and minimize the intra-class variations . Constrastive loss Chopra et al . ( 2005 ) ; Hadsell et al . ( 2006 ) ; Sun et al . ( 2014 ) drives the distances between positive pairs close to 0 , and the distances between negative pairs to fall within an absolute range . Triplet loss and its variants Weinberger & Saul ( 2009 ) ; Hoffer & Ailon ( 2015 ) ; Wang et al . ( 2014 ) ; Schroff et al . ( 2015 ) ; Ding et al . ( 2015 ) ; Cheng et al . ( 2016 ) ; Oh Song et al . ( 2016 ) ; Sohn ( 2016 ) drive the relative distances between positive pairs and negative pairs to fall lower than a preset threshold . Center loss Wen et al . ( 2016 ) drives model to learn a center for features of each class and penalizes the distances between features and their corresponding class center . Metric learning also treats each class equally , just from a metric perspective . They could not take care of the particularity of background class and the inherent characteristics of every semantic class . In addition , it ’ s not appropriate to learn a center for background class because it has no deterministic and unique definition . 2.3 OPEN-SET RECOGNITION . Open-set recognition Bendale & Boult ( 2016 ) ; Ge et al . ( 2017 ) ; Shu et al . ( 2017 ) ; Neal et al . ( 2018 ) ; Liu et al . ( 2019 ) ; Yoshihashi et al . ( 2019 ) ; Oza & Patel ( 2019 ) ; Chen et al . ( 2019a ) ; Qian et al . ( 2019 ) ; Yu & Tao ( 2019 ) ; Sun et al . ( 2020 ) is the one closest to our proposed C + 1 classification , wherein there are no training samples for the background class . However , the classifier needs to detect new classes while inferring . The C + 1 classification deals with another type of problem , wherein there are both training samples for C CoIs and the background class . In practice , it ’ s impossible to contain all other semantic classes except the CoIs into the background class . Many samples of new semantic classes are likely yo be encountered and should be identified as background class while inferring . So we could call the C + 1 classification as semi-open-set recognition . 3 METHOD . 3.1 PROBLEM DEFINITION . Firstly , we define some terminologies as follows . Semantic Class : a set of samples which could be described uniquely and deterministically . CoIs : a set of semantic classes that we are interested in and need to be recognized . Background Class : all other classes beyond C CoIs , including those semantic classes we are not interested and all other stuff . C +1 Classification Problem : categorize each item of a set into one of the C +1 classes . Therein , C+1 classes compriseC CoIs and the ( C+1 ) th class of not interested , also called background class . The C CoIs are deterministic and unique , but the background class is underterministic and includes all stuff beyond those belonging to the C CoIs . The training set comprises many samples from each of the C CoIs and diverse samples from the background class . Being semantically deterministic means we can give a deterministic and unique definition for each of the C CoIs according to its inherent characteristics . Being underministic means the background class has no deterministic and unique definition , because it lacks uniform inherent characteristics . 3.2 C + 1 LOSS . For each of the C CoIs , the C +1 classifier should extract the inherent characteristics , and differentiate the background class from it . In order to achieve this , we consider that a C+1 classifier should satisfy the following three key properties . 1.Separability : It should be able to separate all classes . 2.Compactness : The representation space of C CoIs should be compact enough . 3.Background margin : There should be large enough space which we call background margin between the background class and each of the C CoIs . The property one guarantees that a C+1 classifier has basic categorization ability for those samples that have similar distribution with the training set . The property two makes sure that the classifier has good generalizability for those samples belonging to the C CoIs but with different distribution . And the property three gives the classifier good robustness and generalizability for samples of the background class , especially those whose styles are never present at the training set . In order to drive a C+1 classifier to learn to satisfy the three key properties , we argue that the C+1 loss should include at least three parts : Lseparability , Lcompact and Lbackground , corresponding to the three properties respectively . LC+1 = Lseparability + αLcompact + βLbackground ( 1 ) Herein , Lseparability drives the classifier learn to discriminate all classes without difference , Lcompact drives the classifier to try to grasp the inherent characteristics and to learn a compact representation space for each of the C CoIs , and Lbackground drives the classifier to differentiate the background class from the C CoIs well . In practice , we could adopt cross-entropy loss as an instantialized instance for Lseparability and center loss Wen et al . ( 2016 ) for Lcompact . As for Lbackground , we design a novel loss according to the property three as follows . Lbackground = sign ( yi = C + 1 ) 1 N ∑ i 1 C ∑ k=1 |mb − d ( f ( xi ) , ck ) |+ ( 2 ) Herein , sign ( expression ) = 1 only when the expression is true , otherwise 0 . N is the number of background class sample . mb is the background margin between background class and each of the C CoIs . xi is a sample , and yi is its class label index . ck is the center of kth CoI . f ( x ) is a mapping function for extracting feature of sample x. f ( x ) followed by softmax composes the C+1 classifier . Without losing generality , f ( x ) is a deep neural network . As centers for center loss and the instantialized Lbackground , we define three types of center representation . 1.Represent each center of C CoIs as a learnable weight vector . 2.Represent each center of C CoIs as a moving average of sample features of the corresponding class . 3.Represent each center of C CoIs as the classifier weight vector of each class from the logit FC layer . Specifically the third center representation means that we share the center vector parameters with the softmax layer of the C CoIs , and ignore the weight vector of background . We define a center for every CoI , but not the background class . For every CoI , there are a bunch of inherent characteristics which can uniquely and deterministically define it semantically , however there are no unique and deterministic inherent characteristics for the background class . However , we can calculate the loss term Lcompact based on the distance between every sample of each CoI and its center . Furthermore , we can calculate the loss term Lbackground based on the distance between every sample of background class and the center of each CoI .

Summary: The paper addresses the problem of semantic segmentation/classification, in situations where one has C+1 classes at hand, including  C deterministic classes and a background class. The authors identifies the weakness of the classical softmax classification layer in this context, and propose a novel loss, that can better account for the particularity of the non-deterministic background class. Experiments are performed on VOC-2012, Pascal context, LIP and COCO2017. Quantitative results, ablation study and comparison with related work are reported.