Multi-class classification without multi-class labels

1 INTRODUCTION . One of the most common settings for machine learning is classification , which involves learning a function f to map the input data x to a class label y ∈ { 1 , 2 , .. , C } . The most successful method for learning such a function is deep neural networks , owing its popularity to its capability to approximate a complex nonlinear mapping between high-dimensional data ( e.g . images ) and the classes . Despite the success of deep learning , a neural network demands a large amount of class-specific labels for learning a discriminative model , i.e . P ( y|x ) . This type of labeling can be expensive to collect , requires a-priori knowledge of all classes , and limits the form of supervision required . For example , the classes may be ambiguous or non-expert human annotators may be able to more easily provide information about whether two instances are of the same class or not , rather than identifying the specific class . A final problem is that different methods are necessary depending on what type of data is available , ranging from supervised learning ( known classes ) to cross-task unsupervised learning ( unknown classes in the target domain ) and semi-supervised learning ( mix of labeled and unlabeled with known classes ) . Unsupervised learning with unknown classes is especially difficult to support . To relax these limitations , we propose to reduce the problem of classification to a meta problem that underlies a set of learning problems . Instead of solving the target task directly ( learning a multi-class discriminative model such as a neural network ) , we instead learn a model that does not require explicit class label y but rather a weaker form of information . In the context of classification , the meta problem that we use is a binary decision problem . Note that such a conversion to a different task ( e.g . binary ) is called a problem reduction method ( Allwein et al. , 2000 ) which has had a long history in the literature , especially in ensemble methods and binarization techniques ( Galar et al. , 2011 ) . The most well-known strategies are `` one-vs-all '' ( Anand et al. , 1995 ; Rifkin & Klautau , 2004 ) and `` one-vs-one '' ( Knerr et al. , 1990 ; Hastie & Tibshirani , 1998 ; Wu et al. , 2004 ) . Although they have varied ensembling strategies , all of them share the same task encapsulating scheme , as illustrated in Figure 1a ; specifically the binary classifiers are the sub-modules of a multi-class classifier ( i.e . the multi-class classifier consists of multiple binary classifiers ) . These schemes still require that the class The demo code is available at https : //github.com/GT-RIPL/L2C label y be available to create the inputs for each binary classifier , and therefore these strategies do not relax the labeling requirements mentioned earlier . In this work , we propose a novel strategy to address the above limitations . Our method reverses the task encapsulation order so that a multi-class classifier becomes a sub-module of a binary classifier , as illustrated in Figure 1b . The connection between the two classifiers is elucidated in Section 3 . There are two highlights in Figure 1b . First , class labels yi are not required in the learning stage . Instead , our method uses pairs of data ( xi , xj ) as input and pairwise similarity sij for the supervision . Second , there is only one binary classifier in the scheme and it is present only during the training stage . In other words , the ephemeral binary task assists the learning of a multi-class classifier without being involved in the inference . When using a neural network with softmax outputs for the multi-class classifier , the proposed scheme can learn a discriminative model with only pairwise information . We specifically make the following contributions : 1 ) We analyze the problem setting and show that the loss we can use for this encapsulation can be easily derived , and we present an intuitive probabilistic graphical model interpretation for doing so , 2 ) We evaluate its performance compared to vanilla supervised learning of neural networks which uses multi-class labels , and visualize the loss landscape to better understand the underlying optimization difficulty , and 3 ) We demonstrate support for learning classifiers in more challenging problem domains , e.g . in unsupervised cross-task transfer and semi-supervised learning . We show how our meta classification framework can support all three learning paradigms , and evaluate it against several state-of-the-art methods . The experimental results show that the same meta classification approach is superior or comparable to state of the art across the three problem domains ( supervised learning , unsupervised cross-task learning , and semi-supervised learning ) , demonstrating flexibility to support even unknown types and numbers of classes . 2 RELATED WORK . Supervised learning and problem reduction : Allwein et al . ( 2000 ) presents a unifying framework for multi-class classification by reducing it to multiple binary problems . The concepts for achieving such reduction , one-vs-all and one-vs-one , have been widely adopted and analyzed ( Galar et al. , 2011 ) . The two strategies have been used to create several popular algorithms , such as variants of support vector machine ( Weston & Watkins , 1998 ) , AdaBoost ( Freund & Schapire , 1997 ; Schapire & Singer , 1999 ) , and decision trees ( Fürnkranz , 2003 ) . Despite the long history of reduction , our proposed scheme ( Figure 1b ) has not been explored . Furthermore , the scheme can be deployed easily by replacing the learning objective , which is fully compatible with deep neural networks for classification , a desirable property for broad applicability . Unsupervised cross-task transfer learning : This learning scheme is proposed by Hsu et al . ( 2018 ) . The method transfers the pairwise similarity as the meta knowledge to an unlabeled dataset of different classes . It then uses a constrained clustering algorithm with predicted pairwise constraints ( binarized pairwise similarity ) to discover the unseen classes . This learning scheme shares the same supervision ( pairwise similarity ) as ours , and therefore relates our method to constrained clustering algorithms . One class of such approaches uses the constraints to learn a distance metric , and then applies a generic clustering algorithm such as K-means or hierarchical clustering to obtain the cluster assignments . This includes DML ( Xing et al. , 2003 ) , ITML ( Davis et al. , 2007 ) , SKMS ( Anand et al. , 2014 ) , SKKm ( Anand et al. , 2014 ; Amid et al. , 2016 ) , and SKLR ( Amid et al. , 2016 ) . The second class of methods incorporates the constraints into the cluster assignment objective . Some constrained spectral clustering algorithms , e.g . CSP ( Wang et al. , 2014 ) and COSC ( Rangapuram & Hein , 2012 ) use this strategy . There are also approaches combine both distance metric learning and a clustering objective jointly , such as MPCKMeans ( Bilenko et al. , 2004 ) , CECM ( Antoine et al. , 2012 ) , and Kullback–Leibler divergence based contrastive loss ( KCL ) ( Hsu & Kira , 2016 ; Hsu et al. , 2018 ) . Our new learning objective for Figure 1b can replace the above objectives in the cross-task transfer learning scheme . Semi-supervised learning : Our meta classification strategy can easily plug into a semi-supervised learning scheme . Our comparison focuses on state-of-the-art methods which solely involve adding a consistency regularization ( Laine & Aila , 2017 ; Sajjadi et al. , 2016 ; Miyato et al. , 2018 ; Tarvainen & Valpola , 2017 ) or Pseudo-Labeling ( Lee , 2013 ) for training a neural network . Another line of strategy combines weak supervision , such as similar pairs , and unlabeled data to learn a binary classifier ( Bao et al. , 2018 ) . We present a new method by augmenting Figure 1b with the Pseudo-Labeling strategy . 3 META CLASSIFICATION LEARNING . A natural way to analyze problems with observed and unobserved information is through a probabilistic graphical model . In figure 2 , we show the graphical model for our problem , where classspecific labels Y are latent while pairwise similarities S are observed . Specifically , we denote X = { X1 , .. , Xn } , Y = { Y1 , .. , Yn } , and S = { Sij } 1≤i , j≤n to represent the nodes for samples , class labels , and pairwise similarities , respectively . In the model , we have Yi ∈ { 1 , 2 , .. , C } and Sij ∈ { 0 , 1 } . Then we have P ( Sij = 1|Yi , Yj ) = 1 when Yi = Yj and zero probability otherwise ; similarly , P ( Sij = 0|Yi , Yj ) = 1 when Yi 6= Yj . The output of a discriminative classifier with parameters θ is f ( xi ; θ ) = P ( Yi|xi ; θ ) , where f ( xi ; θ ) outputs a categorical distribution . Now we describe the likelihood that the model explains the observed labeling ( either with class labeling or pairwise labeling ) . L ( θ ; X , Y , S ) = P ( X , Y , S ; θ ) = P ( S|Y ) P ( Y|X ; θ ) P ( X ) ( 1 ) When S is fully observed while Y is unknown , calculating the likelihood requires marginalizing Y by computing ∑ Y P ( S|Y ) P ( Y|X ; θ ) , which is intractable . The pairwise term P ( S|Y ) =∏ i , j P ( Sij |Yi , Yj ) makes all Yi dependent on each other and prohibits efficient factorization . Thus , we approximate the computation by imposing additional independences such that Sij ⊥ S\ { Sij } |Xi , Xj ( see Appendix D for a discussion of these ) . Now we can compute the likelihood with the observed nodes Xi = xi and Sij = sij : L ( θ ; X , S ) ≈ ∑ Y P ( S|Y ) P ( Y|X ; θ ) ( 2 ) ≈ ∏ i , j ( ∑ Yi=Yj 1 [ sij = 1 ] P ( Yi|xi ; θ ) P ( Yj |xj ; θ ) + ∑ Yi 6=Yj 1 [ sij = 0 ] P ( Yi|xi ; θ ) P ( Yj |xj ; θ ) ) . ( 3 ) Equation ( 2 ) omits the P ( X ) since X are observed leaf nodes . It is straightforward to take a negative logarithm on equation 3 and derive a loss function : Lmeta ( θ ) = − ∑ i , j log ( ∑ Yi=Yj 1 [ sij = 1 ] P ( Yi|xi ; θ ) P ( Yj |xj ; θ ) + ∑ Yi 6=Yj 1 [ sij = 0 ] P ( Yi|xi ; θ ) P ( Yj |xj ; θ ) ) ( 4 ) = − ∑ i , j sij log ( f ( xi ; θ ) T f ( xj ; θ ) ) + ( 1− sij ) log ( 1− f ( xi ; θ ) T f ( xj ; θ ) ) . ( 5 ) Then we define the function g by the probability of having the same class label , which is calculated by the inner product between two categorical distributions : g ( xi , xj , f ( · , θ ) ) = f ( xi ; θ ) T f ( xj ; θ ) = ŝij ( 6 ) Here we use ŝij to denote the predicted similarity ( as opposed to ground truth similarity sij ) . By plugging equation 6 into equation 5 , Lmeta has the form of a binary cross-entropy loss : Lmeta = − ∑ i , j sij log ŝij + ( 1− sij ) log ( 1− ŝij ) . ( 7 ) In Figure 1b , the multi-class classifier corresponds to f while the binary classifier corresponds to g. In other words , it is surprisingly simple to wrap a multi-class classifier by a binary classifier as described above . Since there are no learnable parameters in g , the weights optimized with the meta criterion Lmeta are all in the neural network f . To minimize Lmeta , f ( xi ; θ ) and f ( xj ; θ ) must output a sharply peaked distribution with the peak happening only at the same output node when sij = 1 . In the case of sij = 0 , the two distributions must have as little overlap as possible to minimize the loss . In the latter case , the two samples are pushed to be activated at the output nodes of different classes . Both properties of f ’ s output distribution are typical characteristics of a classifier learned with class labels and using multi-class cross-entropy . The properties also illustrate the intuition of why minimizing Lmeta helps f learn outputs similar to a multi-class classifier . Lastly , because of the likelihood nature of Lmeta , we call the learning criterion a Meta Classification Likelihood ( MCL ) in the rest of the paper .

Summary: The work is a special case of density estimation problems in Statistics, with a use of conditional independence assumptions to learn the joint distribution of nodes. While the work appears to be impressive, such ideas have typically been used in Statistics and machine learning very widely over the years(Belief Propagation,  Topic modeling with anchor words assumptions etc...). This work could be easily extended to multi-class classifications where each node belongs to multiple classes. It would be interesting to know the authors' thoughts on that. The hard classification rule in the paper seems to be too restrictive to be of use in practical scenarios, and soft classification would be a useful pragmatic alternative.