MULTI-LABEL METRIC LEARNING WITH BIDIRECTIONAL REPRESENTATION DEEP NEURAL NETWORKS

1 INTRODUCTION . We consider the problem of multi-label classification , because it wide application . For example , in document classification , one document can cover different topics ; in computer vision , one scene image can contain multiple semantic classes like mountain , beach and sea . We address the problem via metric learning approach , where instances with the more different labels move the further away . After the metric learning is completed , the problem is reduced as easy memory-based learning . Thus , the k nearest neighbor ( KNN ) , one of popular and simple methods , can easily handle the multi-label classification . For a new testing instance xtest , the predicted labels are : y = labels with most instances in { f ( xi ) Ki=1 } , i ∈ Nei ( F ( xtest ) ) ( 1 ) f ( · ) can be a linear mapping , a non-linear RKHS mapping , or neural network-like differentiable function . The performance of kNN significantly relies on the learning of distance metric that computes latent distance between different instances and classes . Kwok and Tsang ( Kwok & Tsang , 2003 ) show that in a single-label prediction task , via pushing two origin nearby instances of different classes further apart with a large margin in RKHS , performance of kNN can be greatly improved . Multi-labels prediction setting , however , restricts the capacity of existing advanced metric learning techniques like the popular LMNN Weinberger & Saul ( 2009 ) , as different classes can range from a few to hundred different labels , but the appropriate distance metric could not be provided . The recent work of multi-label metric learning ( Liu et al. , 2018 ) gave a systematic studies on advanced methods and proposed a new paradigm with better performance , but still suffers following limitations : 1 ) inferior capacity : the metric learning is based on linear projection , it doesn ’ t learn flexible representation with non-linearity . 2 ) application restriction : they only handle the hand-craft feature data , but cannon deal with raw image or audio data , or requires pre-processing techniques that may lost key information of the raw data before learning the representation . Therefore , it is non-trivial to break the limitations of existing work and develop capable and extendable framework for solving multi-output learning tasks . In this work , during training , we project the feature data or directly raw image data into the metric space using neural networks model and incorporate labels information in a bidirectional way . Incorporating labels will exploit the label dependency during learning the metric representation f ( · ) , instead of treating them as independent ones . Integrating F ( · ) and G ( · ) in a bidirectional way will allow the model concentrate the instances with identical label and will enable it to get the metric representation of testing data without labels . Our contributions are two-fold . First , we introduce a simple and well-behaved neural network framework for multi-output task and show how it can motivate via output representation and deep neural networks . Secondly , we demonstrate how this form of neural network framework can be used for scalable and extendable multi-label classification tasks and propose how it extend to other multioutput tasks . Experiments on a number of data sets demonstrate that our model compares favorably both in multi-labels classification performance ( micro-F1 , example-based F1 , precision and accuracy ) and extendability ( able to modify to more and complex tasks ) against state-of-the-art methods for multi-label learning . 2 METRIC SPACE WITH BIDIRECTIONAL REPRESENTATION . In this section , we provide motivation for a specific neural network model that we will use in the rest of this paper . We consider a basic and simple neural networks framework with the following propagation : H = σ ( WhX + bh ) ( 2 ) Mx = WxH + bx ( 3 ) My = WyY + by ( 4 ) Here , X and Y are the input features and output labels , respectively . The W s and bs are the weights and bias of corresponding layers . σ ( · ) denotes an activation function , that incorporate the non-linear propagation , such as the ReLU ( · ) = max ( 0 , · ) . Mx and My have the same shapes , i.e . X and Y are mapped to the same metric space via non-linear multi-layer and single-layer networks . In the following , we show that this basic framework is motivated via outputs representation , and deep convolutional network for visual recognition on multi-label data set . 2.1 OUTPUTS REPRESENTATION FOR METRIC LEARNING . We consider a metric spaceM in the network embedding defined as mX ∈ Rd ( a vector in metric space corresponding to an instance X in input space ) with a mapping F ( X ) parameterized by the networks , i.e . : mX = F ( X ) ( 5 ) Another important component of the proposed framework is to model the relationship between metric space and the outputs set Y ∈ Rq . Since mapping of outputs to metric space are generally unknown in advance , we proposed a new representation , namely Reversed Embedding , In order to leverage the G ( Y ) to bridge the connection between metric spaceM and corresponding data X in input space . mY = G ( Y ) ( 6 ) Specifically , our intuition is that if the output labels of any two instances yi , yj are the same then two corresponding xi , xj in the input space should map very closed to each other in the metric space . To capture this intuition , we minimize the following objective function as an bidirectional representations in the same metric space , given by min F , G ‖mY −mX‖22 ( 7 ) In a good metric learning , the model should promote discriminative and predictable representation . To obtain a discriminative representation , the instances of the same outputs will not be confused with ones of different outputs . As for a predictable representation , the above approach provides an opportunity to exploit the dependency between input data and outputs in the metric space . 2.2 NEURAL NETWORK FOR DEEP REPRESENTATION . In the last subsection , we mention that the predictable representation requires dependency between input and outputs . However , in real-world application , two instances for the identical outputs may be very distinct in the input space , while the ones with different outputs may sometime look confusing originally , especially for raw image data . This raised issue lights the challenge to learn the MX = F ( X ) with sufficient representation capacity . A neural networks model can therefore be built by stacking multiple layers based on the form of Eq 2 , where the hidden layer can optionally followed by a point-wise non-linearity . Now , imagine we have to learn an xor mapping from input to metric space , that is non-linear separability , and therefore the non-linear activation like sigmoid or ReLU is applied right after linear mapping to handle it . Our framework is also extendable , since in this way , we can also learn the unstructured data like image and audio by replacing the linear mapping layer with convolutional layer . Additionally , for a fixed computational budget , the batch-wise feeding allows us to build a scalable models , a practice that is known to train on a large number of data set . Unlike MMOC and ML-KNN ( Zhang & Schneider , 2012 ; Zhang & Zhou , 2007 ) , there are no quadratic or exponential computational costs during training with respect to the number of instances . 3 DEEP METRIC LEARNING FOR MULTI-LABEL CLASSIFICATION . Having introduced a basic , yet extendable framework for efficient information propagation and integration on both input data and outputs , now we can be back to the multi-label classification task . As outlined in the introduction , we project input training data with integration of output label during metric learning , but in a bidirectional way , then get the metric representation of testing data without labels , finally a simple KNN will obtain the predicted labels . The overall model , a Bidirectional Representation Deep Neural Networks ( BiRD-NN ) for multi-label metric learning , is schematically depicted in Figure 1 3.1 SIMPLE EXAMPLE FOR FEATURE INPUT DATA . In the following , we consider F ( X ) as a two-layer representation network for input X ∈ Rp×n , i.e . n instances of p-feature data . We incorporate the non-linearity σ ( · ) in the hidden layer of F ( x ) , then the forward representation takes this simple form : F ( X ) = Wx ( σ ( WhX + bh ) + bx ) ( 8 ) Here , Wh ∈ Rh×p is an input to hidden weights matrix for a hidden layer of h features . Wx ∈ Rd×h is a hidden-to-metric weight matrix and d is the dimension of metric space . bh ∈ Rh and bx ∈ Rd are the corresponding bias vectors . The non-linear activation is defined as σ ( · ) = ReLU ( · ) = max ( 0 , · ) . For metric learning , we reversely represent the q-labels of each instance , and pull the inputs of the same labels together : G ( Y ) = WyY + by ( 9 ) min G , F n∑ i=1 ‖F ( Xi ) −G ( Yi ) ‖22 ( 10 ) where Wy ∈ Rd×p , Y ∈ Rp×m and by ∈ Rd , so with F ( X ) , G ( Y ) ∈ Rn×d and we can calculate the instance-wise L2 distance . As there are instances with the identical labels mapping to identical G ( Y ) , their inputs move closed indirectly via F ( X ) and the minimizing distance objective . The neural network weights W s and bias bs are trained using gradient descent . In this work , we can perform batch gradient descent training the full data-set , which is a viable option as long as data sets fit in memory . Using a instance-wise L2 distance , time requirement is O ( n ) , i.e . linear in the number of instance . Another constraint for metric learning is that the instances with different labels should move far away in the metric space . In the multi-labels setting , the difference between two instances can be a few labels to hundred labels . Therefore , we can define how far away are two instance in the metric space according to the difference in the labels space . Following Liu et al . ( 2018 ) ’ s work , we define it as ∆ ( Yi , Yj ) using the euclidean metric to measure the distance between instances Xj and Xj and then learn a new distance metric space , which improves the performance of discriminative classification . To let different instances have a margin of ∆ ( Yi , Yj ) in the learning metric space , we have the following objective function . min F n∑ i max ( 0 , ∆ ( Yi , Yj ) − ‖F ( Xi ) − F ( Xj ) ‖22 ) , j ∈ Nei ( F ( Xi ) ) , ∀j ( 11 ) Here , Nei ( F ( Xi ) ) is the k neighbor instance of F ( xi ) . Note here searching the k nearest instance of F ( xi ) has a time computational cost of O ( n2 ) , but with mini-batch stochastic gradient descent training of neural network , we can search neighbors in each mini-batch instead of the whole data set . Let nmini be the number of each mini-batch , then the kNN time cost is O ( n2mini ) , with n/nmini times of kNN for the whole data set . Therefore the time cost of mini-batch kNN is O ( nminin ) . If we set a small nmini , computational complexity is also linear to the number of instances in training stage ( In the testing stage , we utilize ball-tree k-NN Nitin Bhatia ( 2010 ) ) . We leave efficient-mini-batch and dynamic margin extensions for future work .

Summary: The paper addresses the problem of multi-label prediction.  It proposes a method that uses a co-embedding of instances and labels into a joint embedding space in a way that related instances and labels fall close by and unrelated ones fall far away.  For this purpose, embeddings from input space and label space to a common space are learned from training data. At the prediction time, KNN to the embedding of the test instance in the co-embedding space is used to predict relevant labels.