MULTI-LABEL METRIC LEARNING WITH BIDIRECTIONAL REPRESENTATION DEEP NEURAL NETWORKS

1 INTRODUCTION . We consider the problem of multi-label classification , because it wide application . For example , in document classification , one document can cover different topics ; in computer vision , one scene image can contain multiple semantic classes like mountain , beach and sea . We address the problem via metric learning approach , where instances with the more different labels move the further away . After the metric learning is completed , the problem is reduced as easy memory-based learning . Thus , the k nearest neighbor ( KNN ) , one of popular and simple methods , can easily handle the multi-label classification . For a new testing instance xtest , the predicted labels are : y = labels with most instances in { f ( xi ) Ki=1 } , i ∈ Nei ( F ( xtest ) ) ( 1 ) f ( · ) can be a linear mapping , a non-linear RKHS mapping , or neural network-like differentiable function . The performance of kNN significantly relies on the learning of distance metric that computes latent distance between different instances and classes . Kwok and Tsang ( Kwok & Tsang , 2003 ) show that in a single-label prediction task , via pushing two origin nearby instances of different classes further apart with a large margin in RKHS , performance of kNN can be greatly improved . Multi-labels prediction setting , however , restricts the capacity of existing advanced metric learning techniques like the popular LMNN Weinberger & Saul ( 2009 ) , as different classes can range from a few to hundred different labels , but the appropriate distance metric could not be provided . The recent work of multi-label metric learning ( Liu et al. , 2018 ) gave a systematic studies on advanced methods and proposed a new paradigm with better performance , but still suffers following limitations : 1 ) inferior capacity : the metric learning is based on linear projection , it doesn ’ t learn flexible representation with non-linearity . 2 ) application restriction : they only handle the hand-craft feature data , but cannon deal with raw image or audio data , or requires pre-processing techniques that may lost key information of the raw data before learning the representation . Therefore , it is non-trivial to break the limitations of existing work and develop capable and extendable framework for solving multi-output learning tasks . In this work , during training , we project the feature data or directly raw image data into the metric space using neural networks model and incorporate labels information in a bidirectional way . Incorporating labels will exploit the label dependency during learning the metric representation f ( · ) , instead of treating them as independent ones . Integrating F ( · ) and G ( · ) in a bidirectional way will allow the model concentrate the instances with identical label and will enable it to get the metric representation of testing data without labels . Our contributions are two-fold . First , we introduce a simple and well-behaved neural network framework for multi-output task and show how it can motivate via output representation and deep neural networks . Secondly , we demonstrate how this form of neural network framework can be used for scalable and extendable multi-label classification tasks and propose how it extend to other multioutput tasks . Experiments on a number of data sets demonstrate that our model compares favorably both in multi-labels classification performance ( micro-F1 , example-based F1 , precision and accuracy ) and extendability ( able to modify to more and complex tasks ) against state-of-the-art methods for multi-label learning . 2 METRIC SPACE WITH BIDIRECTIONAL REPRESENTATION . In this section , we provide motivation for a specific neural network model that we will use in the rest of this paper . We consider a basic and simple neural networks framework with the following propagation : H = σ ( WhX + bh ) ( 2 ) Mx = WxH + bx ( 3 ) My = WyY + by ( 4 ) Here , X and Y are the input features and output labels , respectively . The W s and bs are the weights and bias of corresponding layers . σ ( · ) denotes an activation function , that incorporate the non-linear propagation , such as the ReLU ( · ) = max ( 0 , · ) . Mx and My have the same shapes , i.e . X and Y are mapped to the same metric space via non-linear multi-layer and single-layer networks . In the following , we show that this basic framework is motivated via outputs representation , and deep convolutional network for visual recognition on multi-label data set . 2.1 OUTPUTS REPRESENTATION FOR METRIC LEARNING . We consider a metric spaceM in the network embedding defined as mX ∈ Rd ( a vector in metric space corresponding to an instance X in input space ) with a mapping F ( X ) parameterized by the networks , i.e . : mX = F ( X ) ( 5 ) Another important component of the proposed framework is to model the relationship between metric space and the outputs set Y ∈ Rq . Since mapping of outputs to metric space are generally unknown in advance , we proposed a new representation , namely Reversed Embedding , In order to leverage the G ( Y ) to bridge the connection between metric spaceM and corresponding data X in input space . mY = G ( Y ) ( 6 ) Specifically , our intuition is that if the output labels of any two instances yi , yj are the same then two corresponding xi , xj in the input space should map very closed to each other in the metric space . To capture this intuition , we minimize the following objective function as an bidirectional representations in the same metric space , given by min F , G ‖mY −mX‖22 ( 7 ) In a good metric learning , the model should promote discriminative and predictable representation . To obtain a discriminative representation , the instances of the same outputs will not be confused with ones of different outputs . As for a predictable representation , the above approach provides an opportunity to exploit the dependency between input data and outputs in the metric space . 2.2 NEURAL NETWORK FOR DEEP REPRESENTATION . In the last subsection , we mention that the predictable representation requires dependency between input and outputs . However , in real-world application , two instances for the identical outputs may be very distinct in the input space , while the ones with different outputs may sometime look confusing originally , especially for raw image data . This raised issue lights the challenge to learn the MX = F ( X ) with sufficient representation capacity . A neural networks model can therefore be built by stacking multiple layers based on the form of Eq 2 , where the hidden layer can optionally followed by a point-wise non-linearity . Now , imagine we have to learn an xor mapping from input to metric space , that is non-linear separability , and therefore the non-linear activation like sigmoid or ReLU is applied right after linear mapping to handle it . Our framework is also extendable , since in this way , we can also learn the unstructured data like image and audio by replacing the linear mapping layer with convolutional layer . Additionally , for a fixed computational budget , the batch-wise feeding allows us to build a scalable models , a practice that is known to train on a large number of data set . Unlike MMOC and ML-KNN ( Zhang & Schneider , 2012 ; Zhang & Zhou , 2007 ) , there are no quadratic or exponential computational costs during training with respect to the number of instances . 3 DEEP METRIC LEARNING FOR MULTI-LABEL CLASSIFICATION . Having introduced a basic , yet extendable framework for efficient information propagation and integration on both input data and outputs , now we can be back to the multi-label classification task . As outlined in the introduction , we project input training data with integration of output label during metric learning , but in a bidirectional way , then get the metric representation of testing data without labels , finally a simple KNN will obtain the predicted labels . The overall model , a Bidirectional Representation Deep Neural Networks ( BiRD-NN ) for multi-label metric learning , is schematically depicted in Figure 1 3.1 SIMPLE EXAMPLE FOR FEATURE INPUT DATA . In the following , we consider F ( X ) as a two-layer representation network for input X ∈ Rp×n , i.e . n instances of p-feature data . We incorporate the non-linearity σ ( · ) in the hidden layer of F ( x ) , then the forward representation takes this simple form : F ( X ) = Wx ( σ ( WhX + bh ) + bx ) ( 8 ) Here , Wh ∈ Rh×p is an input to hidden weights matrix for a hidden layer of h features . Wx ∈ Rd×h is a hidden-to-metric weight matrix and d is the dimension of metric space . bh ∈ Rh and bx ∈ Rd are the corresponding bias vectors . The non-linear activation is defined as σ ( · ) = ReLU ( · ) = max ( 0 , · ) . For metric learning , we reversely represent the q-labels of each instance , and pull the inputs of the same labels together : G ( Y ) = WyY + by ( 9 ) min G , F n∑ i=1 ‖F ( Xi ) −G ( Yi ) ‖22 ( 10 ) where Wy ∈ Rd×p , Y ∈ Rp×m and by ∈ Rd , so with F ( X ) , G ( Y ) ∈ Rn×d and we can calculate the instance-wise L2 distance . As there are instances with the identical labels mapping to identical G ( Y ) , their inputs move closed indirectly via F ( X ) and the minimizing distance objective . The neural network weights W s and bias bs are trained using gradient descent . In this work , we can perform batch gradient descent training the full data-set , which is a viable option as long as data sets fit in memory . Using a instance-wise L2 distance , time requirement is O ( n ) , i.e . linear in the number of instance . Another constraint for metric learning is that the instances with different labels should move far away in the metric space . In the multi-labels setting , the difference between two instances can be a few labels to hundred labels . Therefore , we can define how far away are two instance in the metric space according to the difference in the labels space . Following Liu et al . ( 2018 ) ’ s work , we define it as ∆ ( Yi , Yj ) using the euclidean metric to measure the distance between instances Xj and Xj and then learn a new distance metric space , which improves the performance of discriminative classification . To let different instances have a margin of ∆ ( Yi , Yj ) in the learning metric space , we have the following objective function . min F n∑ i max ( 0 , ∆ ( Yi , Yj ) − ‖F ( Xi ) − F ( Xj ) ‖22 ) , j ∈ Nei ( F ( Xi ) ) , ∀j ( 11 ) Here , Nei ( F ( Xi ) ) is the k neighbor instance of F ( xi ) . Note here searching the k nearest instance of F ( xi ) has a time computational cost of O ( n2 ) , but with mini-batch stochastic gradient descent training of neural network , we can search neighbors in each mini-batch instead of the whole data set . Let nmini be the number of each mini-batch , then the kNN time cost is O ( n2mini ) , with n/nmini times of kNN for the whole data set . Therefore the time cost of mini-batch kNN is O ( nminin ) . If we set a small nmini , computational complexity is also linear to the number of instances in training stage ( In the testing stage , we utilize ball-tree k-NN Nitin Bhatia ( 2010 ) ) . We leave efficient-mini-batch and dynamic margin extensions for future work .

Summary: This paper aims to solve multi-label problems via learning a share embedding space for instances and its label sets. Specifically, the author considers deep neural networks F(x) as an encoder for the instance (either raw input or features) and a shallow MLPs G(y) as an encoder for the label outputs. In the training stage, the instance embedding and its label embedding are forced to be close. An additional constraint is instances with different labels should be far from each other. After training, the inference can be done in the embedding space by looking up the labels of the query’s kNN instances.