MULTI-LABEL METRIC LEARNING WITH BIDIRECTIONAL REPRESENTATION DEEP NEURAL NETWORKS
Authors: Tao Zheng
Ivor Tsang
Xin Yao
Venue:  (2020)
Keywords: metric learning
representation learning
multi-label classification
multi-output

Abstract: Multi-Label Learning task simultaneously predicting multiple labels has attracted researchers' interest for its wide application. 
Metric Learning crucially determines the performance of the k nearest neighbor algorithms, the most popular framework handling the multi-label problem.
However, the existing advanced multiple-label metric learning suffers the inferior capacity and application restriction. 
We propose an extendable and end-to-end deep representation approach for metric learning on multi-label data set that is based on neural networks able to operate on feature data or directly on raw image data. 
We motivate the choice of our network architecture via a Bidirectional Representation learning where the label dependency is also integrated and deep convolutional networks that handle image data. 
In multi-label metric learning, instances with the more different labels will be dragged the more far away, but ones with identical labels will concentrate together. 
Our model scales linearly in the number of instances and trains deep neural networks that encode both input data and output labels, then, obtains a metric space for testing data. 
In a number of experiments on multi-labels tasks, we demonstrate that our approach is better than related methods based on the systematic metric and its extendability.   


Under review as a conference paper at ICLR 2020
MULTI-LABEL METRIC LEARNING WITH BIDIREC- TIONAL REPRESENTATION DEEP NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Multi-Label Learning task simultancously predicting multiple labels has attracted researchers' interest for its wide application. Metric Learning crucially deter- mines the performance of the nearest neighhor algorithms, the popular framework handling the multi-label problem. However, the existing advanced multiple-label metric learning suffers the inferior capacity and application restric- tion. We propose an extendable and end-to-end deep representation approach for metric learning on multi-label data set that is based on neural networks able operate on feature data or directly on raw image data. We motivate the choice of our network architecture via Bidirectional Representation learning where the label dependency is also integrated and deep convolutional networks that handle image data. In multi-label metric learning, instances with the more different labels will be dragged the more far away, but ones with identical labels will concentrate together. Our model scales linearly in the number of instances and trains deep neural networks that encode both input data and output labels, then, obtains metric space for testing data. In a number of experiments on multi-labed tasks, we demonstrate that our approach is better than related methods based the sys- tematic metric and its extendability.
INTRODUCTION
We consider the problem of multi-label classification, because it wide application. For example, in document classification, one document can cover different topics; in computer vision. one scene image can contain multiple semantic classes like mountain, beach and sea. We address the problem via metric learning approach, where instances with the more different labels move the further away. After the metric learning is completed, the problem is reduced as easy memory-based learning. Thus, the k nearest neighbor (KNN), one of popular and simple methods, can easily handle the multi-label classification. For a new testing instance Tteats the predicted labels are:
y = labels with most instances in (x:) Nei(F((tea))
(1)
f(-) can be linear mapping, non-linear RKHS mapping, or neural network-like differentiable function. The performance of kNN significantly relies on the learning of distance metric that com- putes latent distance between different instances and classes. Kwok and Tsang (Kwok & Tsang 2003) show that in a single-label prediction task, via pushing two origin nearby instances of differ- classes further apart with large margin in RKHS, performance of KNN can be greatly improved. Multi-labels prediction setting, however, restricts the capacity of existing advanced metric learning techniques like the popular LMNN Weinberger & Saul (2009). different classes can range from a few to hundred different labels, but the appropriate distance metric could not be provided. The recent work of multi-label metric learning (Liu al., 2018) gave systematic studies on advanced methods and proposed new paradigm with better performance, but still suffers following limita- tions: 1) inferior capacity: the metric learning based on linear projection, doesn't 't learn flexible representation with non-linearity. 2) application restriction: they only handle the hand-craft feature data, but cannon deal with raw image or audio data, or requires pre- pre-processing techniques that may lost key information of the raw data before learning the representation. Therefore, it is non-trivial to break the limitations of existing work and develop capable and extendable framework for solving multi-output learning tasks.

---

Under review as a conference paper at ICLR 2020
In this work, during training, project the feature data directly raw image data into the metric space using neural networks model and incorporate labels information in a bidirectional way. In- corporating labels will exploit the label dependency during learning the metric representation f(-), instead of treating them as independent ones. Integrating F.)) and G(.) bidirectional way will allow the model concentrate the instances with identical label and will enable it to get the metric representation of testing data without labels.
Our contributions two-fold. First, we introduce simple and well-behaved neural network frame- work for multi-output task and show how it can motivate via output representation and deep neural networks. Secondly, we demonstrate how this form of neural network framework can be used for scalable and extendable multi-label classification tasks and propose how it extend to other multi- output tasks. Experiments on number of data sets demonstrate that our model compares favorably both in multi-labels classification performance (micro-F1, example-based F1, precision and accu- racy) and extendability (able to modify to more and complex tasks) against state-of-the-art methods for multi-label learning.
2 METRIC SPACE WITH BIDIRECTIONAL REPRESENTATION
In this section, we provide motivation for specific neural network model that we will use in the rest of this paper. We consider a basic and simple neural networks framework with the following propagation:
H aW + bn.
(2)
M2 W.H by My = W,Y + by
(4)
Here, X and Y are the input features and output labels, respectively. The Ws and bs are the weights and bias of corresponding layers. o-) denotes an activation function, that incorporate the non- non-linear propagation, such as the (e..U()) = max((0,) -). M2, and M, have the same shapes, i.e. X and Y are mapped to the same metric space via non-linear multi-layer and single-layer networks. In the following, we show that this basic framework is motivated via outputs representation, and deep convolutional network for visual recognition on multi-label data set.
2.1 OUTPUTS REPRESENTATION FOR METRIC LEARNING
We consider a metric space M in the network embedding defined as m.x E IRd (a vector in metric space corresponding to an instance X in input space) with mapping F(X) parameterized by the networks, i.e.:
(5) Another important component of the proposed framework is to model the relationship between met- ric space and the outputs set Y E T29. Since mapping of outputs to metric space are generally unknown in advance, proposed a new representation, namely Reversed Embedding, In order to
mx FX)
leverage the G(Y) to bridge the connection between metric space M and corresponding data in input space.
(6) Specifically, our intuition that if the output labels of any two instances yi Mj are the same then two corresponding 24, 2y in the input space should map very closed to each other in the metric space. To capture this intuition, we minimize the following objective function as bidirectional representations in the same metric space, given by
my = G(Y)
min my mx F.G
In a good metric learning, the model should promote discriminative and predictable representation. To obtain discriminative representation, the instances of the same outputs will not be confused with ones of different outputs. As for a predictable representation, the above approach provides an opportunity exploit the dependency between input data and outputs in the metric space.
2

---

Under review as a conference paper at ICLR 2020
(oouraale,
X2 nnouralle, desert
msdt
Y1:
1100
6011) * FOO)
Y2:
1010
FO(7)
train
Y3:
0011
Y4:
1001
train
ajve) FXX4)
Y5:
0011
GPV2) *
msdt
frres doart
14 jese, mourton)
X5 free, desert)
GEYELG goss)
pradict
I6:
0011
F(X3) * (Q6))
Y7:
1100
test
Figure I: schematic depiction of (Bidirectional Representation Deep Neural Network) BiRDNN for multi-label metric learning. Deep Representation of input X is learned by F(X) into the metric space, where reversed embedding of Y is another representation G(æ) from the opposite direction. The letters, m t, denote the labels, mountain, sec, desert, tree. The corresponding X and Y are mapped to very closed position in the metric space, therefore the different inputs with the same labels will be closed, too. In the testing stage, the predicted label new X is obtained by the labels of the nearest training instances in the metric space.
2.2 NEURAL NETWORK FOR DEEP REPRESENTATION
In the last subsection, mention that the predictable representation requires dependency between input and outputs. However, in real-world application, two instances for the identical outputs may be very distinct the input space, while the ones with different outputs may sometime look confusing originally, especially for raw image data. This raised issue lights the challenge to learn the Mx = F(X) with sufficient representation capacity. stacking multiple layers based on the form of Eq where the hidden layer can optionally followed by a point-wise non-linearity. Now, imagine we have learn an xor mapping from input to metric space, that is non-linear separability, and therefore the non-linear activation like sigmoid or ReLU applied right after linear mapping to handle
neural networks model can therefore be built by
Our framework is also extendable, since in this way, image and audio by replacing the lincar mapping layer with convolutional layer.
also learn the unstructured data like
Additionally, for fixed computational budget, the batch-wise feeding allows build scal- able models, a practice that is known to train on a large number of data set. Unlike MMOC and ML-KNN (Zhang & Schneider 2012, Zhang & Zhou 2007), there are quadratic exponential computational costs during training with respect to the number instances.
3 DEEP METRIC LEARNING FOR MULTI-LABEL CLASSIFICATION
Having introduced a basic, yet extendable framework for efficient information propagation and in- tegration on both input data and outputs, now can be back the multi-label classification task. As outlined in the introduction, we project input training data with integration of output label during metric learning, but bidirectional way, then get the metric representation of testing data without labels, finally a simple KNN will obtain the predicted labels. The overall model, a Bidirectional Representation Deep Neural Networks (BiRD-NN) for multi-labe Malti-label metric learning, is schematically depicted in Figure
3.1 SIMPLE EXAMPLE FOR FEATURE INPUT DATA
In the following, consider F(X) two-layer representation network for input X Roxn iee. n instances of p-feature data. We incorporate the non-linearity a(-) in the hidden layer of F(r), then the forward representation takes this simple form:
F(X) W2 (a (WWW X
bb) + bz)
(8)
3

---

Under review as a conference paper at ICLR 2020
Here, WA E R is input hidden weights matrix for hidden layer of h features. Wa E Rdxh a hidden- tellen-to-mann to-metric weight matrix and d is the dimension of metric space. by E R and by E Rd are the corresponding bias vectors. The non-linear activation is defined a-) R-LU(()) max(0, ))-
For metric learning, we reversely represent the q-labels of each instance, and pull the inputs of the same labels together:
G(Y) - W.Y
(9)
min
[F(X,) - G(X))12
(10)
where Wg E Rdxp, Y E RPxm and by E R so with F(X), G(Y) E R and we can calculate the instance-win L2 distance. As there are instances with the identical labels mapping to identical G(Y), their inputs move closed indirectly via F(X) and the minimizing distance objective. The neural network weights Ws and bias bs are trained using gradient descent. In this work, we can perform batch gradient descent training the full data-set, which is viable option as long as data sets fit in memory. Using a instance-wise L2 distance, time requirement (n)) i.c. linear in the number of instance.
Another constraint for metric learning is that the instances with different labels should move far away in the metric space. In the multi-labels setting, the difference between two instances can be a few labels to hundred labels. Therefore, we can define how far away are two instance in the metric space according to the difference in the labels space. Following Liu et al. (2018)'s work, define it as A(Y, Yj) using the euclidean metric to measure the distance between instances X, and X, and then learn new distance metric space, which improves the performance of discriminative classification. To let different instances have a margin of (Y; Yj) in the learning metric space, we have the following objective function.
min m0 A(Y. (IF(X)) -
Nei(F(X.)).)
(11)
Here, Nei(F(X,)) is the k neighbor instance of F(x;). Note here searching the k nearest instance of F(x;) has a time computational cost of On)²), but with mini-batch stochastic gradient descent training neural network, we can search neighbors each mini-batch instead of the whole data set. Let Amint be the number of each mini-batch, then the kNN time cost is O On with n Remins kNN is O If set small mmini: computational complexity is also linear the number of instances in training stage (In the testing stage, we utilize ball-tree k-NN Nitin Bhatia (0100).
times of KNN for the whole data set. Therefore the time cost of
We leave efficient-mini- batch and dynamic margin extensions for future work.
3.2 EXTENSION FOR UNSTRUCTURED INPUT DATA
The aforementioned simple example is shallow model and the training data are hand-crafted fea- ture instead of original data. Fortunately, our framework can be easily extended more complex application that is involved with unstructured original data like image. Convolutional Neural Net- works (CNN) are designed conduct task where training inputs are two-dimensional images for computer vision tasks. CNNs can be directly applied in our framework to implement the F(X). Stacking bunch of convolution layers to learn representation, generally call deep representa- tion learning.
Deep representation learning (Goodfellow et al., 2016) has achieved great success due to it high representation capacity and end-to-end learning manner, i.e. the raw images data are fed to the deep neural networks directly without the hand-crafted feature extraction as a pre-processing. There are batch of image pre-procee propceccssi methods such such SE, SIFT and PCA, converting an image to feature vector, but they are not about learning and might lose key information of the original image, which is inferior to deep representation method based its richer feature learning capacity.
4

---

Under review as a conference paper at ICLR 2020
3.2.1 COMPARISON WITH RECENT DEEP MULTI-LABEL LEARNING
In our extended model and the last experimental comparison with a deep multi-label learning model AuCNN (Hand et al. 2018), we implement their convolutional component via 4 stacking convo- lutional network blocks, each of which contains 3x3 Convolutional Kernel, BatchNormalization, ReLU and a 2x2 max pool. On the top of the convolutional blocks is 2 layers of full connection networks. This is tailored and modified version of VGG, but the last layer is changed to make the AuCNN and BiRDNN. 1) the last layer is replaced by a multiple- fatiaiiellabll(aarbbuttooo layer and this model act as a baseline. 2) the last layer changed to map to a metric space that is shared with the reversed embedding of BiR and the model is call Bidirectional Representation Deep Neural Net- works (BiRDNN) for multi-label metric learning. Limitation of AuCNN: it learns multiple labels at simultaneously, but it treats them independent ones, and dose not exploit the label dependency during learn the representation as BiRDNN does. AtCNN also was affected by the global skewed data distribution, while BiRDNN predicts multiple labels by local information in the metric space.
4 RELATED WORKS
The fields of conventional multi-output metric learning and recent works on neural network earning deep representation inspire to propose out model. In what follows, we provide brief overview on related works of both fields.
4.1 CONVENTIONAL MULTI- METRIC LEARNING
A number of approaches for multi-output metric learning have been proposed in the past decades, most of which only handle hand-crafted feature data and involve two main approaches: encoding- decoding projection and large margin constraints. Prominent examples for
pro- jection variant include out encoding (Zhang & Schneider (012)) and canonical correlation analy- sis (Zhang & Schneider 201T). Liu & Tsang 52015) and Parameswaran Weinberger (2010)
designed large margin constraints for metric learning on multi-out data.
Recently, attention has shifted to models that can be applied to large-scales data sets. CoHasing (Shen et al., 2018) compressed input and output to compact binary embedding, thus align the input and output space, and the prediction is very efficient. LMMO-KNN (Liu et al. 2018) proposed several strategies to speed up the training and testing time after systematic review is given. For all these methods, however, a multi-step pipeline is required where each step has to be optimized separately, and most of them only handle hand-crafted feature data.
4.2 DEEP METRIC LEARNING OR DEEP MULTI-LABEL LEARNING
Deep networks that not only map from representation to output but also learn representation itself have recently proposed. Learned representation often result in better performance than that can be obtained with hand-designed representation or hand-crafted features. In the context of metric learning multi-label data set, there recent researches on the single field of them: deep metric learning or deep multi-label learning.
For deep metric learning, Multi-Manifold Deep Metric Learning (MMDML) (Lu et al. 2015) learn multiple sets of non-linearly map of multi-class images to shared feature space, under which the margin of different class manifold is maximized. A deep metric learning variant (Niethammer al. 2019) was proposed to address the image registration in medical image analysis. The sample mining methods for deep metric learning are reviewed and (Online Soft Mining) OSM is proposed in Wang (Oh Song et al. 2016) is proposed to improve the deep metric learning in terms of Fl and NMI measures for clustering task and recall @ score for retricval tasks.
et al. (2019) Lifted Structured Feature Embedding
For deep multi-label learning, SH+CNN (Zhao et al. 2015) is a deep semantic ranking based method image retrieval via learning hash functions multi-label images, breaking the bottleneck of early ones that only handle the simple binary similarity. LMLE (Huang et al.) 2016) and CRL (Dong et al.) 2018) are proposed to rectify the class skew distribution when learning multi-label image data set, but their work treat each label as independent one and process them one by one. Recently, Multi- Attribute CNN (AtCCN) (Hand al. 2018) in proposed for aulti-attribttion image classification
5

---

Under review as a conference paper at ICLR 2020
where 40 labels of image trained and test simultaneously in deep CNN architecture. In this paper, we will use the last one as a baseline in the multi-label image classification task. From the best of our knowledge, output dependency during learning the representation.
are the first attempt on deep multi-labed metric learning that also considers the
5 EXPERIMENTS
We test model in variety of real-world multi-label experiments: We first compare a simple implement our BiR with several state-of-the-ard multi-label metric prediction methods on the series of conventional feature data. Then we compare deep representation implement, BiRDNN, with the recent deep multi-label learning on a raw image data set.
5.1
DATA SETS
Table 1: Statistics of Datasets for the simple example of BiR
name
domain
instances
nominal
numeric
labels
scene
image
2407
0
294
6
core15k images
5000
499
374
delicious
text (web)
16105
500
0
983
EUR-Lex(de)
text
19348
0
5000
412
EUR-Lex(ed)
text 19348
0
5000
3993
For the simple implement example, we closely follow the experimental setup in Liu et al. (2018) Data set statistics are summarized in Tabel All data available on the website
For the deep deep representation implement BiRDNN, we use the raw image dataset, called scene 2000, that is available online" with both processed and raw data set. The data set consists of 2,000 natural scene images each of which that contain one or more of the following objects: desert, moun- tains, sunset, and trees. Several typical images of this data set has been showed in the Figure|
5.2 EXPERIMENTAL SETUPS
For simple example, train shallow BiR described in Section and systematically evaluate performance by the micro-F1 measure and example-baseds measures. These two measures are described and analyzed in Mao et al. (2012) and applied the same data and baselines in Liu et al. 2018). 10-fold cross validation is conducted to get the mean and standard deviation.
For the BiRDNN, make fair comparison, we implement identical CNN architecture with that of our baseline AtTCNN as the bottom part BiRDNN's F(X). ). We train both models for a maximum of 200 epochs (training iterations) using Adam (Kingma & Bal 2014) with a learning rate of 0.001. We initialize weights using the xavier uniform distribution for Cull-Connection layers and the normal distribution for Convolution layers. We prevent the models from over-fitting with a Dropout rate of 0.5 in the Full-Connection layers.
5.3 BASELINES
We compare simple implement of BiR against the same baseline methods and the proposed one in Liu et al. (2018), i.c PLST (Tai Lin 2012), MMOC (Zhang & Schnnider] 2012), KNN,ML- kNN (Zhang & Zhou 2007). and LMMO-KNN (Liu al. 2018).
We further compare our deep representation implement BiRDNN against a recent deep multi-label neural networks, AttCN,, proposed in Hand et al. (2018) over multi-label image data set. To make a fair comparison, we tailed the AtCNN for these experiments, and implemented AuCNN and BiRDNN with the same convolutional component: 4 convolutional network blocks is stacking.

---

Under review as a conference paper at ICLR 2020
and each of them contain 3x3 convolutional filter, BatchNormalization, ReLU and 2x2 pool. and the numbers of output channels in
The input image is (128, 128, 3) of the convolutional filters 192. 256, 512, and 1024, respectively. Although it may not the best tailed implement, but it capable to demonstrate our work. On the top of the convolutional blocks 2 layers of full connection networks. The output layer of AtCNN contains 5 node that is same as the number of labels in these experiments.
By the way, we cite the result of original work, MIML (Zhou & Zhang 2006 who published this dataset. MIML trained its model after a hand-crated pre-processing to extract feature vectors, which different from the above 2 deep networks that learns directly on the raw image in an end-to-end manner. Nevertheless, comparing their result in detail helps get insight to discuss our model and feature work.
RESULT
6.1
SIMPLE EXAMPLE FOR FEATURE INPUT DATA
Table 2: The Results of Micro-Fl -l on the Feature Data Sets (Mean + Standard Deviation)
Dataset
PLST
MMOC
KNN
ML-KNN
LMMO-KNN
BiR
scene
0.592410.03
0.729710.03
0.722110.02
0.738710.03
0.730010.03
0.80718900002111
core15K
0.0801+0.01
0.1006+0.001
0.0278+00.01
0.1834+0.02
0.343582+00.01522
delicious
0.142310.01
0.2338110.01
0.173810.01
0.304620.01
0.2495412:.011315
EUR-Lex (dc)
0,2304
0.6988 10.01
0.6705 10.01
0,7388
0.684401 10.02T701
EUR-Lex (ed)
0.105910.01
0.4664.10.01
0.386410000
0.470610000
0.5386710.031739
Table 3: The Results of Example-F1 on the Feature Data Sets (Mean Standard Deviation)
Dataset
PLST
MMOC
KNN
ML-KNN
LMMO-RNN
BiR
0.4588 0.04
0.702510.0
0.881510.02
0.687410.03
0.7101 10.03
0.804263 0.025510
corel5K
0.058710.01
0.077310.01
0.017810.01
0.151710.00
0..059011..013047
delicious
0.1131+0.00
0,2094+0..01
0.1518+0..00
0.2665+00.00
0.253124-0.00232
dEUR-Lex (de)
0.153020.02
0.656910000
0.603820:01
0.7230:10.01
0.671230-00010376
EUR-Lex (ed)
0.072510.01
0.420710.00
0.3385
0,4630100.00
0.395483-0.012000
Results of the simple example of BiR summarized in Table 2 and For the feature input dataset,
we report the Micro-F1 and example-based-Fi with the 10-fold validation on each dataset. The reason with using these measures method are taken from the paper Liu et al. 2018).
analyzed in Mao et al. (2012). Result for all other baseline
Here, the basic and simple example of our model merely from hand-crafted feature to metric space. Nevertheless, it addressed these multi-label metric learn- ing problems and outperformed the baseline in most of the data sets.
shallow and nonlinear projection
6.2 BIDIRECTIONAL REPRESENTATION DEEP NEURAL NETWORKS
Table 4: The predicive accuracy On Multi-Label image data set scene 2000 with 5 labels
labels methods
MIML
AuCNN
BiRDNN
desert
0.86910.004
0.902142400013668
0.9128122400187055
mountains
0.79140.024
0.84357140.229911
0.877187+00018436
sta
0.72910.026
0.71357110.335002
0.792812-100020522
sunset
0.864+0.033
0.943571+0.01
0.951362+0.019440
trees
0.801=0.015
0.787142440.018033
0.879687:400026612
overall accuracy
0.81140.022
0.82751440.01179
0.882812+0.01070:
average precision
0.78310.020
0.850935100024995
0.86206310.034685
We compare deep natural network implementation out model (BiRDNN) with AtCNN Hand et al. 2018) with identical convolutional network paper scene image dataset. We also take the
7

---

Under review as a conference paper at ICLR 2020
result of MIML from the previous work Zhou & (2066) that learning the pre-processed feature on the same dataset instead of the raw data. Results are summarized in the Tables
Because in the scene image data, there much more negative samples than positive samples in all labels/tags, so AtCNN tends to mis-classify positive to negative, but tends not to mis-classify negative to positive. Therefore, false negative(fn) samples is more than false positive( (p) samples, so precision(tp/ (tp + f')) is greater than accuracy (tp + f + tn + fp + fn). MIML is likely not affected by the positive and negative distribution. BiRDNN uses the identical convolution component with AuCNN, but with the bidirectional representation it obtained better performance, and the metric learning predicts label via local information, the global skewed data distribution does not degenerate the result much.
7 DISCUSSION
7.1
MULTI-LABEL CLASSTFICATION MODEL.
In the experiment demonstrate here, our simple example method for multi-label classification out- performs recent methods. Method based output coding (Zhang & Schneider, [012 Zhang & Zhou 2007 Liu et al. 2018) are most likely limited to the capability of the feature mapping, or let say representation learning. The AttCNN (Hand et al.) 2018) the other hand is limited by the fact that it ignore the label dependency. Our proposed framework can overcome both limitations. In the framework of BiR, first can implement the non-linear mapping easily just via 2 full-connection layers with an activation in the hidden layer. Moreover, the input and output representation share the same embedding when learning the metric space, thus the label dependency exploited. For extensions, we can replace the full-connection networks by the more deeper convolutional to learn the representation for image data directly.
7.2 EXTENDIBILITY TO OTHER MULTI-OUT TASK
We have proposed and demonstrate capable and extendable metric learning framework for multi- label classification MI.C). Our framework can extend to multi-target regression (MTR) where the binary label replaced by real value as well the multi-concept retr)
7.3 LIMITATION AND FUTURE WORKS
Here, outline several limitations of current work and how these might be overcome in future work.
(I) Testing time: After learning the discriminative and predictive metric space, found the test stage is still not ideally fast enough using the on-the-shelf kNN tools. (2) Larger data set: Due the limited computational resource dataset with 2000 raw images that not really large data set like the CelebA. Our future work will be conducted in larger and challenging data sets.
handle currently, complete the experiment the scene
CONCLUSION
We have proposed novel framework, BiR, for metric learning on multi-label data in including hand-crafted features and raw images. Our BiR frame uses an efficient bidirectional representation that incorporates label information when learning the diseriminative and predictive metric space where input data are projected into. Experiments on a number of multi-label data sets suggest that the proposed BiR framework is capable of learning the representation for multi-label classification. In this setting, our model outperforms several recently proposed methods by significant margin, while being extendable for more multi-output learning tasks and more complex applications.
REFERENCES
Qi Dong, Shaogang Gong, and Xiatian Zhu. Imbalanced deep learning by minority class incremental rectification. IEEE transactions on paltern analysis and machine intelligence, 1377-1818
8

---

Under review as a conference paper at ICLR 2020
2018.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Emily M Hand, Carlos Castillo, and Rama Chellappa. Doing the best we can with what we have: Multi-label balancing with selective learning for attribute prediction. In Thirty-Second AAAI Conference on Artificial Intelligenc, 2018.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5375-5384, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1r12. 69800, 2014.
James T Kwok and Ivor W Tsang. Learning with idealized kernels. In Proceedings of the 20th International Conference Machine Learning (ICML-03), pp- 400-407, 2003.
Weiwei Liu and Ivor W Tsang, Large margin metric learning for multi-label prediction. In Twenty- Ninth AAAI Conference on Artificial Intelligence, 2015.
Weiwei Liu, Donna Xu, Ivor W Tsang, and Wenjie Zhang. Metric learning for multi-output tasks. IEEE Iransactions pattern analysis and muchine intelligence, 4 (12::4084222, 2018.
Jiwen Lu, Gang Wang, Weihong Deng, Pierre Moulin, and Jie Zhou. Multi-manifold deep metric learning for image set classification. Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 1137-1145, 2015.
Qi Mao, Ivor Wai-Hung Tsang, and Shenghua Gao. Objective-guide image annotation. Transactions on Image Processing, 22(4):1555-1977 2012.
IEEE
Mare Niethammer, Roland Kwitt, and Francois Xavier Vialard. Metric learning for image registra- tion. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 2463-8472, 2019.
Vandana Nitin Bhatia. Survey of nearest neighbor techniques. https:/ariix.orglas///007.000 2010.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4004-4012. 2016.
Shibin Parameswaran and Kilian Weinberger. Large margin multi-task metric learning. In Ad- 2010.
vances in neural information processing systems, pp.
Xiaobo Shen, Weiwei Liu, Ivor W Tsang, Quan-Sen Sun, and Yew-Soon Ong. Compact multi-label learning. Thirty- Second AAAI Conference on Artificial Intelligence, 2018.
Farbound Tai and Hsuan- Seeun--Tien Lin. Multilabel classification with principal label space transforma- tion. Neural Computation, (00)))250--5423 2012.
Xinshao Wang, Yang Hua, Elyor Kodirov, Guosheng Hu, and Neil M Robertson. Deep metric learning by online soft mining and class-aware attention. In Proceedings of the AAAI Conference on Artificial Inielligence, volume 33, pp. 5361-5368, 2019.
Kilian Q Weinberger and Lawrence K Saul. Distance metric learning for large margin nearest neigh- bor classification. Journal of Machine Learning Research, 10(Feb):207-244, 2009.
Min-Ling Zhang and Zhi-Hua Zhou MI-knn: A lazy learning approach to multi-label learning.
Pattern recognition, 40(7):2038-200 2007.
Yi Zhang and Jeff Schneider. Multi-label output codes using canonical correlation analysis. In
Proceedings of fourteenth international conference on artificial intelligence and statistics, pp- 873-882, 2011.

---

Under review as a conference paper at ICLR 2020
Yi Zhang and Jeff Schneider. Maximum margin output coding. pp. 379-386, 2012.
Fang Zhao, Yongzhen Huang, Liang Wang, and Tieniu Tan. Deep semantic ranking based hashing for multi-label image retrieval. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1556-1564, 2015.
Zhi-Hua Zhou and Min-Ling Zhang. Multi-instance multi-label learning with application to scene classification. In Proceedings of the 19th International Conference on Neural Information Pro- cessing Systems, pp. 1609-1616. Citeseer, 2006.
10


Reviews:
[{"review_id": "rylMTp8t5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a metric learning approach for the multi-label classification problem. It basically maps the input features and the output labels into the same space and then uses k-NN to find the closest labels for each inputs. During training, it minimizes the squared Euclidean distance between the input embedding and label embedding. In the experiments, some image datasets and text datasets are used to compare with several multi-label learning algorithms. \n\nThe proposed method is clearly presented in the paper. However, I have several concerns.\n\n1. The proposed method is straightforward and lacks of significant novelty.\n2. The datasets are very small scale in terms of both sample size and label size. \n3. The comparing methods are a little bit out-dated. I would also suggest to compare directly with the naive one-versus-all method using deep learning extraction models. \n4. There are a few typos. For example, on Page 1, \"because it wide application\", \"but cannon deal\", \"may lost key information\"; on Page 2, \"an bidirectional\", \"mapping of outputs to metric space are\"; on Page 3, \"a scalable models\", etc.. Besides, Eq (1) should be written in a more formal way. \n\nOverall, I vote for rejection and the main reason is the lack of novelty. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rJxYss7z5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to solve multi-label problems via learning a share embedding space for instances and its label sets. Specifically, the author considers deep neural networks F(x) as an encoder for the instance (either raw input or features) and a shallow MLPs G(y) as an encoder for the label outputs. In the training stage, the instance embedding and its label embedding are forced to be close. An additional constraint is instances with different labels should be far from each other. After training, the inference can be done in the embedding space by looking up the labels of the query’s kNN instances. \n\nMetric learning for multi-label problems is not new, many works have been proposed such as [1,2]. Using deep neural networks for multi-label problems is also not new, see [3,4]. Thus, the novelty of the proposed method is rather limited. The interesting part is the constraint of Eq(11), where kNN need to be updated whenever the instance encoder model F(X) is changing during the optimization, which is very expensive for large-scale application.\n \n[1] Sparse Local Embeddings for Extreme Multi-label Classification, NIPS 2015.\n[2] Learning Deep Latent Space for Multi-label Classification, AAAI 2017.\n[3] Deep Learning for Extreme Multi-label Text Classification, SIGIR 2017.\n[4] X-BERT: eXtreme Multi-label Text Classification with BERT, ArXiv 2019. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByxiegNpYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of multi-label prediction.  It proposes a method that uses a co-embedding of instances and labels into a joint embedding space in a way that related instances and labels fall close by and unrelated ones fall far away.  For this purpose, embeddings from input space and label space to a common space are learned from training data. At the prediction time, KNN to the embedding of the test instance in the co-embedding space is used to predict relevant labels.\nFeaturized (attributed)  labels are potentially considered, which can facilitate incorporating label dependence and generalization over unseen labels.\n\nMain shortcomings:\n- The novelty of the work is limited. Ideas introduced in this work are present and being investigated in literature for a while now, leading to remaining limited contribution for the paper.  Related work on joint embedding, co-embedding, label-embedding, and zero shot learning seem to be neglected totally. For example, the paper lacks awareness of, citation to and comparison with related work such as [1-5].\n- Presentation of the paper can be highly improved. There are several grammatical and writing problems in the paper.\nFormulation can also benefit from  improved presentation. See for example Eq (1).\n- Technical arguments are not all well founded. For example, the scalability claim in the abstract  of the paper seems to refer to \"prediction\" time complexity being linear in the number of \"training\" examples, which is not actually fast.\n\nIn summary, based on the above reasons, I vote for the paper to be strongly rejected.\n\n[1] Akata, Zeynep, et al. \"Label-embedding for image classification.\" IEEE transactions on pattern analysis and machine intelligence 38.7 (2015): 1425-1438.\n[2] Weston, Jason, Samy Bengio, and Nicolas Usunier. \"Wsabie: Scaling up to large vocabulary image annotation.\" Twenty-Second International Joint Conference on Artificial Intelligence. 2011.\n[3] Li, Xin, and Yuhong Guo. \"Bi-directional representation learning for multi-label classification.\" Joint European conference on machine learning and knowledge discovery in databases. Springer, Berlin, Heidelberg, 2014.\n[4] Mirzazadeh, Farzaneh, et al. \"Scalable metric learning for co-embedding.\" Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Cham, 2015.\n[5] Yeh, Chih-Kuan, et al. \"Learning deep latent space for multi-label classification.\" Thirty-First AAAI Conference on Artificial Intelligence. 2017.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that this submission is still premature to be accepted to ICLR2020.
We hope the review comments are useful for improving your paper for potential future submission.
Decision: Reject