MULTI-LABEL METRIC LEARNING WITH BIDIRECTIONAL REPRESENTATION DEEP NEURAL NETWORKS

Abstract: Multi-Label Learning task simultaneously predicting multiple labels has attracted researchers' interest for its wide application. 
      Metric Learning crucially determines the performance of the k nearest neighbor algorithms, the most popular framework handling the multi-label problem.
      However, the existing advanced multiple-label metric learning suffers the inferior capacity and application restriction. 
      We propose an extendable and end-to-end deep representation approach for metric learning on multi-label data set that is based on neural networks able to operate on feature data or directly on raw image data. 
      We motivate the choice of our network architecture via a Bidirectional Representation learning where the label dependency is also integrated and deep convolutional networks that handle image data. 
      In multi-label metric learning, instances with the more different labels will be dragged the more far away, but ones with identical labels will concentrate together. 
      Our model scales linearly in the number of instances and trains deep neural networks that encode both input data and output labels, then, obtains a metric space for testing data. 
      In a number of experiments on multi-labels tasks, we demonstrate that our approach is better than related methods based on the systematic metric and its extendability.   
      

Reviews and author responses:

The paper addresses the problem of multi-label prediction.  It proposes a method that uses a co-embedding of instances and labels into a joint embedding space in a way that related instances and labels fall close by and unrelated ones fall far away.  For this purpose, embeddings from input space and label space to a common space are learned from training data. At the prediction time, KNN to the embedding of the test instance in the co-embedding space is used to predict relevant labels.
Featurized (attributed)  labels are potentially considered, which can facilitate incorporating label dependence and generalization over unseen labels.

Main shortcomings:
- The novelty of the work is limited. Ideas introduced in this work are present and being investigated in literature for a while now, leading to remaining limited contribution for the paper.  Related work on joint embedding, co-embedding, label-embedding, and zero shot learning seem to be neglected totally. For example, the paper lacks awareness of, citation to and comparison with related work such as [1-5].
- Presentation of the paper can be highly improved. There are several grammatical and writing problems in the paper.
Formulation can also benefit from  improved presentation. See for example Eq (1).
- Technical arguments are not all well founded. For example, the scalability claim in the abstract  of the paper seems to refer to "prediction" time complexity being linear in the number of "training" examples, which is not actually fast.

In summary, based on the above reasons, I vote for the paper to be strongly rejected.

[1] Akata, Zeynep, et al. "Label-embedding for image classification." IEEE transactions on pattern analysis and machine intelligence 38.7 (2015): 1425-1438.
[2] Weston, Jason, Samy Bengio, and Nicolas Usunier. "Wsabie: Scaling up to large vocabulary image annotation." Twenty-Second International Joint Conference on Artificial Intelligence. 2011.
[3] Li, Xin, and Yuhong Guo. "Bi-directional representation learning for multi-label classification." Joint European conference on machine learning and knowledge discovery in databases. Springer, Berlin, Heidelberg, 2014.
[4] Mirzazadeh, Farzaneh, et al. "Scalable metric learning for co-embedding." Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Cham, 2015.
[5] Yeh, Chih-Kuan, et al. "Learning deep latent space for multi-label classification." Thirty-First AAAI Conference on Artificial Intelligence. 2017.

This paper aims to solve multi-label problems via learning a share embedding space for instances and its label sets. Specifically, the author considers deep neural networks F(x) as an encoder for the instance (either raw input or features) and a shallow MLPs G(y) as an encoder for the label outputs. In the training stage, the instance embedding and its label embedding are forced to be close. An additional constraint is instances with different labels should be far from each other. After training, the inference can be done in the embedding space by looking up the labels of the query’s kNN instances. 

Metric learning for multi-label problems is not new, many works have been proposed such as [1,2]. Using deep neural networks for multi-label problems is also not new, see [3,4]. Thus, the novelty of the proposed method is rather limited. The interesting part is the constraint of Eq(11), where kNN need to be updated whenever the instance encoder model F(X) is changing during the optimization, which is very expensive for large-scale application.
 
[1] Sparse Local Embeddings for Extreme Multi-label Classification, NIPS 2015.
[2] Learning Deep Latent Space for Multi-label Classification, AAAI 2017.
[3] Deep Learning for Extreme Multi-label Text Classification, SIGIR 2017.
[4] X-BERT: eXtreme Multi-label Text Classification with BERT, ArXiv 2019. 


This paper presents a metric learning approach for the multi-label classification problem. It basically maps the input features and the output labels into the same space and then uses k-NN to find the closest labels for each inputs. During training, it minimizes the squared Euclidean distance between the input embedding and label embedding. In the experiments, some image datasets and text datasets are used to compare with several multi-label learning algorithms. 

The proposed method is clearly presented in the paper. However, I have several concerns.

1. The proposed method is straightforward and lacks of significant novelty.
2. The datasets are very small scale in terms of both sample size and label size. 
3. The comparing methods are a little bit out-dated. I would also suggest to compare directly with the naive one-versus-all method using deep learning extraction models. 
4. There are a few typos. For example, on Page 1, "because it wide application", "but cannon deal", "may lost key information"; on Page 2, "an bidirectional", "mapping of outputs to metric space are"; on Page 3, "a scalable models", etc.. Besides, Eq (1) should be written in a more formal way. 

Overall, I vote for rejection and the main reason is the lack of novelty. 

Meta-review: All reviewers agreed that this submission is still premature to be accepted to ICLR2020.
We hope the review comments are useful for improving your paper for potential future submission.

Decision: reject