Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduces a new algorithm for extreme multi-label classification. The focus of this paper is on being able to handle short documents with all experiments focussed on matching user queries to advertiser bid phrases. The key novelty in this paper is to split the labels into two buckets: head labels and tail labels. The model learns word embeddings on the head labels + a classifier on top of those embeddings. For the tail labels, the embeddings from the head labels are used as the input for another classifier which is trained on only the tail.

My thoughts on the paper:
- I really like the paper writeup; its succinctness (in most sections, some comments below).
- Small nit: why choose head labels as 0.1L? I would have expected a more natural choice to be based on frequency?
- Figure 1 (and explanation in section 3.1): I understand the head setup completely (although it seems to be missing the ANNS). For the DeepXML-t part, I am not very clear from the picture nor the explanation how the ANSS feeds into the weights and how that leads to the label \hat{y}+{clf-t}?
- Section 3.1, DeepXML-h section: the bit after "Additionally ..." until "DeepXML-t" section is unclear. I think this needs to be explained better.
- I might have missed it but is PSP metric defined explicitly somewhere?
- The experiment section contains lots of baseline comparisons; unfortunately not all on publicly available datasets.
- The paper uses a large number of aconyms which are defined sometimes after their first use, sometimes never: i.e. PLT, ANNS.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper considers extremely multi-label learning (XML) where the label size is very large . It aims to improve the accuracy as well as the scalability of XML algorithms, especially for short text inputs. The accuracy for XML with short text inputs can be significantly improved using deep learning representation than using TFIDF features. This paper proposes several tricks to handle the issue for efficiently learning both neural network parameters and classification weights of extremely large number of labels. The proposed method borrowed ideas from several previous literature, and is mainly based on SLICE, where a pre-trained fixed deep learning representation for the inputs are used with ANNS graph for labels to improve the scalability. The main difference is that instead of using a fixed input embedding, the proposed method learns the word embedding parameters via a set of head labels. The remaining labels are then trained using SLICE with fixed word embeddings from the learned word embedding model. 

Overall the paper tackles the problem well. And the empirical results show improved results. However, I don't think this paper is ready for publication due to the following concerns. 

1. My main concern is that the proposed method seems to be a combination of a number of tricks. This makes the overall algorithm/model very complicated and introduces a lot of hyper-parameters, for example, head label portion, L-h', c, beta, s neural network hyper-parameters and so on. Hence, it will be hard to be used in real applications.  

2. Another concern is about the experiments. 
    a. The most significant improvement of the proposed method over existing method happens in the private dataset, Q2B-3M, which can't be reproduced.  
     b. On the public datasets, DeepXML seems to show good results on small datasets, WikiSeeAlsoTItles-350K and WkipediaTitle-500K, while on large datasets, DeepXML performance is close to the existing methods. 
     c. The largest label size in the experiments is 3M. SLICE can be scaled up to 100M labels. 

3. The writing and the organization of this paper needs to be improved. 
    a. Some notations are not clearly defined. For example, L_h in Line 6 and X' in Line 9 on Page 5. 
    b. Several method names are not defined. For example, DeepXML-fr, AttentionXML-l, Sliced-CDSSM, DeepXML-SW, DeepXML-f. I have to guess what they are. 
    c. The last two paragraphs on Page 4 seems to be related work, while there is a section called "Related work".

Other minor comments:
1. It seems it is not stated how Beta is set. 
2. I am wondering if it's true that the shorter the input text is, the better improvement over non-deep-learning methods DeepXML can achieve.
3. In the first paragraph of Sec 3.1, it is mentioned "clustering just the top 3 labels into 300 clustering". Why choose 3 and 300? Are these numbers used for all datasets?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper presents a deep learning method for extreme classification and apply it to the application of matching user queries to bid phrases. The main idea is to learn the deep models separately for head and tail labels. Since there is abundant training data for the head labels and transfer the learnt word embeddings for the network for tail-labels, while keeping the head network fixed.

On the positive side, given that there are relatively few successful approaches for deep learning in extreme classification, the main contribution of the paper is towards making an attempt towards this goal.

However, since the paper is mostly empirical in nature and based on algorithmic implementation, the experimental evaluation does not seem quite convincing for the following reasons :

1. Firstly, all the datasets used in the work are private and not publically available. This is quite in contrast to all the various works in this community which use publicly available data and codes.

2. It is not clear why the authors did not to evaluate their approach on the "standard" datasets from the Extreme Classification Repository http://manikvarma.org/downloads/XC/XMLRepository.html. Though it is clear that the focus of the paper is on short text classification, but it is important to evaluate what happens when that is not case. Does the method also works well in longer training/test instance, as there is no immediate reason for it to not work well in that case. Or is it that other methods outperform in that scenario.

3. The performance of the proposed method DeepXML is not significantly better than Parabel. For instance,  on two of the four datasets in Table 1, it gives same predictive performance with order of magnitude less training time and much lower prediction time. This begs the question of utility of proposed approach.

4. Related to above is impact of data pre-processing for different methods. DeepXML seems to use tf-idf weighted word embeddings while other methods use simply BoW representation. It is possible that using simialr data representation or combination with bigrams/trigrams might also improve performance of Parabel and DiSMEC, since it is known from short text classification that using this info can improve performance (https://www.csie.ntu.edu.tw/~cjlin/papers/libshorttext.pdf).

5. Lastly, it is unclear why AttentionXML and DiSMEC are shown to be non-scalable for Amazon3M when they have shown to be evaluated on the bigger version of the same datasets in other works. Also, it might be noted that AttentionXML in the latest version can be combined with shallow trees for efficient scaling.
review_title: Official Blind Review #1

Meta-review: The paper proposes a new method for extreme multi-label classification. However, this paper only combine some  well known tricks, the technical contributions are too limited. And there are many problems in the experiments, such as the reproducibility, the scal of data set and the results on well-known extreme data sets and so on. The authors are encouraged to consider the reviewer's comments to revise the paper.
Decision: Reject