Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the extreme multi-label classification (XMC) problem and proposes a method called ELIAS. 
(1) ELIAS adopts a two-layer index for representing extreme-scale labels, and it learns overlapping cluster partitions by assigning each label to multiple clusters.
(2) ELIAS adopts a two-staged training strategy, where a XMC model is trained with fixed cluster partition generated by k-balanced clustering in the first stage, the cluster partition is generated according to the weighted count of labels assigning to corresponding clusters according to the XMC model trained in first stage, and the XMC model with value of the cluster partition are trained in the second stage.
(3) ELIAS++ adopts a sparse ranker and a calibration module for achieving further improvement compared to ELIAS.
Experiments are conducted on benchmark dataset, where ELIAS achieves better performance compared to previous SOTA methods.
strengths_and_weaknesses: Strength:
+ This paper is well motivated and the proposed method achieves significant performance compared to previous SOTA methods.
+ The writing is good and easy to follow.

Weakness:
- It is not rigorous to say that ELIAS is an end-to-end method, though Eq. 7 and Eq. 8 can be optimized in an end-to-end manner in theory. In practice, the model is trained in a two-stage manner, and the label-to-cluster assignment is generated according to Eq. 10, which is non-differentiable.  
- Experiment results of ELIAS^{(d)} in Table 1 have not be analyzed in Table 3. Is ELIAS^{(d)} corresponds to Stage 1+Stage 2+3 x ensemble in Table 3?
- Lack of time complexity analysis. It will be better to add a discussion about empirical wall-clock time of the two-stage training process of ELIAS. 
questions: How does the loss (i.e., classification and shortlist loss) affect the performance of ELIAS? Is it possible to add an ablation study of $\lambda$?

limitations: None
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses a relevant problem in partition based approach to extreme multi-label classification (XMC), where existing methods have 2 stages – the first stage uses a shallow tree-based index to hard partition the label space and the second stage learns classifiers into labels inside that partition. The challenge is that if all the labels are not inside the partition in the first stage, then they cannot be in the output of the second stage at all. This paper introduces a variant, ELIAS, where the first stage is replaced by a weighted graph based index with soft learnable parameters that are learned together with the final task classification objective. Experimental results on popular XMC datasets show reasonable improvements in precision/recall metrics over existing XMC methods. 


strengths_and_weaknesses: Originality: 
The general idea of training the retrieval stage using an objective (supervised, unsupervised or self-supervised) followed by a classification stage is becoming more commonplace now, with works like REALM for language modeling, and retrieval augmented convolutional networks in computer vision. However, this paper appears to be the first work applying jointly training the retrieval and classification stages in XMC for multi-label classification. The authors have taken a natural next step in evolving this general idea.
Quality:
The paper takes a reasonable approach – motivating the problem, challenges with existing approaches to XMC, proposing a solution, describing the details and testing it on datasets that show the value of this approach. 
Clarity:
The paper is reasonably well written and easy to understand, with sufficient details. While I have not looked at the code, I am glad that they have shared the PyTorch implementation for transparency and enable reproducibility. 
Significance:
XMC is an important problem in a number of domains where the output space of labels is large – many information retrieval problems including search, ads, commerce etc. are in this category. As such the work addresses a significant problem in XMC.

questions: The authors claim earlier in section 1 that ELIAS is better suited to handled multi-modal label distributions. However, this aspect is not really demonstrated in the paper. They hypothesize in the recall comparison section in section 4 that the increased recall might be due to the  multi-modal distribution of popular labels. Do they have any quantitative evidence to back this claim?
limitations: This paper addresses a technical improvement in XMC. As such, I don’t see any significant potential negative societal impact of this specific work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Tree-based methods are amongst the popular approaches for extreme multi-label learning problems. Existing approaches either use a decoupled strategy of training the label classifiers on top of a fixed tree index over labels, or alternate between updating the index and training the classifiers. 

This paper proposes a relaxation of the tree-based index to a directed acyclic graph where each label can belong to multiple clusters with non-zero probability. The probability of label belonging to a cluster is learned together with label classifiers in an end-to-end fashion, and this leads to empirical improvement over using a fixed index over labels.
strengths_and_weaknesses: Strengths

1) Proposed approach allows for learning label-to-cluster assignments in an end-to-end fashion which provides empirical improvements over using fixed label-to-cluster assignments.

2) The paper is well-written and easy to follow. The experiments support the main claim of the paper that jointly learning classifiers and index structure in an end-to-end fashion can yield improved performance over using a fixed index structure. 

Weakness

1) Limited Analysis: It would be interesting to see analysis of the proposed approach beyond final accuracy numbers, for example, how much does the final index structure deviate from the fixed index structure used for stage 1 training. 

2) Scalability: Since the proposed approach learns a two-level index, it is not immediately clear if the proposed approach can be applied to relax tree structures with three or more levels into a directed acyclic graph. This might limit the scalability of the proposed approach to larger industry scale datasets. However, this is a minor weakness as the approach is shown to scale well to and also outperform baselines on one of the largest publicly available extreme multi-label classification datasets (i.e. Amazon-3M).
questions: Questions/Suggestions

Q1) Brute-force-OvA: Does this model consist of a BERT encoder followed by a multi-layer perceptron with L outputs? How much time did it take to train this model? Is it possible that BERT-OvA model also suffers from similar optimization challenges as training the proposed model from scratch? If so, then it is not necessarily a strong upper-bound on performance of these models.

Q2) What is the overall training time for the proposed model?

Q3) How were hyper-parameters such as $\kappa$ chosen? Was k-fold cross-validation used? 

Q4) Why have previous XMC papers such as SiameseXML and DeepXML been not compared with?

Q5) The main set of results in the paper compare precision/recall of the proposed model with other baselines. But it might be interesting to have some results/insights/observations about the kind the structure of the index learnt using the proposed approach and how much it deviates from the fixed index from stage 1 training. Some ideas are:

 5a) In the final index structure, what fraction of labels get assigned to multiple clusters? Is it possible to prune assignment of a label to multiple clusters after training without affecting the accuracy of the model?

5b) In “Recall Comparison” para in Sec 4, authors hypothesize that improved performance for top 2 deciles is due to the tendency of popular labels to have multi-modal distribution. Is it the case that in the final index, popular labels tend to be assigned to multiple clusters and rare labels tend to pick a single cluster? 

5c) How does row-sparsity affect the performance of the model?

Q6) Which dataset is used for Figure 5?

Q7) Instead of using a separate sparse ranker, why is the proposed model not trained with a combination of dense and sparse features for input as done in baseline methods such X-Transformer, Overlap-XMC?
limitations: Yes.
review_title: null

Meta-review: metareview: The paper considers extreme multilabel classification (XMC) and proposes a two-stage retrieval and classification model which replaces the usual initial hard-partitioning with a soft learnable partitioning. The reviewers concur that the end-to-end methodology for jointly training the representation, indexing, classification parameters is novel and leads to notable improvements over the performance of current SOTA XMC methods.
confidence: Certain
Decision: Accept