Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the extreme multi-label text classification problem and established new state-of-the-art (SOTA) results with the proposed method (namely CascadeXML), which significantly simplified the multi-stages training of the previous SOTA model, XR-Transformer. CascadeXML enables end-to-end training with two key insights: (1) leverages intermediate representations of Transformer layers for learning XMC sub-problems of the label tree (2) fine-tuning at the full resolution of label space. 
strengths_and_weaknesses: ### Strength
- Leverage intermediate representations for multi-resolution learning of XMC is novel
- Paper writing is clear and easy to follow
- Empirical results are quite promising 

### Weakness
- Lack of ablation study for the choice of intermediate layer of Transformer embeddings 
- Lack of ablation study on loss functions

questions: ### Major Questions
- How to decide the mapping between Transformer intermediate layer embedding \theta^{(t)} and the level of the label tree? For example, if the depth of a label tree is greater than 12 layers (say 15 layers), where the BERT-base model has 12 layers of Transformer encoders, then the mapping is not 1-to-1?
- For Eq(4), It seems to be that the loss function is binary cross entropy. Any comments or justification on that? Did you compare it with other loss functions such as max margin hinge loss or ranking loss?
- Any thoughts of why the inference time is faster than other baselines in Table 3?
- For the CascadeXML trained by sparse-dense concat features, do you retrain the label tree or is it the original label tree constructed by PIFA of sparse TFIDF input features?
- It seems like the "coarsening-op" R_{\uparrow}^{(t)}(S) (latter one in Eq(1)) is never used in the main paper. Maybe you can consider removing it for more simpler presentation and ease of notation?

### Minor Comments
- Line 111: we often write t_k \in \mathcal{V} in convention.
- Line 181-182: For Algorithm 1, the short list at t+1 layer should be R_{\downarrow}^{(t)} (S_{top}^{(t)}) \union R^{(t+1)}(y). In Algorithm 1, the “refining-op” is missing?
- Line 327&332: merge duplicate bibtex entry [6] and [8], which points to the same paper.
- Line 432-433: update citation bibtex with latest info (Xiong et al., NAACL 2022)
- Line 446-447: update citation bibtex with latest info (Yu et al., JMLR 2022)


limitations: No potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes CascadeXML to solve the extreme multi-label text classification problem. Compared to prior works, the proposed method utilizes features from multiple layers from Bert for different granularity of the hierarchical label tree. The coarser labels with lower scores can be terminated early for more efficient learning. The proposed method is compared with several SOTA methods, including  LightXML and XR-Transformer, on benchmark datasets - Wiki-500K, Amazon-670K, and Amazon-3M.
strengths_and_weaknesses: I am not an expert in text classification, and thus not familiar with most of the related works. My judgment here is mostly based on the merits and the presentation of the paper on its own.

The paper in general is very well written. It explains the main drawbacks of existing works (although I am not sure if there are other works that already addressed these issues) and motivates from the drawbacks to propose CascadeXML. The core idea of CascadeXML is intuitively sound, well explained, and should be easy to reproducible. The authors also include the code in the supplementary.

I did not find too many weaknesses in the paper, but I do have some questions.

1. Does all compared (or the main) SOTA methods based on the same Bert model? This is not clearly stated in the paper. 
2. For early termination of the coarse labels, there are risks of early wrong predictions as well.  Is there any measurement to prevent that? 
questions: Please refer to the weakness section
limitations: NA
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose CasccadeXML, an end-to-end deep learning pipeline for extreme multi-label text classification. CascadeXML leverages the multi-layer architecture of transformer models to handle different label resolutions with separate feature representations. Specifically, for each label resolution in the hierarchical label tree (HLT), features from the corresponding transformer layer are used to predict a shortlist of candidate labels or meta-labels. The authors empirically show that CascadeXML achieves state-of-the-art performance on three extreme multi-label datasets. The computation of training and inference are both faster than the current state-of-the-art transformer-based models.
strengths_and_weaknesses: Strength
1. The approach to leveraging the multi-layer architecture of transformers for multi-resolution learning is interesting.
2. The experiment results are good concerning both performance and computation.
3. The analysis of the impact of label cluster size explains the improvement of CascadeXML well.
4. The paper is well-organized and easy to follow.

Weakness
Major
1. The method requires much more hyper-parameters, such as the corresponding transformer layer and the number of meta-labels of each label resolution.
2. Lack the discussion of the sensitivity of the newly introduced hyper-parameters.
Minor
3. The high/low resolution mentioned in Section 1 sounds ambiguous to me. May replace them with refined/coarse.
4. The definition of \mathfrak{C}^{(t)} and \mathfrak{R}^{(t)} are provided but the definition of \mathfrak{C}^{(t)}(y) and \mathfrak{R}^{(t)}(y) are not.
5. Line 234: Attention -> AttentionXML
questions: 1. How did you decide the hyperparameters, especially the corresponding transformer layer of each label resolution?
2. Are the results sensitive to the hyperparameters?
3. Is there a rule of thumb for deciding these hyperparameters if given a new dataset?
limitations: The authors said the limitations and the potential negative societal impact are discussed in the appendix, but I didn't find them. Some discussion about the lack of theoretical guarantee or the limited search space of the hyper-parameters would be good.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes CascadeXML which can leverage the multi-layered architecture to learn data representation corresponding to different resolutions in the Hierarchical Label Tree as well as fine-grained labels at the level of the extreme classifier. Each layer of the model can perform label shortlisting on a level.

The authors compare the proposed CascadeXML to strong baselines on three XMC benchmark datasets. Experiments show that the proposed method outperforms these baselines, and can achieve faster training and inference.

strengths_and_weaknesses: Strength:

1. The proposed CascadeXML is end-to-end differentiable and achieves better results compared to previous baselines on three benchmark datasets.

2. The proposed method is faster in both training and inference, which is an important aspect of the XMC task.

3. This work is well-written and easy to follow.


Weakness:

1. The idea of dynamic label shortlisting is very interesting, but I wonder how to solve the cases where HLT contains more levels than the layer number of Transformers? Also, for the HLT containing fewer levels, do we have to change the model structure? If the model is trained on HLT-1 containing $n$ layers, can we transfer it to an HLT with $m$ layers where $m \neq n$?

2. Why do the earlier layers in the Transformer correspond to the higher levels of the HLT? More explanations over this may help a bit with the answer to Q1.

3. The proposed method highly relies on the multi-layer structure of the Transformer, which makes it not easy to extend to other frameworks. As mentioned above, the scalability and transferability of this method may be much more limited compared to the baselines adopted in this work.
questions: 1. Can you explain more on Eq(1)? How can we present a shortlist with its descendants, which might be a set of another shortlists? How can we get the corresponding vector representations?

2. If any prediction error happens on a higher level during inference, will this error propagate easily to lower levels?
limitations: The authors have discussed and addressed some of the concerns I have in the paper.
review_title: null

Meta-review: metareview: This paper proposes CascadeXML, which is an end-to-end framework for the task of tree-based extreme multi-label text classification. It extracts the representations from different layers of a BERT model and then maps them to different levels of the hierarchical label tree (HLT). 

The proposed method shows strong performance in  P@k on benchmark datasets with an  improved efficiency during inference compared to other state-of-the-art methods including XR-Transformer and LightXML. Two of the reviewers pointed out the lack of ablation studies for the choice of intermediate layers of the BERT model and the mapping between those Transformer layser to the HLT layers. The problems were addressed in the updated version of the paper during rebuttal and the reviewers increased their scores as a result.

Given that 3 out of the 4 reviewers give a score of 7,  the recommendation is to accept the paper.

confidence: Certain
Decision: Accept