Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose an approach for extreme multi-label classification  that is based on a transformer network and leverages a reduction approach that breaks down the problem to smaller ones via a hierarchical approach that constructs a label tree. The authors use both the representation of the transformer as well as classical TF-IDF ones as the final one for the model. The approach is validated in 6 large datasets and compared with other state-of-the-art approaches.
main_review: Using tree structures for extreme multilabel classification is not new as an approach. The authors can refer to approaches like the ones developed by Langford and team (strangely enough these works are not cited). So, reductions are not new.

What it seems to be new is the multi-resolution learning. The authors have some results on the size of the clusters but is not very clear how much this way of training helps. I think what is missing here is a deeper exploration in terms of experiments. For, example is interesting that the TF-IDF features alone do not show variation based on the cluster size while DNN ones yes. The authors should put some more effort here to better understand why this happens.

Improvements are rather small, even though the proposed approach improves in most of the cases. Training time is greatly improved though.
We need to be in a really large output space to see a good difference in the metrics. It would be nice if the authors could elaborate a bit more on this.

Would it be possible also to have the results of Table 2 with/wo TF-IDF features? 

Why the authors did not try to compare with simple concatenations of TF-IDF and aggregations based on a pre-trained language model or even a word2vec approach.

Is after-all the use of the simple statistical features that helps to improve the performance? 
limitations_and_societal_impact: Still the approach is expensive and to be honest my take away if that TF-IDF is still a powerful scheme.

No negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors address the problem of fine-tuning transformers models for extreme multi-label text classification (XMLTC). Even with pre-trained transformer models, fine-tuning it to the task of predicting for large label space is computationally expensive and requires significant GPU resources. The authors propose a new routine to train the X-Transformers-like model that significantly reduces that computational cost. The routing resembles AttentionXML learning routine, where one model is trained by tree level to predict meta-labels at that level. Training starts from the top level of the tree, and the model for the next level is initialized with weights of the final model from the previous level. Here underlying transformer model is first tuned on the task of predicting the same meta-labels. It starts with tuning using meta-labels from the top level and then continues tuning layer by layer. In addition, the authors apply a few additional techniques like cost-sensitive training (positive meta-labels are weighted by a number of relevant labels they consist) and bootstrapping of output weights for meta-labels/labels based on current transformers embeddings. After tuning, new label-partitioning is constructed, and the model is retrained with fixed transformer weights. The authors evaluate the attractiveness of their method on six popular benchmark datasets. New routing leads to a significant reduction in training time at the same often achieving better predictive performance.
main_review: Strengths:
+ The organization of this paper is good, and it's easy to understand. 
+ The paper has clear motivation.
+ Proposed method significantly reduces the computational cost of training achieving better or similar results to its parent method X-Transformers and SOTA LightXML and AttentionXML. 
+ Reduction of training cost makes XMLC more accessible cost, which is vital for further research and practitioners deploying such models.
+ Proposed training routine should be applicable to other label tree-based approaches methods with transformers and possibly other types of underlining deep models.
+ Solid experimental evaluation that proves the attractiveness of the proposed method.
+ The appendix seems to consist of enough details to reproduce this work and results.

Weaknesses:
- I don't find any serious issue in this work. It seems to me that the scope of contribution is a bit limited since the proposed approach is composed of elements that were already explored by the XC community, but this is a very successful combination. 

Other comments:
* "Nevertheless, multi-resolution learning has not been well-explored in the XMC literature. In XR-Transformer, we leverage the label hierarchy defined by the HLT and train the transformer model on multi-resolution objectives." - I would argue with that statement, as I mentioned in summary, your routine resembles AttentionXML approach. And while it's not directly multi-resolution learning, I would personally acknowledge it here more.
* After reading the appendix (Table 6 to be precise), I realized that the initial Hierarchical Label Tree (HLT) used for tuning is different from the refined HLT (I mean pre-leaf structure). The initial is more "wide", which makes sense since you want to have more fine-grained meta-labels. But I don't find this stated in the main text. Maybe it's worth commenting on it.
* I think that Recursive_Learn procedure could be a bit more readable and general if presented in an iterative fashion, where the whole HLT is already given.
* There is a mistake in Table 4, Row "Wiki-500, X-Transformers-1" is the same as the row "Amazon-670K, X-Transformers-1". 

---

Edit after the authors' response: I think the authors responded well to the reviewers' comments. I would like to keep my rating and recommend this paper for acceptance.
limitations_and_societal_impact: No issues with negative social impact were found.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a coarse-to-fine learning framework to solve the problem of extreme multi-label text classification.  Experiments show the model not only accelerates existing models 20x times but also achieves a slightly better performance on the dataset with an extremely large label set. The techniques are sound and the empirical results are promising.
main_review: Advantages:
1. The paper is well-written and well-organized. The authors include enough background information and clearly illustrate the efficiency-resolution trade-off in this task. Readers without prior knowledge of the XMC problem can easily follow it.
2. The paper is well-motivated. The proposed method is natural, sound, and potentially helpful in other problems.
3. The model achieves state-of-the-art performance on 14 out of 18 metrics on 6 datasets.
4. The authors will publish their codes.
5. Though the empirical improvements over the baseline models are not very significant on many datasets, the great time efficiency makes the model more applicable in many real-world scenarios.


Weaknesses:
1. Since the proposed method incorporates TF-IDF into the text representation and incorporating TF-IDF is often helpful in improving model performance, these improvements may mainly come from TF-IDF instead of the proposed framework. The superiority of the proposed framework would be more convincing if the authors enhance baselines models with TF-IDF and compare the proposed XR-transformer with the enhanced baselines OR compare the unenhanced-XR-transformer with baseline models.

limitations_and_societal_impact: 1. The first figure in the Appendix that overviews the model architecture is relatively hard to understand. It would be very helpful for readers to understand this paper if the model architecture is presented in a more understandable way.
2. To the best of my knowledge, this paper has no potential negative societal impact.
3. The authors do not address the limitations mentioned in Sec. 6 (understand why the coarse-to-fine scheme leads to better performance). The experiments discussed in above weaknesses 1 may help them better understand where the improvements come from.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: To accelerate the fine-tuning procedure of transformer models on XMC, the paper proposes XR-Transformer, which is recursively fine-tuned on coarse-to-fine multi-resolution tasks and cost sensitive learning. Experiments on the benchmarks show that XR-Transformer generally outperforms previous methods and is faster than the competing transformer-based models.
main_review: The paper proposes a novel method to accelerate fine-tuning transformer-based models for XMC, which is inspired by the multi-resolution learning in image generation and curriculum learning. This technique allows for fast fine-tuning and good performance. Whereas concatenating neural features and TF-IDF features to produce the predictions is already discussed in the X-Transformer paper and thus is not new.

The paper is technically sound and well written overall. The method is generally well established. The experiment results seem to be solid. The DNN-based methods use the same inputs for fair comparisons, and the running time is also well reported. Since the results of the DNN-based methods are slightly different from the original X-Transformer and LightXML papers, it could be better to give the number of run times, the standard deviation of the results, and the statistical significance. In addition, there is a slight gap between the training time of ensemble model (Table 3) and single model (Table 4), this might be improved by adding the standard deviation.
 
 **Question for the authors:**
 
1. The text inputs are truncated to 128 tokens, while the documents are generally longer on average. As the training time is significantly reduced, perhaps XR-Transformer could use longer input text to obtain even better results?
2. The authors mention the PSP and tail labels, do the coarse and fine resolutions have connections with the label frequency? Such as higher resolutions focus more on tail labels. Are the HLT and indexing matrices $C$ related to the tail labels? I would like to see more discussion about this.

**Typo:**

- Line 66  model -> models
- Line 247 The **comparison** of ... **are** ...
limitations_and_societal_impact:  The paper should not pose any negative societal impacts.
review_title: null

Meta-review: The paper introduces an improved version of a transformer-based algorithm for extreme multi-label classification. The empirical results show significant speed-ups and very competitive predictive performance. The paper is well-written and the proposed modifications are sound. The reviewers were satisfied by the responses given by the authors. 
Decision: Accept (Poster)