Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Extreme classification (or deep information retrieval) has been a popular research field that matches an input text (query) to a label that often is also a text. This paper focuses on a sub-field where both the input text and the label text are short. The proposed training algorithm is a data augmentation algorithm that generates similar examples to the observation based on the label to label similarities. Authors claim this new data augmentation technique was able to improve significantly on benchmark datasets.

strength_and_weaknesses: **Strength**
* Since Gandalf is a pure data augmentation technique that can be applied to some set of algorithms easily. (however, it’s a bit arguable for some other sets of algorithms that can do similar techniques more organically. see weakness #1).
* Authors try to set up a reasonable data-augmentation baseline, LabelMix, influenced by the popular “Mix-up” technique.

**Weakness**
* Selecting k most confusing examples via label similarities is not novel. It is sometimes called “hard-negatives” that is widely adopted in this field (for example, DPR [1], RocketQA [2] and ANCE [3]). The main difference is to use dense label space or to use the label relationship graph (they are quite interchangeable, but the dense space has nicer properties). It is also partially observed in Table 3 as using the soft-labels from the label graph further drastically increases the performance over hard labels.
* Defining the label graph solely based on the label co-occurrences might have its limitations because it does not use any textual understanding of each label. Commonly, the dense label space is trained jointly with the actual retrieval task [1,2,3] and hence captures deeper understanding of the label to label similarities based on both bipartite co-occurrence relationships and textual understanding. Moreover, solely depending on co-occurrence relationships can be very noisy when we have large label space (e.g. thousands of millions or billions).
* The work was not evaluated against common dense retrieval algorithms such as [1,2,3].

[1] Karpukhin, Vladimir, et al. "Dense passage retrieval for open-domain question answering." arXiv preprint arXiv:2004.04906 (2020).
[2] Qu, Yingqi, et al. "RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering." arXiv preprint arXiv:2010.08191 (2020).
[3] Xiong, Lee, et al. "Approximate nearest neighbor negative contrastive learning for dense text retrieval." arXiv preprint arXiv:2007.00808 (2020).

clarity,_quality,_novelty_and_reproducibility: **Clarity**
* The paper can be improved on their clarity on how their proposal is presented. It is quite unclear how the actual training objectives are defined (e.g. loss functions) and how the newly generated examples are integrated into the objective.
* Algorithm 1 should be written in pseudo-code for those who do not have prior Python knowledge.

summary_of_the_review: The paper does not convey a significant novelty in this field and was not sufficiently compared to many core techniques in this field.

-----
Post rebuttal response:

I thank authors to provide detailed feedback. I also read other reviewers' feedback and concluded this paper does require further development to contain further contribution in this field. I still concerns the novelty of this paper (similar to Reviewer g2YZ).

Here are some detailed points that the author replied.

> Gandalf is only a method to leverage Label-Text/Features as data points whose ground truth labels are unknown.

This actually limits the novelty of the method. Gandalf's novelty is limited in introducing a heuristic method to generate additional labels. 

> Contrast to Hard Negative Mining 

Thanks for the detailed description. I wonder then what's the actual benefit of using Galdalf's soft-labels. We often found strong negatives that are closer to the decision boundary helps achieving more discriminative models especially when the label space is large (XMC). Wouldn't it be more beneficial Galdalf to focus on closer but not positive labels?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper present two data augmentation strategies, Gandalf and LabelMix, that adds label features or the instance-label interpolations as training instances in extreme multi-label classification (XMC) training. Empirical results show that the proposed Gandalf method is able improve the performance of various XMC models on the benchmark dataset where label features are present.
strength_and_weaknesses: **Strength**

* The proposed method Gandalf shows relatively large improvement over 5 XMC models across 4 different XMC-LF dataset.

**Weaknesses**

* The proposed method, Gandalf, is rather straightforward. The core idea seems to be just applying normalization and some thresholding on top of the LCG constructed in ECLARE.
* It is not clear to me how the Gandalf and LabelMix are related to each other. They seem to be two independent data augmentation tricks and I failed to see the necessity in introducing LabelMix in this paper.
* In Section the authors claim that LCG is better than Label co-occurrence matrix but both are not as good as Gandalf which adopts a heuristic thresholding after row-wise normalization. However, I didn't see any empirical study that compares these three settings. It would be more convincing if there are some label-to-label augmentation baselines in the comparison.




clarity,_quality,_novelty_and_reproducibility: The paper is well presented and easy to follow. However, as mentioned above, the main method is a straightforward extension of existing work thus the novelty is limited. I have no concerns for reproducibility as long as the authors plan to release the code.
summary_of_the_review: The paper present two independent data augmentation tricks for the XMC-LF applications. Apart from the limited novelty concerns mentioned above, I think the topic of this paper is more related to a data mining venue rather than ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose to conduct graph induction for data augmentation for extreme classification based on label features. Specifically, they capture label-label correlations based on a label-correlation graph and use label features as data points. They also leverage label features to have query-label interpolation and then sample labels for mixing up augmented labels. The experiments based on four extreme classification benchmarks with label features demonstrate that the proposed method can enhance some extreme classification models. 
strength_and_weaknesses:  Strengths
* Simple and effective approaches that potentially can apply to arbitrary methods.
* Publicly available benchmark datasets for reproducibility.

Weaknesses
* The authors “propose” LabelMix as a method as the “baseline”, but the more authentic approach should be to compare with published state-of-the-art works in data augmentation for XMC tasks.
* Experiments only cover datasets with only short text (i.e., titles), which are comparatively limited in the real-world settings. The authors should consider conducting experiments on more comprehensive datasets with general information, such as LF-Amazon-131K, LF-WikiSeeAlso-320K, and LF-Wikipedia-500K.
* Although the proposed techniques are potentially able to be applied to arbitrary methods, some methods like AttentionXML, XR-Transformer, and SiameseXML++ are not experimented with either LabelMix or Gandalf. Especially when vanilla XR-Transformer and SiameseXML++ outperform LabelMix and Gandalf in certain cases, I cannot see why the authors do not demonstrate that the proposed techniques can further improve the best performance.
* The proposed techniques require label features, but in real-world settings, it usually lacks label features. 

clarity,_quality,_novelty_and_reproducibility: * For clarity, the paper needs to be polished, especially for the organization. The flow is not easy to follow, especially in the part “proposing the baseline LabelMix”. 
* The idea is interesting and kind-of novel, but the technical quality of experiments are unsatisfactory.
* For reproducibility, the authors provide some implementation details.

summary_of_the_review: In sum, I would recommend “3: reject, not good enough” as there are so many flaws in the experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studied extreme multi-label classification problems where labels short text descriptions are available. The author proposed Gandalf, a data-augmentation technique that augmented the label text as input data, and used the label correlation graph as its soft-labels. The proposed Gandalf technique was applied to various XMC models, and showed consistent improvement on four XMC benchmarks.
strength_and_weaknesses: *Strength*
- The data augmentation techniques (Gandalf and LabelMix) are novel in XMC
- Promising performance gain

*Weaknesses*
- Lack of ablation study to back the effectiveness of certain component in Gandalf
clarity,_quality,_novelty_and_reproducibility: - Clarity: The writing is clear
- Novelty: The idea of data augmentations are novel for XMC community
- Reproducibility: No code provided, hence no data point to verify the reproducibility

summary_of_the_review: This work presented two novel data augmentation techniques, Gandalf and LabelMix, for the XMC problems. When applied to various downstream XMC  models, both Gandalf and LabelMix demonstrated consistent improvement over the baseline. Overall writing and presentation also are clear to follow.

The paper quality can be improved if the following technique questions are addressed.
- Choice of label-to-label graph: Gandalf with Co-occurrence graph instead of LCG?
- Sensitivity of hyper-parameters such as delta in Algo 1 and the design choice of y_tilde in Eq(9) versus Eq(8).
- Does Gandalf enjoys larger performance gain on tail labels (e.g., label count < 5)? Can you plot the performance gain in y-axis, and the groups of label freq in x-axis?
- I suppose Gandalf and LabelMix are complementary to each other? Can we apply both?
- Can you apply Gandalf/LabelMix to two-tower models (e.g., DPR/ANCE) and do the evaluation on both XMC and IR benchmark datasets?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a new approach for extreme multi-label classification based on data augmentation. The authors considers a specific instance of the problem where objects are short texts and the labels are associated with short textual descriptions. The experimental results are promising. 

The underlying idea is simple, but it does not mean that it should not be considered for publication. Nevertheless, the reviewers had found several problems and three of them recommended to reject the paper. The authors were able to defend some of the critical comments. However, the consistency of the ratings shows that the paper needs to improve in the presentation layer in order to eliminate any confusions.  

The authors should make it more clear how the method is different from the existing ones, what is their contribution and what is the relation between both methods introduced in the paper. Also the empirical studies could be improved as the current ones seem to be designed to favor the introduced method.

On the other hand it might be that the simple augmentation approach is the best one can do for this specific problem. In such a case, the paper would need to be positioned differently. It should rather discuss the problem than promote a method.
justification_for_why_not_higher_score: The paper received rather too strict scores, but certainly it needs to be improved to eliminate any confusions. Therefore, it has to be rejected.
Decision: Reject