Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: 1. This paper analysis the weaknesses of current HMC benchmarks: the statistical and structural properties not adequately represent the problem space and can confound performance analysis among competing methods. 

2. This paper propose a toolkit – Ontologue, to declaratively construct and analyze a subset from DBpedia as a benchmark for HMC problems with indirect control over structural and statistical features.

3. This paper perform comparative analysis of the new benchmarks against the old benchmarks, and evaluate the performance of state of the art models using four new benchmarks derived by Ontologue.
strengths: 1. The proposed Ontologue can provide customizable benchmarks.

2. This paper try to evaluate the proposed method from different aspects.
weaknesses: 1. In the introduction part, the paper mentioned "We analyze 20 HMC benchmark datasets...", but in the results part (both in table 1 and figure 2), there are 6 benchmarks are included.

2. This paper compared the four new benchmarks derived by Ontologue with 6 existing benchmarks in section 5 Comparative Analysis of Benchmarks. But the domains of the four new benchmarks are different form the 6 existing benchmarks. Therefore, it is not clear whether the difference of the performance is caused by the benchmarks themselves or caused by the classification problems (domains).

3. In table 3, only one method is used to evaluate the effectiveness of contextual information. It is better to include more methods to evaluate if the contextual information can help to improve the performance for more methods.


correctness: Okay
clarity: Okay
relation to prior work: I am not quite sure but it seems different from prior work.
additional feedback: Please refer to the weak points above.
review_title: NeurIPS 2022 Track Datasets and Benchmarks Paper241

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The authors propose a tool for generating hierarchical multi-label classification benchmark data sets, discuss the issues with currently available solutions, generate their own example benchmark datasets and provide baselines for them using well-established methods. 
strengths: As a person who also spent a significant amount of time on tackling problems with established multi-label data sets, their label distribution, how some of them are mostly single-label or multi-class problems and should not be evaluated in the multi-label field, I found it very satisfying that the authors provide a view on shortcomings of data sets that are currently used. Shortcomings that may confuse the modeler to take artifacts for progress. It is a strong contribution of the work, although in the appendix. Comparing the figures 3 and 4 in the appendix is the main reason for accepting this paper. 

Additional strengths of this paper:
- taking the application context and looking at triviality or impossibility of labels in Figure 2.
- discussion of the HMC vs few-shot learning
- using a selection of widely used methods for baseline establishment


weaknesses: A strong weakness of the contribution is that at the time of review, and this review is a little late, the code and data are still not available in the GitHub repository. It is therefore impossible to assess whether the authors actually deliver on what they write. 

Another weakness is that with DBpedia labels, quality is uncertain, a lot depends on which field one chooses and the choice of the hierarchy depth. The problem is especially with recall, as the taxonomy on Wikipedia evolves, it takes time for articles to be reclassified by editors, not to mention new articles to be added to specific labels. The visual analysis method introduced in the paper, helps to identify deterioration as it starts to happen, but it still does not remove the problem that there will be cases of topic + depth combinations where the DBpedia data is not going to allow generating a good HMC data set. I would've welcomed a limitation section discussing the limits of DBpedia in this regard.
correctness: It is impossible to verify the soundness of the data set, as it was not made available. The described approach is sound to the limitation of DBPedia quality, but should produce a correct data set on a quality-checked data source.
clarity: Yes.
relation to prior work: The paper differs from prior work by 
additional feedback: The need for a toolkit like Ontologue is clear. Should the code delivered by the authors turn out to be well-documented, easy to use, abstract enough to allow running it on other data sources than Wikipedia/DBpedia, this would be a significant proposition. It is unfortunately impossible to assess at the moment of review as the code is not available.

In my opinion, this paper could be located within the top 15% papers, and that is my verdict in the end, but I make it by trusting the authors that they actually deliver on what they write, without being able to verify that.
review_title: A long-needed toolkit, with a high quality discussion of current limitations, hopefully turns out not being limited to DBpedia.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The main proposal in this paper is a way for filtering the dbpedia dataset so that it can be used for a hierarchical classification task, called Ontologue. Ontologue allows control over the depth of the tree on the label space, the number of examples in the specific label, and the topic of the dataset based on the choice of the starting node. The paper shows the usefulness of Ontologue by generating 4 different datasets and running baseline and state of the art HMC models on the datasets. The paper also presents the argument that existing HMC datasets present 2 artificial problems, with some examples having a multi-label classification and another sizable set of examples having a few shot learning problem. They also note that existing datasets dont text data associated with labels which may be leveraged by modern few shot learning methods. The datasets which one can generate with Ontologue remedy this.
strengths: - The Ontologue framework seems useful for furthering methods for HMC, it is also nice that it comes with a framework to visualize the generated datasets.
- The experiments presented demonstrate that the datasets generated with Ontologue offer a meaningful challenge to models (Fig 2).

weaknesses: - The motivation of the paper seems to be based too much on a specific model for HMC developed by the authors in prior work (Surj). The paper uses Surj to demonstrate that specific desirable properties are fulfilled by the dataset (eg the benefit of text associated with labels - lines 280-287) but also argues that these properties are important for leveraging modern few shot learning methods (lines 50-55). This seems like a circular argument which makes it seem like the datasets built here were built to demonstrate the capabilities of a model with focus on a realistic task somewhat lost. I would encourage broader discussion and framing of the challenges presented by the dataset beyond those specific to one model.

- The paper also repeatedly (lines 79-81, line 157) argues that the proposed framework generates datasets which are more "realistic" than existing benchmarks. Given that the existing datasets are realistic datasets and the proposed allows one to generate synthetic datasets this claim is strange. What makes a dataset "realistic"? For example, what makes nodes with multiple parents "more realistic"? I would encourage greater discussion of what makes the generated datasets more realistic (quantitative comparisons to other hierarchical datasets (eg perhaps ones not used for HMC?) and perhaps more grounding in the literature) or discuss the implications of the proposed framework which generates challenging synthetic (yet challenging) datasets -- synthetic + challenging wouldn't be a slight on this work.

- The exact nature of the task proposed in the paper is unclear. What exactly is the input "abstract" and what do the labels associated with it say about the input?
correctness: The claims seem somewhat correct (discussed in weakness above). Additional questions:

1. Given that other datasets for HMC are not text datasets (eg protein function prediction or medical image annotation), what is the extent to which models developed with Ontologue will be applicable on these other datasets? Eg would it be possible to obtain meaningful text associated with labels in these datasets? I would encourage discussion of the framework to other application domains with realistic HMC problems.

2. "We intentionally designed Ontologue to limit explicit, low-level control over benchmark properties to discourage the creation of cherry-picked datasets that would invalidate generalizability claims." - what exactly are these design elements? What is an example of cherry picking in the case of Ontologue which is prevented by its design? What are the benchmark properties?

clarity: Paper is well written.
relation to prior work: Relation to prior work is mostly clear. Exceptions:

1. "Some of these HMC benchmark datasets are arguably more difficult than standard few-shot benchmark datasets" -- I would highly encourage the discussion of specific existing few-shot benchmark datasets in this paragraph.
additional feedback: Line 273: "We see that unlike prior results, which showed that Surj was superior on all but three datasets, there is more diversity in the ranking." -- what are the prior results? Please summarize if this is referencing results from other papers.

Figure 3: Why is performance influenced by tree height?

How does the proposed dataset connect/differ from entity linking datasets such as [MedMentions](https://github.com/chanzuckerberg/MedMentions) or [TypeNet](https://aclanthology.org/P18-1010.pdf)? 

**Post author response:**

Dear authors, thank you for your responses. I think these are reasonable and have updated my score. I would also encourage discussion of entity linking datasets in Sec 2 perhaps, especially since you also run Ontologue on MedMentions.
review_title: Framework for generating HMC datasets for specific kinds of models.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: 1. The authors develop a toolkit Ontologue, which allows to extract HMC datasets from DBedia, which meet certain pre-defined conditions;
2. The authors extract four datasets and evaluate performance of SoTA methods using them; 
3. The authors have conducted an extensive analysis of existing HMC benchmarks and revealed their limitations and challenges;
4. The paper presents several novel and useful analyses of the HMC datasets with respect to descriptive statistics, domain and structural diversity,  label distirbution and Global Hierarchy Violations.
strengths: 1. HMC is an important problem. This paper increases our understanding of HMC methods by inroducing data for method benchmarking;
2. The Ontologue tool allows to extract such datasets from DBPedia, which meet experimental needs. At the same time the Ontologue tool prohbits cherry picking behaviour;
3. The four benchmarks create a reasonable problem space for HMC methods.
weaknesses: 1. [minor] Along with four datasets in English, the authors could extract one or two datasets in other languages;
2. [minor] At the moment the GitHub repo of the paper is empty. The authors are requested to update the repo. 
3. [minor] Line 283: For reproducability reasons the authors are requested to state explicitly which BERT model was used.
correctness: The experiments and the datsets are designed in a sound manner.
clarity: The paper is well-written. It is easy to follow. 
relation to prior work: The paper is grounded in previous works on HMC.
additional feedback: Please consider a few minor issues, described above.
review_title: First version of an official review of the paper "Ontologue: Declarative Benchmark Construction for Ontological Multi-Label Classification"

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: 1. This paper analyzes the weaknesses of the existing 20 HMC benchmark datasets.
2. This paper proposes a toolkit called Ontologue to construct customized benchmarks.
3. The authors derive four new benchmarks and evaluate the performance of state-of-the-art models.
4. The authors perform a comparative analysis of the new benchmarks against the old benchmarks.
strengths: 1. This toolkit takes the application context, distribution of labels and data, and hierarchy structure into consideration. 
2. It can be used to produce a customized benchmark based on any topic in Wikipedia, which contributes to the accurate evaluation of the effectiveness of the HMC methods. 
weaknesses: 1. The figures in this paper are not clear enough. Please modify them.
2. This proposed system called Ontologue is not open so far, and the proposed datasets in the GitHub repository are empty.
correctness: The claims made in the submission are correct. The datasets proposed in this paper are constructed soundly.
clarity: This paper is well written.
relation to prior work: Yes, this paper clearly discusses the existing benchmarks' weaknesses, and comprehensive analysis of the new benchmarks against the old ones is presented.
additional feedback: 1. Can the authors give an example about the use of the toolkit? What are the input and output of the toolkit? Please explain the whole generation process step by step.
2. In the right part of Figure 3, what do the X-axis and the Y-axis mean, respectively? How to calculate the distribution of labels in this figure? What is the purpose of demonstrating the distribution of labels over tree depth?
3. The terminology “DAG” is first used without its full name in line 69.
review_title: In this paper, the authors analyze the weaknesses of multiple HMC benchmark datasets and a toolkit Ontologue is designed to construct benchmarks for HMC. The analysis is comprehensive and this toolkit is beneficial to evaluate the performance of HMC methods correctly.

Meta-review: metareview: This paper proposed a benchmark for hierarchical and ontological multi-label classification. Most reviewers find that the paper is interesting and reveals some important shortcomings of the existing multi-label classification datasets. During the rebuttal period, the authors' response addressed most concerns raised by reviewers. I recommend acceptance.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept