Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The problem tackled in this paper is that of multi-label classification, where class labels form a taxonomy (i.e., a hierarchy), and the goal is to enforce the predictions to comply with the label taxonomy.  This is a well-known problem in machine learning and it is important as it imposes a more structured learning space, by implicitly enforcing predictions to be consistent with the label taxonomy. The approach taken in this paper is based on box embeddings, where the idea is to view classes as boxes in the space, and apply a certain probabilistic semantics of box embeddings to eventually model taxonomic relations between classes.

main_review: Strengths:

- The paper tackles an interesting problem and it also applies when we have no access to the taxonomy.
- The presentation is relatively clear and self contained.
- The technical details appear sound.

Weaknesses:

- The novelty/originality is limited: box embeddings, probabilistic semantics of their intersections, gumbel boxes, bessel volume, etc are all adopted from existing literature. Propositions and the corollary appear more like observations rather than results. 

- The overall setup is not well-motivated. While I see why box embeddings can be useful for encoding class interactions and their hierarchies (which are in the form of directed acyclic graphs), it is not very clear to me whether one needs a probabilistic semantics as is used in this work. 

- The BoxE paper (Abboud et al, BoxE: A Box Embedding Model for Knowledge Base Completion, NeurIPS, 2020), which is also a box embedding model (proposed for link prediction), is very relevant for this work: It shows that boxes can capture arbitrary relational hierarchies in the space. This is shown for binary relations and therefore it is more general, but it clearly applies to class hierarchies which are only a special case. This suggests that one can model hierarchies using box embeddings in an even simpler way, where box containment implies a subclass relationship (without the probabilistic interpretation). If we already know the class taxonomy, we can even inject this information to the space, i.e., if C is a subclass of D, one can enforce the corresponding C-box to be contained in the corresponding D-box in the space, etc. That is, if the class taxonomy is known, one can provably enforce these using the ideas presented in (Abboud et al). If the taxonomy is not known, then it can still be learned.

- The probabilistic interpretation of boxes may bring in some value, but it is not clear to me whether this is the case, after reading the paper. In fact, the above-outlined approach would achieve the similar goals in a simpler way in my understanding - This may possibly result in an even stronger baseline than the given baselines. Notice that when a C-box is contained in the D-box after training, any prediction for class C will be consistent, since it will also be a D by the space configuration (and there are many possible configurations that can achieve the same thing). 



summary_of_the_review: The paper is well-written and interesting but limited in originality and novelty given works of Vilnis et al., Dasgupta el al. It is an interesting application of earlier findings to the domain of multi-label classification, but it is lacking in motivation for the precise model choices. The comparison with earlier works is limited without which the significance of the work is unclear, which led me to suggest a weak reject.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Most existing multi-label classification methods predict labels without considering natural/latent taxonomic constraints. This work introduces the multi-label box model (MBM), a method for multi-label classification that combines the encoding power of neural networks with the inductive bias and probabilistic semantics of box embeddings (trainable Venn-diagrams based on hyper-rectangles). MBM has some nice properties: they can be trained via gradient descent from data, and box embeddings can be read as calibrated conditional probabilities. Experiments are conducted on seven benchmark datasets with promising results, and results suggest that the label embeddings seem to capture the latent label taxonomy.
main_review: Strengths:
- Most existing multi-label classification methods predict labels that are inconsistent with natural/latent taxonomic constraints.
- This work introduces the multi-label box model (MBM), a method for multi-label classification that combines the encoding power of neural networks with the inductive bias and probabilistic semantics of box embeddings (trainable Venn-diagrams based on hyper-rectangles). This enables MBMs to capture taxonomic relations among labels.
- MBMs can be trained via gradient descent from data.
- Box embeddings can be read as calibrated conditional probabilities.
- The model is naturally more interpretable than other methods for multi-label classification.
- Partial information about label-label relationships can be injected into the model training, improving the model's consistency. This is done via the "label interaction loss".
-  MBM is theoretically grounded, and it can improve taxonomic consistency while preserving or surpassing state-of-the-art predictive performance.
- Multi-label classification is an important problem that has real world applications and is relevant to the ML/AI community. Enforcing taxonomic constraints in multi-label classification is an important subproblem.
- The approach seems quite novel.
- The method is well-framed in the existing literature.
- The method is very-well motivated in theory.
- The baseline models for comparison seem reasonable.
- Evaluations are conducted over 7 benchmark datasets, which provide label space taxonomies which are additionally useful for evaluating the learned taxonomy.
- The evaluation metrics seem reasonable.
- The evaluation results are promising.
- Experimental results suggest that the label embeddings seem to capture the latent label taxonomy without it being explicitly provided.
- the approach can reasonably scale to arbitrarily deep hierarchies as might be present in extreme multilabel classification.


Weaknesses:
- Multi-label classification is a well-studied problem, and there has been work on enforcing taxonomic constraints for multi-label classification in the past. This somewhat limits novelty; however, the approach seems novel, so this is not a major concern.
- The encoder neural network tested is relatively simple: a three-layer MLP; it would be interesting to see whether this method generalizes to more complex neural networks (i.e., can good box representations still be learned with deeper networks on more challenging problems).
- It would be nice to see which results are statistically significant in Table 2.
- Figure 2 is a bit hard to interpret due to the large number of settings.
- Evaluations w.r.t. the taxonomy quality assume the ground-truth taxonomies are perfect, but in practice, there might be a human-bias in how labels are grouped. It would be interesting to see if you could reverse engineer the taxonomy from the embeddings and see if humans agree that the taxonomy makes logical sense and doesn't have many atypical/meaningless relationships detected.

Additional Questions/Comments:
- Is there any disadvantage to using hyper-rectangles over different geometric assumptions (e.g., hyperspheres)?
- Since the method naturally outputs calibrated probabilities and utuilizes taxonomic constraints, I wonder if it is more robust to imbalance problems that tend to plague multi-label classification. Do you have any insights into whether this is true or not?
- The improvements in the metrics seem relatively small when inroducing taxonomy information in the training. Do you think this is because the base model is already capturing model of the latent taxonomic structure and doesn't need the additional guidance.
summary_of_the_review: Multi-label classification is an important problem that has real world applications and is relevant to the ML/AI community. Enforcing taxonomic constraints in multi-label classification is an important subproblem. The proposed approach is a novel solution to this subproblem, and is well-framed in the literature and theoretically-grounded. the paper is relatively well-structured. Experiments and analysis seem sound, and experimental results are proming: MBM is improve taxonomic consistency while preserving or surpassing state-of-the-art predictive performance on several benchmark datasets compared to reasonable baselines.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method using box embeddings to perform hierarchical multi-label classification.

main_review: Strengths:

The idea to use box embeddings for hierarchical multi-label classification seems good and promising.

The paper has mostly been written clearly.

Weaknesses:

The proposed method is a relatively straight-forward application of box embeddings as developed in earlier literature (including e.g. Dasgupta et al., 2020a) into hierarchical multi-label classification.

I am not sure I understand why one would need 'a model that uses box embedding to capture general label-label relationships without the explicit use of label taxonomy'. Would it not be beneficial to explicitly use the label taxonomy? The paper writes that requiring a 'complete label taxonomy either at inference time or both at training as well as inference time, making these models hard to scale to large label spaces'. I would like to see concrete examples of when this really happens. To me it seems that it is relatively easy to learn a mostly complete label taxonomy as the first task and then use e.g. CHMCNN.

No clear enough definition has been provided for the MAP measure. Is the mean taken over the labels and for each label there is an area under the precision-recall curve calculated?

It has not been explained sufficiently why MAP is a good measure to look at. As the paper correctly states, 'it does not take into account inconsistencies in the predicted scores w.r.t. the label taxonomy'. Instead of MAP, one could consider different score thresholds and for each threshold calculate how many labels would get correct predictions. However, indeed, Giunchiglia & Lukasiewicz, 2020 also seems to use the same MAP measure.

The baselines in the experiments could have included HMCN-R and HMCN-F from Giunchiglia & Lukasiewicz, 2020. While it is true that 'HMCN does not try to enforce consistency strongly and focuses solely on predictive performance', it does not mean that HMCN could not be a strong competitor in the experiments.

The origin of the MVM (Multi-label Vector Model) as a baseline has not been explained or referenced. What is the motivation behind this method?

Statistical significance of the differences of MBM over MVM has not been demonstrated through a statistical test. Currently the text about Table 2 states that 'We observe that the predictive performance of MBM measured using MAP is better than that of MVM and MHM on 5 out of the 7 datasets.'

It is not clear whether the label interaction loss has been used as a regularizer on top of the main NLL-loss or in some other way. If as regularizer, then is there a regularization parameter lambda defining how much each loss contributes to the overall loss? How is this parameter lambda tuned?

Supplementary figure 3 shows the trajectories of MAP, but what would be the trajectories of CV and CMAP?

The interpretability claims seem too bold in my opinion. It is still hard to interpret high-dimensional boxes and their overlaps. The experiments would benefit from more baselines.

page 9: 'hyperberbolic' -> 'hyperbolic'

summary_of_the_review: The paper is interesting and results seem to be quite good, but the justification of why availability of the label taxonomy should not be available is a bit lacking. The proposed method is quite a simple step from existing literature, and there are several shortcomings about the text highlighted above. The experiments would benefit form more baselines.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work the authors propose a new model for multi-label classification problems. In their model, the authors represent the labels as boxes and they use neural networks to embed the input datapoints in the same space. Thanks to the box representation, the model achieves higher interpretability, and the authors can inject background knowledge (in the form of taxonomies) into the model to improve the coherence of the learnt labels representation with the taxonomy. 
The model has been tested on 7 publicly available datasets, and evaluated using three different metrics, namely, CMAP, CV and and MAP. 

main_review: I really like the idea at the base of this paper, however I think some revisions are needed. 

1. The authors should change the term “consistent” with “coherent” throughout the paper. Indeed, the authors use the word “consistent” to express the fact that the model is coherent with the taxonomy. This however, has nothing to do with the Fisher consistency and can mislead the readers. 

2. At page 4 the authors first write $Y \in S$ and then $Y \in \mathcal{S}$, did you mean in both cases $Y \in \mathcal{P}(S)$?

3. At page 4, why do the authors use the function $f$? The equation can easily rewritten by substituting $f$ with its value. Also, why was $\delta$ chosen to be equal to $10^{-5}$?  

4. The functional genomics datasets come in two versions, the Funcat version with a tree taxonomy, and the GO version with a forest taxonomy. I see that the authors have used the GO version for expr, cellcycle, derisi and spo, but they have used the Funcat version for diatoms. It would be interesting to have both the GO and the Funcat version for all the datasets. Further, the authors have just used Imclef07a, while Imclef07d is also available. I would recommend to test the model also on Imclef07d. 

5. In order to check the statistical significance of the obtained results the authors should perform the pairwise Wilcoxon test (at least to compare the performance of MVM vs MBM and MHM vs MBM).

6. C-HMCNN does not apply a post-hoc modification, as the max layer is embedded in the network itself. Notice that post-hoc means something that is applied just at inference time (and not at training time).

7. Would it be possible to also do MVM-T? 

8.  In the analysis of the learned embeddings, why weren’t the ROC curves for MHM shown? 

9. In the data pre-processing, what do the authors mean by “The datasets were pre-processed to remove noisy characters, fix encoding issues”?

10. Can you also report in the appendix the final hyperparameters for each model?

11. Why is there such a big difference between the performances of C-HMCNN (as implemented by the authors) and your implementation?
 
12. Why aren’t the results in terms of AU(PRC) of MHM reported? 


Minor comments: 
- It should be C-HMCNN and not CHMCNN
- Add upward arrows in Table 2 next to MAP and CMAP
summary_of_the_review: The paper shows an interesting idea, however to have full acceptance some revisions are needed (see full review).
review_title: null

Meta-review: The paper studies multi label classification problem. Particularly, they introduce multi-label box model, which uses probabilistic semantics of box embeddings, representing labels as boxes instead of vectors. Their model is evaluated extensively on 12 datasets, and reviewers agreed the paper was well written and well motivated. While it is pretty straightforward application of box embeddings to multi label problem, it is well motivated and the paper adds to the existing literature on box embeddings. 

Reviewer Eo7g had a concern with experimental setting, including missing a baseline Abboud et al. (2020). Even after the baseline was added, the reviewer was not convinced about the model’s performance, as the baseline was not extensively tuned. The authors responded with the heavy computational costs of associated with tuning the margin. Reviewer X5cP also pointed the problem with HMC dataset, for which the experimental results should be updated. Given the issues, the paper could benefit from another round of revisions.
Decision: Accept (Poster)