Visually Consistent Hierarchical Image Classification

## Abstract

Abstract Hierarchical semantic classification requires the prediction of a taxonomy tree instead of a single flat level of the tree, where both accuracies at individual levels and consistency across levels matter. We can train classifiers for individual levels, which has accuracy but not consistency, or we can train only the finest level classification and infer higher levels, which has consistency but not accuracy. Our key insight is that hierarchical recognition should not be treated as multi-task classification, as each level is essentially a different task and they would have to compromise with each other, but be grounded on image segmentations that are consistent across semantic granularities. Consistency can in fact improve accuracy. We build upon recent work on learning hierarchical segmentation for flat-level recognition, and extend it to hierarchical recognition.
It naturally captures the intuition that fine-grained recognition requires fine image segmentation whereas coarse-grained recognition requires coarse segmentation; they can all be integrated into one recognition model that drives fine-to-coarse internal visual parsing.
Additionally, we introduce a Tree-path KL Divergence loss to enforce consistent accurate predictions across levels.
Our extensive experimentation and analysis demonstrate our significant gains on predicting an accurate and consistent taxonomy tree.

## 1 Introduction

The goal of hierarchical classification is to predict all classes at each hierarchical level simultaneously (e.g., Ruby-throated Hummingbird - Bird).
Figure 1 provides an example of various challenges in the hierarchical classification problem.
For our analysis, we conduct training on the iNaturalist 2018 dataset [], with a two-level hierarchy.
Following the most commonly used approach for hierarchical classification [, , ], we extract features from a shared backbone (ImageNet []-pretrained ResNet-50 []) and train separate classifiers to predict classes at each level.

Unlike flat-level classification, hierarchical classification has multiple possible outputs. The challenges arise when predictions at the fine and coarse levels differ (2nd and 3rd rows in Figure 1).
Even if predictions are correct at one level, it is difficult to trust the prediction if predictions at other levels are incorrect (e.g., Ruby-throated Hummingbird - Plant), especially when we do not have ground truth labels at test time.
Therefore, our focus lies in achieving consistent and accurate predictions across all levels simultaneously.

However, predicting multiple levels of hierarchy is more challenging than predicting a single flat-level category, leading to a confrontation between coarse and fine level learning [].
That is, models trained to predict all hierarchical levels show degraded performance compared to the model trained to predict a single flat level of class.
This is because hierarchical classification entails addressing multiple classification tasks concurrently.
For instance, distinguishing between similar-looking bird species at the finest level (e.g., Anna’s Hummingbird - Ruby-throated Hummingbird) and differentiating between birds and plants at the coarse level represent distinct tasks, each requiring attention to different regions in the image.
This phenomenon is illustrated in Figure 2, where we observe that classifiers at each level often internally attend to different parts of the image.
From this observation, we attribute performance degradation in hierarchical recognition to the inconsistency among hierarchy classifiers.
We seek to enhance consistency by guiding classifiers to consistently focus on the same areas, where “bird” (for the coarse classifier) and “Ruby-throated Hummingbird” (for the fine-level classifier) are located.

Figure: Figure 1: Consistency is not only necessary for hierarchical classification but also helps improve accuracy. In hierarchical classification, categorizing all levels of classes accurately is important to establish the model’s reliability. However, training on multiple levels can lead to conflicts between classifiers, hindering performance compared to single-level training []. We address this challenge by focusing on inconsistent cases (like rows 2 & 3), where some classifiers correctly identify relevant image regions. Our method encourages classifiers at all levels to consistently focus on the same image regions, potentially leveraging the correctly identified parts from one of the classifiers. Compared to the baseline (separate classifiers with shared ResNet-50 [] backbone pre-trained on ImageNet []), by promoting consistency, our method significantly boosts accuracy on the iNaturalist 2018 dataset []. (Green border indicates correct predictions after applying our method, and red indicates incorrect predictions. The outer lines represent coarse-level predictions, while the inner lines represent fine-level predictions.)
Refer to caption: extracted/5665578/figs/problem6.png

Our insight is that although the fine-level classifier and the coarse classifier may have different focal points, they are inherently connected.
For instance, while the fine-level classifier examines aspects like the beak, wings, and feathers to classify bird species, the coarse classifier can discern between birds and flowers by integrating these details into the overall body of the bird.
In other words, the coarse classifier should focus on the larger part formed by grouping the features attended to by the fine classifier.
Therefore, we hypothesize that by consistently aligning the areas of focus for the fine classifier and coarse classifier, we could improve both consistency and accuracy.

To this end, we employ the recent unsupervised image segmentation method, CAST [].
CAST exhibited hierarchical segmentation, transitioning from finer to coarser segments.
CAST has demonstrated the capability to group related pixels consistently through internal parsing within images, without segmentation labels.
Inspired by this, we propose Hierarchical-CAST (H-CAST), which utilizes CAST to align the focus of different levels of classifiers on the same areas for hierarchical recognition.
Unlike previous hierarchical classification approaches that train separate models for each level [, , ], H-CAST is an integrated model where features learned at the fine level are grouped and passed to the coarse level (Figure 2), ensuring spatial consistency.
Since our method is an integrated model, if details initially captured at the fine level are incorrect, it will receive negative signals (errors) during the learning process toward coarser levels.
As training processes, the model is encouraged to capture accurate fine-level details to improve learning at subsequent levels.

Additionally, we propose Tree-path KL Divergence loss to further enhance semantic consistency by considering label relationships across levels so that predictions at the fine-level and coarse-level align within the taxonomy.
In Figure 1, we can observe that our method of improving consistency surprisingly leads to an enhancement in accuracy as well.
Many previously inconsistent and incorrect cases are transformed into correct classifications.

To assess both accuracy and consistency, we evaluate our method with a new metric called Full-Path Accuracy (FPA), which measures the proportion of samples in the dataset correctly predicted at all hierarchy levels.
Validated on common benchmarks for hierarchical recognition, our method consistently outperforms the ViT-based baseline by more than 17-25%p in FPA metric. We empirically demonstrate its effectiveness through extensive experimentation and analysis. Furthermore, compared to the common segmentation learned through a flat semantic taxonomy, we demonstrate that a hierarchical semantic taxonomy also in turn improves image segmentation.

Our key contributions can be summarized as follows.

- •
Hierarchical-CAST (H-CAST): We propose H-CAST to consistently align the focus of hierarchy classifiers within the image. Our model enables the utilization of various granular segments, ensuring that focus remains on the same region while fine-detailed features transitioning to broader features at the coarse level (spatial consistency). This expansion of the focus region fosters a natural and consistent learning process for hierarchical recognition.
- •
Tree-path KL Divergence loss: We introduce a novel loss function that considers label relationships across hierarchy levels (semantic consistency).
Our loss enforces predictions to be correct at different levels, demanding accuracy and consistency at the same time.
- •
These contributions combined lead to significant improvements in both accuracy and consistency, which we demonstrate via a new metric: Full-Path Accuracy (FPA).

Figure: Figure 2: While prior works train separate branches for different levels, our integrated model encourages internal consistency for consistent prediction among classifiers. Commonly used separate classifiers often lead to inconsistency between hierarchy levels due to different attending regions between classifiers. Our Hierarchical-CAST (H-CAST) enables the utilization of various granular segments, ensuring that focus remains on the same region while fine-detailed features transitioning to broader features at the coarse level. In the segmentation images inside H-CAST, consistent groupings are represented by identical colors. The segmentation images are the results of 32-way and 8-way, respectively. For instance, when observing the red bird segments, we notice finer details such as wings, body, head, and tail in the fine level (32-way), while in the subsequent level (8-way), it is grouped as the entire bird. By leveraging this consistent internal parsing, we encourage the model to focus on the coherent regions within images.
Refer to caption: extracted/5665578/figs/prior_vs_ours_2.png

## 2 Related Work

Hierarchical Classification (HC) is a problem that has been extensively studied. Before the advent of deep learning, works in this area mainly focused on tree-based [, , ] and DAG-based [, ] methods. For a more thorough study on this topic, we refer the reader to survey [, ]. Recent deep learning-based methods can be broadly categorized into three main types: 1) label-embedding methods, 2) hierarchical losses, and 3) hierarchical architectures.

1) Label-embedding methods focus on incorporating hierarchical information into labels to replace flat one-hot encodings. Label-embedding space can be derived by hierarchical relationships [, ], class characteristics [], or learned by a language model []. The label-embedding can be constructed on the hypersphere [, ] or in the hyperbolic space [, ] as well to enhance the expressiveness. Our approach diverges from these methods in
two fundamental ways: 1) our tree path encoding is designed to encourage semantic consistency across granularities. 2) Instead of only encoding the finest level, our method provides the complete label hierarchy.

2) Hierarchical losses modify objective functions to integrate hierarchical information as a form of supervision. One straightforward method involves utilizing tree structures, either based on tree-metrics [], or the lowest common ancestors [, ]. Another approach directly calculates losses at different levels according to the hierarchy [, ]. However, recent work showed that sharing information between coarse and fine-grained levels may hamper performance []. This necessitates a more ingenious design for combining information across granularities. Our work utilizes hierarchical features in a cascade manner, leveraging the connection between coarse and fine-grained levels.

3) Hierarchical architectures enable models to output predictions directly at different levels of hierarchy. A common approach designs models with separate branches, each responsible for a specific level [, , , , , ], or by structuring the feature space according to the hierarchy [, ]. Other works construct the prediction tree in a *coarse-to-fine* manner based on the conditional likelihoods of the ancestors [, , ]. In contrast, our method establishes a *fine-to-coarse* internal parsing and ensures that different hierarchical granularities contribute consistently to learning at the next level.

Hierarchical Fine-grained Recognition (HFR) aims to assign hierarchical labels to the object within images, it is harder than traditional HC due to smaller inter-class differences and emphasis on outputting the complete label hierarchy. Typical HFR model designs adhere to a hierarchical architecture, allowing the model to produce multiple outputs at varying levels of granularity [, , , , ]. While branching is effective, it is prone to inconsistent predictions.

Inconsistency occurs when the coarse and fine-grained prediction of an object violates the hierarchy structure, i.e., the coarse prediction is not the parent category of the fine-grained prediction. Several prior works have attempted to solve the inconsistency by post-processing [, , ] or adding violation penalization to the loss function [, , ]. More recently, Wang et al. [] proposed a consistency-aware approach by explicitly adjusting prediction scores through coarse-to-fine deduction and fine-to-coarse induction. In contrast, our method implicitly encourages consistency through hierarchical label encoding without any post-processing. In addition, our framework improves semantic consistency and spatial consistency simultaneously.

Unsupervised Image Segmentation learns to segment the input images according to the extracted features without human-annotated segmentation [, , ]. Recently, CAST [] adopted the Vision Transformer (ViT) [] with two significant modifications. One change involved using superpixels instead of fixed-shape patches to capture more complex visual contours for segmentation. Secondly, graph pooling was applied for hierarchical segmentation. Our framework takes a step further by utilizing the information from hierarchical labels via cross-granularity supervision to enhance the consistency and accuracy of hierarchical recognition.

## 3 Hierarchical Classification based on Internally Consistent Segmentations

Our goal is to enhance the consistency of hierarchical recognition, thereby concurrently improving the accuracy of the model.
The inconsistent predictions in hierarchical recognition indicate that, in other words, labels at some levels are correctly predicted.
Therefore, our motivation is that fostering consistent predictions across hierarchical levels, leveraging information from such accurately predicted levels, would further enhance overall accuracy.
Based on this motivation, we design a progressive learning scheme for hierarchical recognition, where the learning of each level contributes to the learning of the next level, instead of training separate models focusing on each individual level.

Specifically, we address two types of inconsistency in hierarchical recognition. One is spatial inconsistency, where classifiers at different levels attend to different regions (Figure 2).
To address this, we propose H-CAST in Section 3.1.
The other is semantic inconsistency, where predictions at different levels are not aligned within the taxonomy (e.g., “Hummingbird” - “Plant”).
We propose a new Tree-path KL Divergence loss that encodes parent-child relations to handle semantic inconsistency in Section 3.2.
Figure 3 provides an overview of our method.

Figure: Figure 3: Our method consists of two parts: Spatial Consistency and Semantic Consistency module. In the Spatial Consistency module, the parsed images using superpixels are grouped based on related parts as they transition from fine to coarse levels. This ensures that each hierarchical classifier internally focuses on the same corresponding regions. In the Semantic Consistency module, we incorporate hierarchical relationships between labels. This approach allows us to achieve consistent learning across the entire hierarchy. By promoting consistency, our method encourages classifiers at different levels to enhance overall performance, rather than conflicting with each other.
Refer to caption: extracted/5665578/figs/pipeline_2.png

### 3.1 H-CAST for Spatial Consistency

The areas of focus within the image differ when conducting classification at the fine-grained level compared to the coarse level.
When distinguishing between similar-looking species (e.g., Anna’s Hummingbird vs. Ruby-throated Hummingbird), the fine-grained recognition requires attention to fine details like the bird’s beak and wings; meanwhile, at the coarse level (e.g., bird vs. plant), the attention shifts to larger parts such as the overall body of the bird.
However, this shift in focus towards larger objects does not imply a sudden disregard for the previously focused details and a search for new larger objects.
Rather, a natural approach involves combining detailed features such as the bird’s beak, belly, and wings for accurate bird recognition.
Therefore, to maintain a coherent focus during hierarchical recognition and ensure spatial consistency, we design a model where the details learned at the fine level (e.g., bird’s beak and wings) are transferred to the coarse level as broader parts (e.g., bird’s body) through consistent feature grouping.

For internally consistent feature grouping, we build upon recent work CAST [].
CAST develops a hierarchical segmentation from fine to coarse, an internal part of the recognition process. However, their segmentation is driven by a flat recognition objective at the very end of visual parsing.
We extend it by imposing fine-to-coarse semantic classification losses
at different stages of segmentations throughout the visual parsing process.
Our design reflects the intuition that finer segments can be helpful in capturing fine-grained details (e.g., beaks and wings) required for fine-grained recognition, whereas coarser segments can be effective in representing broader features (e.g., the body of a bird) needed for coarse-grained recognition.
We have a single hierarchical recognition grounded on internally consistent segmentations, each driven by a classification objective at a certain granuality.
We refer to our method as Hierarchical-CAST (H-CAST).

Consider a hierarchical recognition task where $x$ denotes an image associated with hierarchical labels $y_{1},\ldots y_{L}$, encompassing a total of $L$ levels in the hierarchy.
Level $L$ is the finest level (*i.e.,* leaf node), and Level 1 is the coarsest level (*i.e.,* root node).
Then, given an image $x$, the hierarchical image recognition task is to predict labels at all levels across the hierarchy.

Let $Z_{l}$ and $S_{l}$ denote the feature and segments at $l$-th hierarchical level, respectively,
Then, we obtain superpixels for image $x$ by using the off-the-shelf algorithm SEEDS [] to divide the image into regions with similar colors and local connectivity.
These superpixels serve as input for the Vision Transformer (ViT) instead of fixed-size patches and simultaneously become the finest (initial) segments, $S_{L+1}$.
$l$-th feature tokens $Z_{l}$ is the concatenation of class tokens ($Z_{l}^{class}$) and segment tokens ($Z_{l}^{seg}$).
Then, Graph pooling [] aggregates segments with high feature similarity, allowing feature $Z_{l}$ to progressively learn a more global visual context as it transitions from $Z_{L}$ to $Z_{1}$.

For hierarchical recognition, we add a classification head ($f_{l}$) consisting of a single linear layer at each level.
Then, we define the Hierarchical Spatial-consistency loss as the sum of $L$ cross-entropy losses ($L_{CE}$), denoted as

$$ $L_{HS}=\sum_{l=1}^{L}L_{CE}(f_{l}(Z_{l}^{class}),y_{l}).$ (1) $$

Our approach differs from CAST in that while CAST uses the class token as the final objective, we design our model to incorporate hierarchical supervision during the training process.
This ensures that labels from different levels progressively contribute to each other.
In the Experimental section 4.3, we will demonstrate the effectiveness of our design, compared to alternative designs, including hierarchy supervision in the coarse-to-fine direction.

### 3.2 Tree-path KL Divergence loss for Semantic Consistency

To improve semantic consistency, we propose a new loss function called Tree-path KL Divergence loss, which directly incorporates hierarchical relationships between labels.
Our idea is to encode the entire hierarchical structure so that a model can learn the hierarchy by outputting the tree of hierarchy.
To this end, we first concatenate labels from all levels to create a distribution, as $Y=\frac{1}{L}[1_{y_{L}};\ldots;1_{y_{1}}]$, where $1_{y_{l}}$ represents the one-hot encoding for level $l$.
Next, we concatenate the outputs of each classification head and then apply the log softmax function (LogSoftmax). We use Kullback–Leibler divergence loss ($KL$) to align this output with the ground truth distribution $Y$. Then, TK loss is calculated as follows.

$$ $L_{TK}=KL(\text{LogSoftmax}([f_{L}(Z_{L}^{class});\ldots;f_{1}(Z_{1}^{class})] ),Y)$ (2) $$

This loss penalizes predictions that do not align with the taxonomy by simultaneously training on multiple labels within the hierarchy.
Therefore, despite the simplicity, TK loss enables the model to enhance semantic consistency through this vertical encoding from the root (parent) node of the hierarchy level to the leaf (children) node.
Our final loss becomes as follows, where $\alpha$ is a hyperparameter to control the weight of $L_{TK}$,

$$ $L=L_{HS}+\alpha L_{TK}.$ (3) $$

## 4 Experiments

### 4.1 Experimental Settings

Datasets.
We use three widely used benchmarks in hierarchical recognition: FGVC-Aircraft [], CUB-200-2011 [], and BREEDS [].
FGVC-Aircraft consists of a 3-level hierarchy including maker, family, and model (*e.g.*, Boeing - Boeing 707 - 707-320 );
CUB-200-2011 comprises a 3-level hierarchy with order, family, and species;
BREEDS, a subset of ImageNet [], includes four 2-level hierarchy datasets with different depths/parts based on the WordNet [] hierarchy: Living-17, Non-Living-26, Entity-13, Entity-30.
For BREEDS, we conduct training and validation using their source splits.
We include detailed dataset descriptions in Table 2.

**Table 1: FPA takes both correctness and consistency into consideration. The table illustrates the difference between TICE [] and FPA. The first row lists all possible predictions. The predictions at different levels are represented as highlighted circles. The borders of the nodes that are correctly predicted are marked in green, and vice versa. The second and third rows show whether TICE and FPA mark the prediction results as positive under different circumstances. While TICE only measures consistency, FPA will only mark the prediction that is correct and consistent as positive.**
|  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TICE | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ |
| FPA | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |

**Table 2: Benchmark Datasets.**
| Datasets | Aircraft | CUB | L-17 | NL-26 | E-13 | E-30 |
| --- | --- | --- | --- | --- | --- | --- |
| # Levels | 3 | 3 | 2 | 2 | 2 | 2 |
| # of classes | 30-70-100 | 13-38-200 | 17-34 | 26-52 | 13-130 | 30-120 |
| # Train images | 6,667 | 5,944 | 44.2K | 65.7K | 167K | 154K |
| # Test images | 3,333 | 5,794 | 1.7K | 2.6K | 6.5K | 6K |

Evaluation Metrics.
We evaluate our models using metrics for both accuracy and consistency.
First, we present level-accuracy, following community convention [], which is the proportion of correctly classified instances at each level.
Additionally, we evaluate the weighted average precision (wAP) [] which considers the classification difficulty across different hierarchies, as
$\text{wAP}=\sum_{l=1}^{L}\frac{N_{l}}{\sum_{k=1}^{L}N_{k}}P_{l}$,
where $N_{l}$ and $P_{l}$ denote the number of classes and Top-1 classification accuracy at level $l$, respectively. This metric gives more weight to the accuracy at the fine-grained levels, which is more challenging.
To evaluate consistency we calculate Tree-based InConsistency Error rate (TICE) [], which tests whether the prediction path exists in the tree.
$\text{TICE}={n_{\text{i}c}}/{N}$, where $n_{\text{i}c}$ denotes the number of samples with inconsistent prediction paths, and $N$ refers to the number of all test samples.
Finally, we propose Full-Path Accuracy (FPA) as a metric to assess both accuracy and consistency: $\text{FPA}={n_{\text{a}c}}/{N}$, where $n_{\text{a}c}$ refers to the number of samples with all level of labels correctly predicted.
This metric straightforwardly calculates the proportion of correctly classified samples at all levels, ultimately representing our primary metric of interest.
The difference between FPA and TICE is illustrated in Table 1.

Comparison methods.
Our architecture is based on the ViT architecture [], so we use the ViT model as our baseline.
As a ceiling-performance baseline, we compare with “Single” models which are trained with a single flat level in the hierarchy.
ViT-Single is trained to classify one hierarchy level using the class token of the ViT model.
Therefore, for the Aircraft Dataset, ViT-Single entails training three independent models.
Likewise, CAST-Single trains independent models, each dedicated to learning a single hierarchy level with the CAST architecture [].
ViT-Hier is similar to our approach in that it trains each hierarchy level using the class token from the last $l$ blocks.

Architecture and Training.
For a fair comparison, we use ViT-S and CAST-S models of corresponding sizes. As in CAST, segmentation granularity is set to 64, 32, 16, 8 after 3, 3, 3, 2 encoder blocks, respectively.
Our training progresses from fine to coarse levels, with each segment corresponding accordingly.
The initial number of superpixels is set to 196, and all data is trained with a batch size of 256 for 100 epochs.
Following the literature, we use ImageNet pre-trained models for the Aircraft and CUB datasets. For the ImageNet subset BREEDS dataset, we train the models from scratch.
Detailed hyperparameter settings can be found in the Appendix  A.1.

### 4.2 Hierarchical Recognition Benchmarks

**Table 3: Comparison of Methods in terms of Accuracy(%) and Consistency Metrics(%) on FGVC-Aircraft and CUB-200-2011 benchmarks. Single-level serves as a celing-performance baseline, with ViT-Hier showing decreased accuracy compared to individually trained Single-level models. However, Single-level models’ consistency performance suffers due to separate models. In contrast, our model (H-CAST) demonstrates comparable or higher accuracy than Single-level models while significantly enhancing Consistency performance. (Higher the metric is the best, except TICE.)**
| Method | FGVC-Aircraft (30-70-100) | CUB-200-2011 (13-38-200) |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FPA | maker | family | model | wAP | TICE | FPA | order | family | species | wAP | TICE |  |
| ViT-Single | 76.99 | 94.27 | 91.93 | 80.14 | 86.39 | 10.98 | 82.30 | 98.50 | 94.84 | 84.78 | 87.01 | 5.76 |
| Cast-Single | 78.22 | 92.95 | 88.93 | 82.39 | 86.26 | 10.77 | 81.50 | 98.38 | 94.82 | 83.78 | 86.21 | 6.14 |
| ViT-Hier | 72.10 | 92.35 | 86.26 | 75.94 | 82.01 | 15.75 | 77.03 | 98.40 | 92.94 | 79.43 | 82.46 | 8.72 |
| Ours | 83.72 | 94.96 | 91.39 | 85.33 | 88.90 | 5.01 | 83.28 | 98.65 | 95.12 | 84.86 | 87.13 | 4.12 |
| *Our Gains* | +11.62 | +2.61 | +5.13 | +9.39 | +6.89 | +10.74 | +6.25 | +0.25 | +2.18 | +5.53 | +4.67 | +4.60 |

FGVC-Aircraft and CUB datasets.
Table 3 presents results on Aircraft and CUB datasets.
In our experimental results, we first observe a significant performance drop of ViT-Hier compared to ViT-Flat.
This highlights a common issue in hierarchical recognition using shared backbones, where a coarse-fine tradeoff leads to performance degradation, as seen in prior ResNet-based hierarchical recognition [].
Our experiments reveal that this problem also exists in ViT architectures.
This indicates that hierarchical recognition is a challenging problem that cannot be solely addressed by providing hierarchy supervision to class tokens.

On the other hand, our method consistently outperforms most Flat models. Notably, we achieve substantial improvements in consistency performance as well.
While Flat-level models exhibit high accuracy, they require $L$ times the training time and memory to train each model.
Moreover, since they are separate models, consistency suffers.
In contrast, our approach excels in both accuracy and consistency.
Specifically, from the FPA metric viewpoint, which emphasizes both accurate and consistent predictions across all levels, we observe an approximately +11.6%p improvement in the Aircraft dataset and a +6.3%p improvement in the CUB dataset compared to ViT-Hier.
These results validate our approach of utilizing segments in progressive learning to enhance both accuracy and consistency in hierarchical recognition.

### 4.3 Analysis

BREEDS datasets.
Next, we compare model performance on the BREEDS dataset, which has a larger number of training data compared to Aircraft and CUB.
As shown in Table 4, we observe a similar pattern to the results from Aircraft and CUB datasets.
When the models are trained from scratch without using ImageNet-pretrained models, our model surpasses ViT-Hier by a large margin in both accuracy and consistency, by more than 10.63-11.82%p in the FPA metric.
H-CAST even surpasses CAST-Single models by 2.2-7.1%p in the FPA metric.
This demonstrates that utilizing hierarchical information results in a significant performance improvement in recognition tasks when pre-trained weights are not employed.

Figure: Figure 4: H-CAST enhances the interpretability of the model’s predictions. We visualize the feature grouping from fine to coarse for full-path correct and incorrect predictions of our approach on Entity-30 dataset. The hue for all segments is consistent, while saturation and value vary to represent different granularities. For full-path correct predictions (*i.e.*, predictions at all levels are correct), visual details are effectively captured and grouped to identify larger objects at coarser levels. In contrast, the model’s segments fail to recognize the object accurately for full-path incorrect predictions (*i.e.*, predictions of all levels are incorrect). This highlights an additional advantage of utilizing segments in hierarchical recognition models, as it not only provides guidance on where to focus but also enhances interpretability of the model’s predictions.
Refer to caption: extracted/5665578/figs/ours_segs/consistent_correct/imgs/00002667_img.png

**Table 4: Comparison of Methods in terms of Accuracy(%) and Consistency Metrics(%) on BREEDS benchmarks. Our model (H-CAST) significantly outperforms ViT-Hier in both Accuracy and Consistency metrics (Our Gains).**
| Method | Living-17 (17-34) | Non-Living-26 (26-52) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FPA | coarse | fine | wAP | TICE | FPA | coarse | fine | wAP | TICE |  |
| ViT-Single | 66.24 | 75.71 | 72.06 | 73.28 | 17.11 | 57.46 | 65.73 | 66.32 | 57.46 | 23.27 |
| Cast-Single | 79.82 | 88.06 | 82.88 | 84.61 | 8.82 | 76.77 | 84.77 | 81.08 | 82.31 | 11.77 |
| ViT-Hier | 74.06 | 80.94 | 74.88 | 76.90 | 10.50 | 72.04 | 73.31 | 68.39 | 70.03 | 12.45 |
| Ours | 85.12 | 90.82 | 85.24 | 87.10 | 3.19 | 82.67 | 87.89 | 83.15 | 84.73 | 5.26 |
| Our Gains | +11.06 | +9.88 | +10.36 | +10.20 | +7.31 | +10.63 | +14.58 | +14.76 | +14.70 | +7.19 |
|  | Entity-13 (13-130) | Entity-30 (30-120) |  |  |  |  |  |  |  |  |
| ViT-Single | 64.22 | 76.28 | 76.06 | 76.08 | 21.33 | 66.93 | 76.47 | 74.35 | 74.77 | 18.75 |
| Cast-Single | 78.63 | 87.80 | 83.72 | 84.09 | 10.65 | 82.67 | 87.89 | 83.15 | 84.73 | 5.26 |
| ViT-Hier | 74.63 | 86.95 | 75.39 | 77.70 | 5.19 | 73.01 | 81.38 | 74.10 | 74.76 | 11.61 |
| Ours | 85.68 | 93.42 | 86.15 | 87.60 | 1.69 | 84.83 | 90.23 | 85.45 | 85.88 | 2.57 |
| Our Gains | +11.05 | +6.47 | +10.76 | +9.90 | +3.50 | +11.82 | +8.85 | +11.35 | +11.12 | +9.04 |

Fine-to-Coarse vs Coarse-to-Fine learning.
Our model adopts a Fine-to-Coarse learning strategy, wherein it initially focuses on learning fine labels in the lower block and progressively integrates to learn coarse labels.
This is contrary to conventional ResNet-based models, which typically learn coarser features in the lower block [, , ].
Thus, to demonstrate the effectiveness of this Fine-to-Coarse learning architecture, we compare our model with two baselines.
The first baseline, referred to as the Coarse-to-Fine approach, initiates training by focusing on coarse labels in the lower block and progresses towards fine labels, which is contrary to our approach.
The second baseline, the Fine-Coarse Merging method, concatenates the class token from the lower block with coarser segments from the upper block to learn coarse labels, and vice versa concatenates the class token from the upper block with finer segments from the lower block to learn fine labels.
This approach leverages class tokens and segments of varying granularities intuitively.
For the sake of evaluating architecture effectiveness, we do not use Tree-path KL Divergence loss. The comparison results of these three architectures on FGVC-Aircraft dataset are summarized in Table 5.

**Table 5: Coarse-to-Fine learning scheme achieves best overall performance. We report the impact of hierarchical learning direction on FGVC-Aircraft.**
| Learning Direction | FPA | maker | family | model | wAP |
| --- | --- | --- | --- | --- | --- |
| Coarse-to-Fine | 82.01 | 93.16 | 89.92 | 84.10 | 87.50 |
| Fine-Coarse merging | 81.76 | 93.52 | 90.31 | 84.58 | 87.93 |
| Fine-to-Coarse | 82.66 | 94.27 | 90.19 | 84.40 | 87.91 |

We find that Coarse-to-Fine learning shows the lowest performance, contrary to convention.
Fine-Coarse Merging exhibits slightly better accuracy at the fine level compared to our model.
However, the performance improvement is marginal considering the significant increase in the number of parameters in the classification head when using segment features as input.
Our Coarse-to-Fine learning scheme not only demonstrates simplicity in implementation but also showcases good performance, making it a favorable choice for our study.

**Table 6: Utilizing both losses yields best performance. We report the effect of proposed losses on FGVC-Aircraft.**
| $L_{HS}$ | $L_{TK}$ | FPA | maker | family | model | wAP |
| --- | --- | --- | --- | --- | --- | --- |
| ✗ | ✓ | 82.48 | 94.30 | 90.37 | 84.04 | 87.80 |
| ✓ | ✗ | 82.66 | 94.27 | 90.19 | 84.40 | 87.91 |
| ✓ | ✓ | 83.72 | 94.96 | 91.39 | 85.33 | 88.90 |

Effect of proposed losses.
To validate the efficacy of the two losses we proposed, we conduct an ablation study on FGVC-Aircraft dataset.
Table 6 reveals that each loss term, Hierarchical Spatial-consistency loss, and Tree-path KL Divergence loss, significantly contributes to performance enhancement.
Ultimately, utilizing both losses concurrently yields the best performance in terms of accuracy and consistency.

H-CAST can guide consistent spatial focus and improve the interpretability of the model’s predictions.
To explore how our model learns segments with varying granularity from fine to coarse levels, we visualize segments based on hierarchy levels on BREEDS Entity-30 dataset in Figure 4.
In full-path correct prediction cases, where predictions at all levels are correct (Figure 4, Left), visual details are effectively captured at the fine level and consistently grouped to identify larger objects at the coarse level.
However, in full-path incorrect prediction cases, where predictions at all levels are incorrect (Figure 4, Right), the model’s segments fail to recognize the object accurately.
This underscores an added benefit of incorporating segments in hierarchical recognition models, as it not only contributes to consistent predictions but also enhances the interpretability of the model’s predictions.

Hierarchical semantic recognition enhances internal segmentation.
While our primary task is hierarchical recognition, we further examine whether incorporating additional hierarchical label information can be beneficial for image segmentation.
To investigate this, we compare CAST, which employs flat-level recognition, with segmentation results.
In Figure 5, it can be observed that utilizing additional taxonomy benefits segmentation.
For example in the first ‘bird’ image, H-CAST is able to segment meaningful parts such as the face, belly, and a branch, with less fractured compared to the CAST.

To quantitatively evaluate the method, we use the ImageNet segmentation dataset, ImageNet-S [], to obtain the ground-truth segmentation data for BREEDS dataset.
The number of samples in the BREEDS validation data for which ground-truth segmentation data can be obtained from ImageNet-S is 381 for Living-17, 510 for Non-Living-26, 1,336 for Entity-30, and 1,463 for Entity-13.
To calculate the region mIOU for fine-level objects, we use the last-level segments (8-way) for segmentation.
Following CAST, we name the 8-way segmentations using OvSEG [].

Table in Figure 6 presents a comparison of region mIOU.
Since H-CAST uses coarse-level supervision for the last-level segments, H-CAST can underperform compared to CAST, which employs fine-level supervision. Nevertheless, H-CAST outperforms CAST in most datasets, such as Living-17, Non-Living-26, and Entity-13.
Additionally, Figure 6 illustrates the segmentation results.
In the visualized images, we can observe that H-CAST better captures the overall shape in a more coherent manner compared to CAST.
In the Entity-30 example of “llama” in the second row, CAST demonstrates better results than H-CAST. H-CAST, in this case, incorrectly segments the face and body into different groups. We believe that finetuning the model, which was trained with hierarchical recognition, using segmentation data can further improve its performance.
These quantitative and qualitative results demonstrate that utilizing hierarchical information is effective not only for recognition but also for segmentation.

Figure: Figure 5: Additional taxonomy information improves segmentation. We compare our segmentation results on Entity-30 dataset with CAST at the coarsest level to explore the potential benefits of incorporating additional hierarchical label information for image segmentation. For example in the first ‘bird’ image, H-CAST is able to segment meaningful parts such as the face, belly, and a branch, with less fractured compared to the CAST. Thus, H-CAST delivers an improvement in segmentation with the benefits of hierarchy. More segmentation results are shown in Appendix A.2.
Refer to caption: extracted/5665578/figs/ours_vs_cast/imgs/00013407_img_a.png

Figure: Figure 6: Using hierarchical information helps in segmenting object parts on BREEDS. To further verify the effectiveness of using hierarchy information in segmentation, we visualize segmentation results and measure the region mIOU of Fine-level objects for samples whose segmentation ground truth (GT) data can be obtained from ImageNet-S []. Following CAST, we use the last-level segments and name the 8-way segmentations using OvSEG []. In the visualized images, we can observe that H-CAST better captures the overall shape in a more coherent manner compared to CAST. In the Entity-30 example of “llama” in the second row, CAST demonstrates better results than H-CAST. H-CAST, in this case, incorrectly segments the face and body into different groups. We believe that finetuning the model, which was trained with hierarchical recognition, using segmentation data can further improve its performance. In quantitative evaluation, H-CAST outperforms CAST in most datasets despite using coarse-level supervision for the last-level segments whereas CAST employs fine-level supervision. These qualitative and quantitative results demonstrate that utilizing additional taxonomy is effective not only for recognition but also for segmentation.
Refer to caption: extracted/5665578/figs/segmentation/imgs_living/1.png

## Appendix A Appendix

### A.1 Hyperparameters for Training.

**Table 7: Hyper-parameters for training H-CAST and ViT on FGVC-Aircraft, CUB-200-2011, and BREEDS. We follow mostly the same set up as CAST [].**
| Parameter | Aircraft | CUB | BREEDS |
| --- | --- | --- | --- |
| batch_size | 256 | 256 | 256 |
| crop_size | 224 | 224 | 224 |
| learning_rate | $1e^{-3}$ | $5^{e-4}$ | $5^{e-4}$ |
| weight_decay | 0.05 | 0.05 | 0.05 |
| momentum | 0.9 | 0.9 | 0.9 |
| total_epochs | 100 | 100 | 100 |
| warmup_epochs | 5 | 5 | 5 |
| warmup_learning_rate | $1e^{-4}$ | $1e^{-6}$ | $1^{e-6}$ |
| optimizer | Adam | Adam | Adam |
| learning_rate_policy | Cosine decay | Cosine decay | Cosine decay |
| augmentation [] | RandAug(9, 0.5) | RandAug(9, 0.5) | RandAug(9, 0.5) |
| label_smoothing [] | 0.1 | 0.1 | 0.1 |
| mixup [] | 0.8 | 0.8 | 0.8 |
| cutmix [] | 1.0 | 1.0 | 1.0 |
| ViT: # Tokens | $[196]_{\times 11}$ |  |  |
| CAST-S: # Tokens | $[196]_{\times 3},[64]_{\times 3},[32]_{\times 3},[16]_{\times 2}$ |  |  |

### A.2 Additional Visulizations

Figure: Figure 7: Additional visual results on segmentation show that H-CAST with additional taxonomy information improves segmentation.
Refer to caption: extracted/5665578/figs/ours_vs_cast/imgs/00000582_img_a.png

### A.3 Limitations

A limitation of our work, similar to CAST, is that the performance of segment discovery relies on the superpixel algorithm. However, our work aims to increase consistency at the hierarchical level, ensuring classifiers focus on the same regions, rather than achieving precise segmentation. Therefore, the inability to find accurate contours is not a significant issue.

## NeurIPS Paper Checklist

- 1.
Claims
- Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
- Answer: [Yes]
- Justification: The abstract and introduction clearly state our claims and contributions.
- Guidelines:
•
The answer NA means that the abstract and introduction do not include the claims made in the paper.
•
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
•
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
•
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
- 2.
Limitations
- Question: Does the paper discuss the limitations of the work performed by the authors?
- Answer: [Yes]
- Justification: Limitation is described in Appendix.
- Guidelines:
•
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
•
The authors are encouraged to create a separate "Limitations" section in their paper.
•
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
•
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
•
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
•
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
•
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
•
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
- 3.
Theory Assumptions and Proofs
- Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
- Answer: [N/A]
- Justification: The paper does not include theoretical results.
- Guidelines:
•
The answer NA means that the paper does not include theoretical results.
•
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
•
All assumptions should be clearly stated or referenced in the statement of any theorems.
•
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
•
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
•
Theorems and Lemmas that the proof relies upon should be properly referenced.
- 4.
Experimental Result Reproducibility
- Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
- Answer: [Yes]
- Justification: We provide details of our experiments in Section 4.1 and training hyperparameters in Appendix A.1.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
•
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
•
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
•
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a)
If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b)
If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
- 5.
Open access to data and code
- Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
- Answer: [No]
- Justification: We will provide the code and checkpoints of the model upon publication.
- Guidelines:
•
The answer NA means that paper does not include experiments requiring code.
•
Please see the NeurIPS code and data submission guidelines ([https://nips.cc/public/guides/CodeSubmissionPolicy](https://nips.cc/public/guides/CodeSubmissionPolicy)) for more details.
•
While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
•
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ([https://nips.cc/public/guides/CodeSubmissionPolicy](https://nips.cc/public/guides/CodeSubmissionPolicy)) for more details.
•
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
•
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
•
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
•
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
- 6.
Experimental Setting/Details
- Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
- Answer: [Yes]
- Justification: We provide details of our experiments in Section 4.1 and training hyperparameters in Appendix A.1.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
•
The full details can be provided either with the code, in appendix, or as supplemental material.
- 7.
Experiment Statistical Significance
- Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
- Answer: [No]
- Justification: Due to computational resource limitations, we were unable to conduct a sufficient number of experiments to generate error bars.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
•
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
•
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
•
The assumptions made should be given (e.g., Normally distributed errors).
•
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
•
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
•
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
•
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
- 8.
Experiments Compute Resources
- Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
- Answer: [Yes]
- Justification: We use four 40GB NVIDIA A40 GPUs for our experiments. We provide details of training hyperparameters in Appendix A.1.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
•
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
•
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
- 9.
Code Of Ethics
- Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics [https://neurips.cc/public/EthicsGuidelines](https://neurips.cc/public/EthicsGuidelines)?
- Answer: [Yes]
- Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.
- Guidelines:
•
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
•
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
•
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
- 10.
Broader Impacts
- Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
- Answer: [N/A]
- Justification: There is no societal impact of the work performed.
- Guidelines:
•
The answer NA means that there is no societal impact of the work performed.
•
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
•
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
•
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
•
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
•
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
- 11.
Safeguards
- Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
- Answer: [N/A]
- Justification: The paper poses no such risks.
- Guidelines:
•
The answer NA means that the paper poses no such risks.
•
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
•
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
•
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
- 12.
Licenses for existing assets
- Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
- Answer: [Yes]
- Justification: Our code is based on CAST [], and CAST follows MIT license.
- Guidelines:
•
The answer NA means that the paper does not use existing assets.
•
The authors should cite the original paper that produced the code package or dataset.
•
The authors should state which version of the asset is used and, if possible, include a URL.
•
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
•
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
•
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, [paperswithcode.com/datasets](paperswithcode.com/datasets) has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
•
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
•
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
- 13.
New Assets
- Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
- Answer: [N/A]
- Justification: Our code is based on CAST. We will make our codes publicly available.
- Guidelines:
•
The answer NA means that the paper does not release new assets.
•
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
•
The paper should discuss whether and how consent was obtained from people whose asset is used.
•
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
- 14.
Crowdsourcing and Research with Human Subjects
- Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
- Answer: [N/A]
- Justification: The paper does not involve crowdsourcing nor research with human subjects.
- Guidelines:
•
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
•
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
•
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
- 15.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
- Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
- Answer: [N/A]
- Justification: The paper does not involve crowdsourcing nor research with human subjects.
- Guidelines:
•
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
•
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
•
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
•
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: Please see questions listed alongside weakness.
soundness: 2
strengths: (1) The model shows better results than prior works.

(2) The examples given in the introduction help to explain and illustrate the reasoning behind the hierarchical focus.

(3) The ablations confirm that the added additional loss contributes to the performance.
summary: The paper proposes H-CAST architecture for hierarchical classification tasks. The architecture builds on top of prior CAST work: superpixels are fed into a ViT network, where periodic graph pooling operation aggregates the tokens of high similarity. This produces a fine-to-course hierarchy of features. Linear layers at each level of the hierarchy are used as classification heads. The paper also presents tree-path KL loss, where the entire path in the hierarchical class tree is matched. The method shows strong performance over baselines in several benchmark datasets.
weaknesses: (1) In L61, the difference in available labelling is presented as one of the motivations for hierarchical classification. However, the presented method assumes that all levels of the hierarchy are available. Can the technique work if the finest levels of supervision are not available?

(2) Similarly, given the availability of the finest-level label, the other course levels in the tree are implied, so perhaps a more appropriate flat baseline would be a ViT that only predicts finest-level classes (and thus parent nodes by simple aggregation). It would also provide a more appropriate comparison in terms of architecture.

(3) Given the relatively "small" sizes of the datasets (Tab 1.) by modern standards and some occasional closeness to the flat baselines in the scores (Tab 2.) Has there been any significant variability observed in the results? Would it be possible to include some measure of variance for some key results?

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the above.
soundness: 2
strengths: 1. The motivation for visual consistency is sound.
2. This work introduces new metrics for hierarchical classification, which can measure the coherence of hierarchical predictions.
3. The proposed method achieves SOTA on all datasets.
summary: This work proposes H-CAST, a model built upon CAST for hierarchical image classification, by addressing both visual consistency and semantic consistency across predictions at different hierarchical levels. To achieve this, hierarchical supervision at different network layers and a tree loss are introduced. Experiments on three datasets verify the proposed method can achieve better performance than the baseline method.
weaknesses: 1. The key point of this paper, which is making classifiers at different levels attend to consistent visual cues, lacks support from quantitative results or theory. For example, if wrongly classified cases are all associated with incorrect CAMs? What is the transfer rate after adopting the proposed model? Qualitative comparisons are subjective and lack statistical significance. In addition, Grad-CAM is an approximation and does not truly explain how the network operates.
2. This work relies heavily on CAST. Though the authors propose the concept of visual consistency, its implementation is directly borrowed from CAST. Consequently, the primary contributions of this work are merely the hierarchical supervision loss and tree KL loss, neither of which can ensure visual consistency. Additionally, the fine-to-coarse training strategy is also derived from CAST. The tree-path KL loss is trivial.
3. The tree KL loss defines the ground truth distribution according to the number of hierarchical levels (i.e., 1/L). It would be beneficial to address whether the proposed solution can effectively manage larger hierarchies, particularly those with depths of up to 10 or 20.
4. While the clustering module would potentially guide the model to attend to spatially-coherent regions, it is unclear why the model would “ensure that each hierarchical classifier focuses on the same corresponding regions”. In fact, the model still has the opportunity to find shortcuts (attending to different regions in different levels as Fig. 2) and meanwhile deliver correct classification results. In addition, I expect visualizations of Grad-CAM results in different hierarchical levels from the proposed model. 
5. The experiments are somewhat questionable. Firstly, the latest hierarchical competitor, HRN, published in CVPR'22, is relatively outdated. Secondly, the experimental results of HRN differ from those reported in the original publication. Thirdly, there are several competitors [a-b] (might be more, not carefully checked), released before two months prior to the DDL, that outperform this work.
6. Results in Fig. 5 do not totally make sense to me. Examples in a) and b) are not equally recognizable, i.e., all examples in b) are much harder to distinguish/group than those in a). As a result, it is hard to confirm whether poor clustering of pixels in b) is the cause or the effect of incorrect predictions. One way for improvement is to examine for similar hard-level images in a) whether correct predictions are achieved along with better clustering results. 
7. According to [c], existing work has explored various loss functions for hierarchical classification. It would be useful to compare the effectiveness of tree KL loss against these alternatives. Of course, the analysis should also be provided.
8. The comparison to hierarchical approaches is unfair. While FGN and HRN use ResNet-50 as the backbone， this work adopts ViT-S.
9. More top-leading hierarchical classification work should be included in the comparison.
10. The datasets used for evaluation are small. larger datasets with complex hierarchy (ie.g., ImageNet-1K, iNaturalist) should be evaluated to better assess effectiveness.
11. How about the training and inference speed of the proposed method? Given the incorporation of superpixels and segmentation in addition to classification, it is necessary to provide a comparison of resource costs, including both time and memory usage.
12. The literature review is far from complete. In addition to [a, b, c], numerous efforts in hierarchical scene parsing is totally missing; see  related work section in [d, e]. As a top-conference paper, a comprehensive literature review is a basic requirement. I believe the reference part should be greatly extended. 
13. In fact, the hierarchical loss function in [d] is superior to the proposed Tree-PATH KL loss, in that it guarantees hierarchy-aware coherent predictions while the proposed loss cannot. A strict quantitative comparison of the two loss functions should be provided.
14. Minor issues include: a) the usage of the terms "interpretability" (L447) and "explainability." b) citation format. c) vector images. d) missing period after Eq. 2. e) the presentation of Eq. 3.
15. A Conclusion section should be added to properly conclude the work and offer insights of downside of impact of the work. 
[a] HLS-FGVC: Hierarchical Label Semantics Enhanced Fine-Grained Visual Classification.
[b] Hierarchical multi-granularity classification based on bidirectional knowledge transfer.
[c] Hierarchical classification at multiple operating points. NeurIPS 2022
[d] Deep Hierarchical Semantic Segmentation. CVPR 2022
[e] LogicSeg: Parsing Visual Semantics with Neural Logic Learning and Reasoning, ICCV 2023

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: The following paragraphs are meant to be an open reflection. I'm not an expert in hierarchical classification, so I may have missed some key parts of the problem, modeling, and overall reflection. 

I think fine-to-coarse works well here because it fits the network architecture, not because it's a better choice overall. Indeed, if the hierarchical structure of the labels is a tree, one could simply put all the effort into classifying the fine level and infer the higher levels by going up the tree, since each class has only one parent. Of course, this would mean that a single error on the fine level would invalidate all other classifications. 

Overall, I think the coarse-to-fine direction makes more sense, since it's all about descending a tree and thus reducing the search field of the lower layers. I just don't think this architecture is suited for that. However, it may be possible to design a sequential architecture in coarse-to-fine logic, for example a U-net. We could imagine a U-net or an encoder-decoder using the CAST structure with a symmetric decoder. It would then be necessary to find a way to divide the super-pixels. The classification will be perform sequentially by going up the decoder

Whether it's one way or the other (but rather the other), I think that the sequential classifications should directly influence each other (like in a Markov chain or a transition table), and not just as different normalizations of the classification probabilities.
soundness: 4
strengths: The paper is well written and organized. The authors have made a good effort to put the problem in the context of the state of the art. The problematic behavior of the state-of-the-art architecture is well illustrated in Figures 1 and 2. In general, the illustrations and explanations allow a non-expert to understand the specificity of hierarchical classification, the state-of-the-art choices, and the resulting problems. I also appreciated the care taken in explaining the metrics to understand what they represent and how they complement each other. 

The solution proposed by the authors makes sense, as they have found that CAST's architecture is very well suited to solve this problem. The explanations and schematics of the architecture are clear, I just had to go to the CAST article to understand how super-pixel pooling is done (which is not critical to understanding the article). The two losses also make sense, as the authors realized that the problem of the state of the art comes from an independent resolution at each level of the hierarchy, and added a loss that forces a (more) globally correct classification. The experimental part (Figure 4, TICE metric) shows that their model makes fewer structural errors (compared to the tree structure), even if it is smaller.

Table 2 and the different ablations clearly show the impact of the improvements made. The last part, which shows that hierarchical learning can improve segmentation, is an interesting addition, which may lead us to think of an opening towards hierarchical segmentation (for which super-pixel-based embedding seems to be well suited).

On a more personal note, at a time when the tendency is to build gigantic, high-consumption networks, I find it appreciable to see methods demonstrating that designing a solution specifically for a problem can increase its efficiency.
summary: This article deals with hierarchical image classification using neural networks. This classification is actually composed of several classifications performed at different levels of precision (coarse to fine). It is important that the classes detected for an image at different levels match the logical organization of the labels. Labels are organized in a tree structure, where each level of precision is a level of the tree. Therefore, there must be a direct path between the different classes identified in an image for the classification to be correct. 

So we understand that in hierarchical classification, there are different levels at which to judge the quality of the classification. At each level, there's the accuracy with respect to the expected label, and there's the logical connection between classifications at different levels.

A simple solution would be to use a single encoder to represent the image and different classification heads for each hierarchical level. The article explains that this is a bad solution because it puts the different hierarchical levels in competition with each other because they require different levels of representation. The current state of the art divides the architecture into independent branches, each of which generates a representation for a different hierarchical level. The authors have found that this separation has led to a non-causality in the classification of the different levels (decisions are independent), which could lead to inconsistencies on the relationship between classifications of different levels. 


The article focuses on CAST, a variant of the transformer network that does not slice the image into patches but into superpixels and integrates a hierarchical structure by including superpixel merging. The architecture thus goes from the fine to the coarse level. Thus, the authors have found that this architecture is adapted to the problem of hierarchical classification by adopting a fine-to-coarse classification logic. Therefore, they have adopted the same architecture, but instead of classifying the different levels in parallel like the state-of-the-art, they classify them sequentially. Thus, the super pixel embedding and the class token pass through this architecture, and in the course of this processing, the class token is classified several times. The classifier heads follow each other and classify at increasingly coarse levels. 

The authors also propose to combine two losses, one which is the cross entropy of the independent classification at each level, and one which is the concatenation of the probability of all classes (renormalized) where several classes are expected (one for each level). Therefore, the losses promote a local and a global level of good classification. 

In the experimental section, the authors show that their sequential classification allows a good improvement in the collection of metrics (local and structural) compared to architectures that process the task in parallel. In addition, parallel architectures are larger because they have to divide the network into as many branches as there are hierarchical levels. The authors show that a smaller network can outperform a larger one if it is designed to match the structure of the problem to be solved. 

The authors also added an ablation study to investigate the advantage of fine-to-coarse over coarse-to-fine, and the influence of both local and structural loss. The authors also show that hierarchical training improves segmentation results.
weaknesses: In part 4.4, I'm not sure how to interpret the result. We can assume that the result is a failure if the superpixel segmentation doesn't make sense. But to me, this just shows that superpixel segmentation is not necessarily adapted to a semantic problem, because it's done at the color level, and semantic in images is not necessarily associated with color.  

In Figures 1 and 4, I find that the space between class names (like "Chimpanzeeferret" in Figure 1) and their position/alignment in the tree can be difficult to read. In two-word classes, you might want to break a line, because at first sight the second part of the name seems to be another class link to another node.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Why KL divergence is better than CE? Both experiments and the theoretical explanation should be provided.
soundness: 2
strengths: 1. The overall presentation is easy to follow. There is no difficulty in understanding the work.
2. The improvement on the baseline is considerable.
summary: This work aims to tackle hierarchical image classification. It is motivated by a hierarchical image segmentation work, which also conducts image classification. This work makes an extension on this basis and proposes a tree KL loss to deliver semantic consistency predictions regarding hierarchy. Evaluation on three datasets verifies the effectiveness over the baseline, which focuses on segmentation rather than classification.
weaknesses: 1. The technical contribution is not enough. Specifically, the proposed method contains visual consistency and semantic consistency. However, the major design and implementation of visual consistency are directly borrowed from CAST [1]. The claim in L260-264 is not convincing. Simply adding supervision in each decoding level can not be considered a vital contribution compared to directly using the overall architecture of CAST.

[1]. Learning hierarchical image segmentation for recognition and by recognition. ICLR 2024

2. The review only contains hierarchical image classification. However, pixel-level hierarchical classification (i.e., hierarchical image segmentation [2,3,4]) can also provide insights for this work. In fact, the visual consistency mentioned in this work is conducting segmentation on the image, and the baseline method (CAST) is also a hierarchical image segmentation work. Therefore, a literature review on hierarchical image segmentation should be included.

[2] AIMS: All-Inclusive Multi-Level Segmentation for Anything. NeurIPS 2023.

[3] Hierarchical Open-vocabulary Universal Image Segmentation. NeurIPS 2023

[4] LOGICSEG: parsing visual semantics with neural logic learning and reasoning. ICCV 2023. 


3. Concatenate labels from all levels to create a distribution has already been explored in [3,4]. The difference is [3,4] using cross-entropy loss, while this work uses KL divergence. 

4. The motivation for using KL divergence instead of cross-entropy is unclear. Since CE loss contains both the KL term to minimize the differences between distribution and a punishment term to minimize the uncertainty. Why KL divergence is better than CE? Both experiments and the theoretical explanation should be provided.

5. The evaluation of the proposed method is limited. 

      i) The comparison to hierarchical counterparts only contains out-of-date methods published before 2022 and the baseline (focusing on segmentation rather than classification). The top-leading solutions (e.g., [5]) are all ignored.

      ii) The label hierarchy is shallow, with only up to 3 hierarchical levels.

[5]. TransHP: Image Classification with Hierarchical Prompting. NeurIPS 2023

6. The claims in 4.6 are similar to the insights provided in [1].

Overall, the technical contribution, evaluation, and insights provided by this work are all limited.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Please specific what is the difference between your work and TransHP?
soundness: 2
strengths: NA
summary: The paper explores hierarchical image classification (HIC) task, which is old but worthy to explore. I have experienced many bad reviews in this domain, for instance, “the proposed method relies on coarse labels and therefore not useful in the real world”. However, I think they are the misunderstanding of this area and I want to give some new one.
weaknesses: Overlap with former work? I noticed that an important reference, TransHP: Image Classification with Hierarchical Prompting (NeurIPS 2023), is not cited in your paper. This work may be directly relevant, as it appears to share similarities with your proposed method, specifically in the use of different ViT blocks for different levels of hierarchy. 

Limited novelty. Your proposed approach introduces elements such as Superpixel and Graph pooling. While these are effective, both are well-established techniques in computer vision. A more detailed explanation of how these additions provide novel contributions within the hierarchical framework would clarify the unique aspects of your work.

Limited evaluation. ImageNet, as a large-scale hierarchical dataset, might provide a stronger test of your method's capabilities compared to the smaller datasets currently used. 

Formatting problem: Please ensure the readability of all components of the paper. For instance, the font size in the tables is small, which may make it difficult for readers to check the data presented

Meta-review: The paper proposes an architecture that extends CAST for hierarchical classification tasks. The proposal fuses superpixels of high similarity using a graph-pooling operation within the ViT tokens. The hierarchical classification is achieved through a set of classification heads per level in the hierarchy.

Strengths:
- Improved results over existing works
- Introduction of new metrics for hierarchical classification
- Reported significant improvements in the experiments
- Ablations confirm the contribution of the added losses

Weaknesses:
- The primary contributions of this work are merely the hierarchical supervision loss and tree KL loss, neither of which can ensure visual consistency
- It is unclear why the model would “ensure that each hierarchical classifier focuses on the same corresponding regions”
- The experiments are somewhat questionable; the reviewers asked to compare against newer methods, but the authors mentioned that the used methods are relevant
- The experiments do not fully compare against the most recent methods
- The paper should be updated to include the latest comparisons and discuss the overlap with existing methods such as CAST and TransHP

The paper received mixed reviews with critical comments due to the limited technical contributions, mainly because it uses existing methods such as CAST, and its similarity to existing approaches like TransHP. I agree with the authors that there are nuanced differences between the approaches and that demonstrating the effectiveness of the proposal in these settings is a contribution in itself. Moreover, as some reviewers mentioned, the method shows improvements over existing methods. While it would be interesting to see experiments on larger datasets, the authors' rationale for selecting the datasets used is sound and follows the literature on hierarchical classification. I also do not agree with requesting more experiments merely for the sake of experimentation. Utilizing existing methods in a new way to exploit instance-level classification and demonstrate its advantages is a contribution in itself. Thus, I recommend the acceptance of the paper.

Decision: Accept (Poster)