Visually Consistent Hierarchical Image Classification
Authors: Seulki Park
Youren Zhang
Stella X. Yu
Sara Beery
Jonathan Huang
Venue: ICLR 2025 Poster (2025)
Keywords: Hierarchical classification
visual grounding
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We highlight the significance of visual consistency in hierarchical classification that classifiers across levels should be grounded in consistent visual cues.

Abstract: Hierarchical classification predicts labels across multiple levels of a taxonomy, e.g., from coarse-level \textit{Bird} to mid-level \textit{Hummingbird} to fine-level \textit{Green hermit}, allowing flexible recognition under varying visual conditions.  It is commonly framed as multiple single-level tasks, but each level may rely on different visual cues.  Distinguishing \textit{Bird} from \textit{Plant} relies on {\it global features} like {\it feathers} or {\it leaves}, while separating \textit{Anna's hummingbird} from \textit{Green hermit} requires {\it local details} such as {\it head coloration}.  

Prior methods improve accuracy using external semantic supervision, but such statistical learning criteria fail to ensure consistent visual grounding at test time, resulting in incorrect hierarchical classification.  We propose, for the first time, to enforce \textit{internal visual consistency} by aligning fine-to-coarse predictions through intra-image segmentation. Our method outperforms zero-shot CLIP and state-of-the-art baselines on hierarchical classification benchmarks, achieving both higher accuracy and more consistent predictions. It also improves internal image segmentation without requiring pixel-level annotations.

Published as a conference paper at ICLR 2025
VISUALLY CONSISTENT
HIERARCHICAL IMAGE CLASSIFICATION
Seulki Park', Youren Zhang', Stella X. Yu! Sara Beery". Jonathan Huang"
'University of Michigan UC Berkeley
*IIT
"Scaled Foundations
{seulki, yourenz, aayllayy] umich. edu, beery edu, amangll1gmaat com
ABSTRACT
Hierarchical classification predicts labels across multiple levels of taxonomy, e.g., from coarse-level Bird to mid-level Hummingbird to fine-level Green hermit, allowing flexible recognition under varying visual conditions. framed as multiple single-level tasks, but each level may rely on different visual cues: Distinguishing Bird from Plant relies on global features like feathers or leaves, while separating Anna's 's hummingbird from Green hermit requires local details such as head coloration. Prior methods improve accuracy using external semantic supervision, but such statistical learning criteria fail to ensure consistent visual grounding at test time, resulting incorrect hierarchical classification. We propose, for the first time, to enforce internal visual consistency by aligning fine- to-coarse predictions through intra- intra-image segmentation. Our method outperforms zero-shot CLIP and state-of-the-ar baselines hierarchical classification bench- marks, achieving both higher accuracy and more consistent predictions. It also improves internal image segmentation without requiring pixel-level annotations.
commonly
INTRODUCTION
Hierarchical classification (Silla & Freitas, 2011; Chang et al., 2021; Jiang et al., 2024) predicts labels along a semantic taxonomy (e.g., Bird - Hummingbird -> Green hermit), rather than choosing from a single flat label set (Fig. 1). This flexibility aligns well with real-world needs: casual observer may only recognize general category like Bird, while biologist might identify the exact species. More importantly, when visual details are lacking such as bird seen from afar, hierarchical models still offer informative predictions (Bird), whereas flat fine- ane-ggaieed classifiers may fail entirely.
Hicrarchical classification is routinely framed as multiple single-level tasks, but each level may depend on different visual cues: Distinguishing Bird from Plant relies on global features like feathers or leaves, while separating Anna's 's humminghird from Green hermit requires local details such as head coloration. Our work is the first to enforce visual consistency within an image improve both the accuracy of hierarchical classification and the consistency of predictions across levels (Fig. I),
External semantic consistency
Internal visual consistency
full- tillttxxooyy prediction
(FGN. CVPR
bird
CAFI, MM 2003,
(H-CAST, ICLR 2025)
preen
hermit
single-level prediction
(HiMulConff 2022,
green
TransHiP NeunPS BIOCUP, CVPR 2024, ...]
(CAST,
hermst
Figure I: We propose enforcing internal visual consistency to improve hierarchical classification across taxonomy levels. Prior works rely on external semantic supervision, statistical criterion that fails to ensure consistent visual focus at test time. Our approach is the first to align predictions through intra-image consistency, improving both accuracy and coherence. Our code is available at https:///g thub. com/ htan/pseulki/heeee

---

Published as a conference paper at ICLR 2025
Correct Prediction
Inconsistent Errors
Aquatic
Prinele
Nesteear Moke
Getment Primacc
White alonk
Chimpanzee
Forret
Feato boa
Puf coel
Showor cop
Coorse
Finc grained
Coarte
Fine-gaaiee
Fine-grained
Fine-gratned
a) Coarse V. Fine /
b) Coarse V. Fine
e) Coarse Fine
d) Coarse Fine X
Figure 2: Incorrect hierarchical classification often results from inconsistent visual grounding across hierarchy levels. We show Grad-CAM visualizations (Selvaraju et al., 2017) of FGN (Chang al., 2021) trained on BREEDS (Entity-30) (Santurkar et al., 2021). a) consistent case: both classifiers focus the same object, with the fine-grained classifier capturing details (bird leg), and the coarse classifier attending to the whole bird. b) The coarse classifier localizes the chimpanzee but the fine classifier fails to attend to its crucial details and makes wrong prediction. The fine-grained classifier correctly identifies the feather boa, while the coarse classifier wrongly attends to the bicycle. d) Both classifiers attend to misaligned areas and make wrong predictions. These cases show that semantic accuracy relies on consistent visual grounding. Our model aligns visual attention across levels, capture different details within coherent region, and predict all four correctly.
Existing works improve fine-grained accuracy by training with externl,, consistent semantic supervi- sion across hierarchy levels (Chang et al., 2021; Chen et al., 2022; Wang et al., 2023a). However, statistical alignment across levels during training does not guarantee visual consistency test time. Such inconsistencies worsen when multiple semantic concepts appear in an image, leading to conflicting predictions that violate the hierarchical taxonomy.
Using Grad-CAM visualizations (Selvaraju et al., 2017) on FGN (Chang et al., 2021) trained on the two-level BREEDS (Entity-30) dataset (Santurkar et al., 2021), we observe that incorrect predictions result from coarse and fine-grained classifiers attending to entirely different regions (Fig. 2). In Fig. 2c, the coarse classifier focuses on the bicycle and predicts Motor Vehicle, while the one-ggained classifier attends to the headwear and predicts Feather Boa instead. In contrast, when both levels focus on related regions, e.g., the whole bird and its leg in Fig. 2a, the predictions are consistent and conrectly aligned with the taxonomy. See quantitative validation of this observation in Appendix A.
We propose enforcing internal visual parsing consistency to achieve semantic coherence and thus more accurate hierarchical classification (Fig. 3). To recognize Green hermit, the fine-grained classifier attends to the beak, wings, and tail, while the coarse classifier attends to their collective
Roct
Prior Works
Our Work
Coarso
Leael
Floot
Inpul Image
Occes Harel
Figure 3: Our method ensures internal visual consistency by aligning coarse and fine classifiers on hierarchical segmentation, unlike prior approaches that rely only external semantic losses without visual grounding. Segmentation outputs show how fine details (c.g.. wings, head, tail) at the 32-way level are grouped into unified bird region the 8-way level. Identical color hues indicate consistent groupings, encouraging the model to attend to coherent image regions.
2

---

Published as a conference paper at ICLR 2025
region recognize the Bird. Our model progressively groups fine details into larger areas and shares features across levels, addressing inconsistencies in prior methods that treat each level independently.
We develop H-CAST, supervised hierarchical extension of CAST (Ke al., 2024), which leverages CAST's coarse-to-fint segmentation for hierarchical classification (Fig. 1). We apply fine- and coarse- grained classification losses early and late stages, respectively, using Tree-path KL. Divergence loss to jointly optimize accuracy across all levels. H-CAST ensures visual consistency with its architecture and propagates semantic errors across levels produce aligned and accurate hierarchical predictions.
H-CAST outperforms state-of-the-art methods with 10% absolute improvement in Full-Path Accu- racy (FPA) compared to ViT-based baselines BREEDS, demonstrating superior consistency and robustness. FPA, our proposed metric, measures the proportion of samples correctly classified at all hierarchy levels. Even vision foundation models like CLIP (Radford et al., 2021) struggle with maintaining hierarchical consistency. Our extensive experiments demonstrate that, H-CAST not only significantly improves hierarchical classification, but also enhances internal image segmentation without requiring pixel-Ievel annotations.
Our contributions. 1) We introduce internal visual consistency, aligning fine and coarse classifica- tion with consistent hierarchical segmentation to ensure classifiers across levels focus on coherent and related regions. 2) We propose a Tree-Path KI. Divergence Loss to enforce semantic consistency across hicrarchy levels. 3) Our method significantly outperforms ViT-based baselines, achieving +3-10% FPA across all datasets, demonstrating superior consistency and robustness.
2 RELATED WORK
Hierarchical classification problem presents a unique challenge: the image remains the same, but the output changes in the semantic (text) space (e.g. "Birds"- "Hummingbird" "Anna's hummingbird"). Due to this structure, prior work has primarily focused on embedding data into the semantic space, using additional loss functions (Bertinetto al., 2020; Zeng et al., 2022) or encoding entire taxonomies as lengthy text inputs, as in BIOCLIP (Stevens et al., 2024). In confrast, we approach the problem from the visual space, exploring how hierarchical classification relates to visual grounding by ensuring consistency across different levels of detail, from fine-grained features broader features. This visual-groundigg perspective is novel and has not been explored in prior work (see Fig. I). Existing hierarchical classification methods can be divided into two approaches:
Single-level prediction: Single-level prediction can be further divided into two approaches. First, fine- fine-grained classification (bottom-up) up) targets fine fine-graine grained classes (leaf nodes). based on the assumption that coarse categories can be inferred using predefined taxonomy (Karthik et al., 2021; Zhang et al., 2022; Wang et al., 2023b; Stevens et al., 2024). While effective for clear images, it struggles with ambiguous test-time conditions when fine-grained classification is impossible (e.g.. birds at high altitudes). Second, the local classifier (top-down) predicts a single label at the most confident level by refining coarse-to-fi Denzler, 2019). However, errors from coarse levels can propagate downward, limiting accuracy. We address this by predicting across the entire taxonomy for greater robustness.
predictions (Deng et al., 2010; Wu et al., 2020; Brust &
2. Full-taxonomy prediction: predicts all levels simultaneously using shared backbone with separate branches (Zhu & Bain, 2017; Wehrmann et al., 2018; Chang et al., 2021; Liu et al., 2022). A key challenge is maintaining label consistency. While Wang al. (2023a) adjusted predictions to enforce consistency, separate branches process images independently, leading to misalignment. To address this, we propose a model based on consistent visual grounding. To the best of our knowledge. no prior work has utilized visual segments to resolve inconsistency in hierarchical classification.
Semantic Segmentation aims to group pixels without pixel-level annotations or using only class labels (Hwang et al., 2019; Ouali et al.. 2020; Ke et al., 2022; 2024). These works employ hierarchical grouping to achieve meaningful segmentation without pixel-level labels. In this field, "hierarchical" refers to part-to-whole visual grouping, where smaller units (e.g., person's 's face or arm) are grouped into larger regions (e.g., the whole body). Based on our intuition that fine- fine-grained classifiers need more detailed information, while coarse classifiers focus on broader groupings, our approach leverages these varying types of visual grouping. To implement this, adopt the recently proposed CAST (Ke et al., 2024), whose graph pooling naturally supports consistent visual grouping. Notably, our work introduces the novel insight that part-to-whole spatial
3

---

Published as a conference paper at ICLR 2025
granularity can align with taxonomy hierarchies (e.g., finer segments for fine-graines labels, coarser segments for coarse labels). a connection not previously explored.
Detailed related work, including Hierarchical Semantic Segmentation is included in Appendix B.
3 CONSISTENT HIERARCHICAL CLASSIFICATION
We improve the consistency and accuracy of hierarchical classification via a progressive learning scheme, where each level builds upon the previous one instead of training separate models per level (Fig. 4). We address two key inconsistencies: visual inconsistency, where classifiers attend to different regions (Fig.3), and semantic inconsistency, where predictions violate the taxonomy (e.g.. -umminnnmm)) We propose H-CAST (Sec. 3.1) for visual consistency, and novel Tree-path KL Divergence loss that encodes parent-child relations to enforce semantic alignment (Sec. 3.2).
Spatial Consistoncy
Semantic Conelstency
Frand
11/31
(9.90)
Hatrarchcca Labst
Heearchical Spalisl-onniilee
Imon-path Dovergenoo coos
Figure 4: Our method implements visually grounded hierarchical classification through visual and semantic modules. The Visual Consistency module uses fine-to-coarse superpixel groupings to ensure classifiers at different levels focus on corresponding regions while capturing different details. The Semantic Consistency module encodes label hierarchies align predictions across levels. Together, they encourage cooperative learning across the hierarchy and improve overall performance.
3.1 H-CAST FOR VISUAL CONSISTENCY
The areas of focus within the image need to differ when conducting classification at the fine ine-grained level compared the coarse level. When distinguishing between amilar-looking species (e.g.. "Green Hermit" vs. "Anna's Hummingbird"), the fine- fine-grained recognition requires attention to fine details like the bird's beak and wings; meanwhile, at the coarse level (e.g., "bird" vs. "plant"), the attention shifts to larger parts such the overall body of the bird. However, this shift in focus towards larger objects does not imply a sudden disregard for the previously focused details and search for new larger objects. Rather, natural approach involves combining detailed features such as the bird's beak, belly, and wings for accurate bird recognition. Therefore, we argue that the hierarchical model should be grounded in consistent visual cues. From this insight, design model where the details learned at the fine level (e.g., bird's beak and wings) are transferred to the coarse level as broader parts (e.g., bird's body) through consistent feature grouping.
For internally consistent feature grouping, we build upon recent work CAST (Ke et al., 2024). CAST develops hierarchical segmentation from fine coarse, an internal part of the recognition process. However, their segmentation is driven by a flat recognition objective at the very end of visual parsing. We extend by imposing fine-to-coarse semantic classification losses at different stages of segmentations throughout the visual parsing process. Our design reflects the intuition that finer segments can be helpful in capturing fine- ane-grained details (e.g., beaks and wings) required for fine-grained recognition, whereas coarser segments can be effective in representing broader features (e.g., the body of a bird) needed for coarse- grainedd recognition. We have a single hierarchical recognition grounded on internally consistent segmentations, each driven by classification objective at a certain granuality. We refer to our method as Hierarchical CASS (H-CAST).
4

---

Published as a conference paper at ICLR 2025
Consider hierarchical recognition task where x denotes image associated with hierarchical labels 91.... VL, encompassing a total of L levels in the hierarchy. Level L is the finest level (i.e. leaf node), and Level 1 is the coarsest level (i.e, root node). Then, given an image z, the hierarchical image recognition task is to predict labels at all levels across the hierarchy.
Let Zi and St denote the feature and segment representations at the I-th level. We extract superpixels from image x using SEEDS (Van den Bergh et al., 2012), which groups pixels by color and local connectivity. These superpixels serve as ViT input in place of fixed patches and form the initial (finest) segments S1+11- Each feature Zi consists of class tokens (Z,lass) and segment tokens (Z( Graph pooling (Ke et al., 2024) then merges similar segments, enabling features to capture increasingly global context from Σμ to Z1. For hierarchical recognition, we add classification head (fi) consisting of a single linear layer each level. Then, we define the hierarchical visual consistency loss as the sum of I, cross-entropy losses (LcE), denoted as
LHV Σ LCE(fi(Zgten) y1).
(1)
H-CAST applies hierarchical supervision across all levels, unlike CAST, which uses flat supervision only the final class token. This design allows labels different levels to inform one another. In Sec. 4.6, we show its effectiveness over alternatives, including reverse coarse-to-fin supervision.
3.2 TREE-PATH KL DIVERGENCE LOSS FOR SEMANTIC CONSISTENCY
To improve semantic consistency across hierarchy levels, we propose Tree-path KI. Divergence loss that imposes tree path relationships between labels. We first concatenate labels from all levels to create distribution, as Y = L i..:: J, where L represents the one-hot encoding for level l. Next, we concatenate the outputs of each classification head and then apply the log softmax function (LogSoftmaxx. We use Kullback-eeibler divergence loss (K to align this output with the ground truth distribution Y. Then, TK loss is calculated as follows.
LTK K (LogSoofftaxx (1
f1(2(lass))) ())
This loss penalizes predictions that do not align with the taxonomy by simultaneously training on multiple labels within the hierarchy. Therefore, despite the simplicity, TK loss enables the model to enhance semantic consistency through this vertical encoding from the root (parent) node of the hierarchy level to the leaf (children) node. Our final loss becomes as follows, where is a hyperparameter to control the weight of LTK.
L = LAv + +LLK.
EXPERIMENTS
We first show that hierarchical classification remains challenging even for vision foundation models, which often yield inconsistent predictions. Our method outperforms existing approaches and flat baselines on benchmark datasets. We further validate our design through ablations and demonstrate that hierarchical supervision also benefits semantic segmentation.
4.1 EXPERIMENTAL SETTINGS
Datasets. We use widely used benchmarks in hierarchical recognition: BREEDS (Santurkar et al., 2021), CUB-200- 2011 (Welinder et al., 2010), FGVC-Aireraft (Maji et al., 2013), and iNat21- Mini (Van Horn et al., 2021). BREEDS, a subset of ImageNet (Russakovsky al., 2015). includes four 2-level hierarchy datasets with different depths/parts based on the WordNet (Miller, 1995) hierarchy: Living- 17, Non-Living-26,, Entity-13, Entity-30. For BREEDS, we conduct training and validation using their source splits. BREEDS provide wider class variety and larger sample size than CUB-200-200 and FGVC-Airrraft making it better suited for evaluating generalization performance. CUB LUB-200-201 200-2001 comprises a 3-vell hierarchy with order, family, and species; FGVC-Aircres consists of a 3-level hierarchy including maker, family, and model (e.g., Boeing Boeing 707 - 707-320 ); For experiments provided in Sec. E.4. Table 4 in Appendix summarizes a description of the datasets.
larger dataset, used the 3-level iNat21-Mini. Details of the
are
Evaluation Metrics. We evaluate our models using metrics for both accuracy and consistency.
5

---

Published as a conference paper at ICLR 2025
level-accuracy: the proportion of correctly classified instances each level (Chang al., 2021). Ni weighted average precision (wAP) (Liu al., 2022): WAP Σi= Σg=, Ne Pt, where N₁ and Pa
denote the number of classes and Top-1 classification accuracy at level 1, respectively. This metric considers the classification difficulty across different hierarchies.
Tree-based In InConsistency Error rate (TICE) (Wang et al., 2023a): TICE nic/N, where Vie denotes the number of samples with inconsistent prediction paths, and N refers to the number of all test samples. This tests whether the prediction path exists in the tree (consistency).
Full-Path Accuracy (FPA): FPA = Rac/N, where nac refers to the number samples with all level of labels correctly predicted. This metric evaluates both accuracy and consistency, ultimately representing our primary metric of interest.
The difference between FPA and TICE is illustrated in Table 6 in Appendix.
Comparison methods. First, we evaluate our H-CAST with representative models in hierarchical classification, FGN (Chang et al., 2021) and HRN (Chen al., 2022). FGN uses level-specific heads to avoid negative transfer across granularity levels, while HRN employs residual connections to capture label relationships and a hierarchy- based probabilistic loss. We also compare TransHP (Wang et al., 2023b), a ViT-based model that learns prompt tokens to represent coarse classes and injects them into an intermediate block to enhance ane-grrined predictions. Lastly, we compare Hier- ViT, variant without visual segments and TK loss. Like our approach, Hier-ViT trains each hierarchy level using the class token from the last 6, 9, 12 blocks. To establish ceiling baseline, we compare with flat models trained at single hierarchy level. Flat-ViT classifies one level using the ViT class token, while Flat-CAST trains independent models for each level using the CAST architecture (Ke et al., 2024). We also compare with Hierarchical Ensembles, HiE (Jain et al., 2023), which improves fine-grained predictions via post-hoc correction using a coarse model. Note that flat models require separate models for each hierarchy level, leading to increased storage and training costs. For baseline methods, we use the official codebases and their reported optimal settings. All models are trained for 100 epochs, except HP, which is trained for 300 epochs in the original paper. Details on the architecture and hyperparameter settings for H-CAST can be found in Appendix D.
4.2 HIERARCHICAL CLASSIFICATION WITH VISION FOUNDATION MODELS
First, to demonstrate that longstanding hierarchical classification not easily solved by today's vision foundation models, we evaluate CLIP (Radford et al., 2021)'s performance on the 2-level BREEDS dataset. The top row of Fig. 5 shows prediction rates on the test set, while the bottom row presents
Courra Connec graieed Comert
Czaree Dosugnaieed Incomet
Courra Incoment Cinegg Connet
Courre Insomeet www.grare
2eo CUP
Zer CUP
HIGHT
Dervves
Equipment
Equipmnnt Gament
Gorment
Craft
Linar
Montor
Mirskert
Bioketball
Cloak
Samoyed
a) Coarse / Fine
b) Coarse V Fine X c) Coarse X. Fine
d) Coarse X, Fine X
Figure 5: Vision foundation models struggle with consistent predictions in hierarchical classifi- cation. We evaluate CLIP (Radford et al., 2021) on the 2-level BREEDS dataset (top) and present misclassification examples from Entity- (bottom). a) CLIP struggles to maintain consistency and correctness, achieving only about 50% accuracy on Entity- b) CLIP more frequently predicts the coarse category correctly while misclassifying the fine-grained category compared H-CAST across all datasets. c) CLIP often predicts the fine-grained category correctly but fails at the coarse level, a mistake that is rare in H-CAST, suggesting difficulty grasping broader conceptual understanding. H-CAST accurately predicts cases a-c. d) Both CLIP and H-CAST fail in complex scenes.

---

Published as a conference paper at ICLR 2025
examples from the Entity-13 dataset. Even considering the zero-shot prediction, Fig. 5 (a) shows that the overall ratio of correct predictions for both coarse and fine- ane-grained classifications is significantly low, with only around 50% accuracy on the Entity-13 dataset. Fig. 5 (c) further highlights significant errors in coarse predictions when addressing broader concepts. Our findings support the recent study from Xu al. (2024) that VLMMs excel at fine-graine graiind prediction but struggle with general concepts. This highlights the ongoing challenges of hicrarchical classification, even with vision foundation models. Furthermore, when we examine the misclassification examples in bottom row of Fig. we can see that CLIP focuses on different object areas for coarse and fine- -eniiin predictions. For example, in (b), CLIP predicts "Equipment" in the coarse prediction but predicts "Miniskirt" in the fine-grained prediction instead of "Monitor" (a child of Equipment). However, our model, based on consistent visual segments, can make correct predictions in all cases.
4.3 CONSISTENT HIERARCHICAL CLASSIFICATION ON BENCHMARKS
Table 1 shows benchmarks on BREEDS. H-CAST outperforms both hierarchical (FGN, HRN, TransHP, ViT) and flat (ViT, CAST, HiE) baselines. It exceeds ViT-based models like ViT and TransHP by over 11 points, highlighting the effectiveness of our visual grounding and Tree-Path Loss beyond simply adding hierarchy supervision. While TransHP uses coarse labels as prompts to aid fine-grained prediction, it does not jointly optimize both levels, resulting in lower consistency. H-CAST also improves FPA by 4.3-6.4 6.4 points over HRN, despite having far fewer parameters.
Table I: H-CAST achieves both high consistency and accuracy, outperforming both hierarchical and flat baselines on BREEDS. It achieves a 4.3-6.4 percentage point gain in FPA metric over HRN with significantly fewer parameters. Additionally, H-CAST surpasses Hier- ViT and TransHP by over 11 percentage points, demonstrating that its success is due to our consistent visual grounding and Tree-path Loss, rather than adding hierarchy supervision to VIT backbone. (Higher the metric is the best, except TICE.) 'VIT-S' refers to ViT-Small, while 'RN-50' denotes ResNet-50.
Configuration
Living. 17 (17-34)
Non-I Living-20 26 (26-52)
hackbone
#params
FPA
fine
WAP
TICE
FPA
fine
WAP
TICE
Flat-VIT Flat-VIT + HIE
VII-S
65.0M
66.24
75.71
72.06
73.28
17.11
57.46
67.50
65.73
57.46
23.27
VIT-S
65.0M
67.59
75.71
71.35
72.81
9.88
59.73
67.50
65.31
66.04
13.69
Flat CAST
VIT-S
78.5M
78.82
88.06
82.88
84.61
8.82
76.17
84.77
81.08
82.31
11.77
Fla-CCAS + HRE
VIT-S
78.5M
81.59
88.06
83.24
84,85
5.18
79.23
84.77
81,39
82.51
6.19
FGN
RN-50
24.8M
63.82
72.59
68.00
69.53
12.12
60.81
69.46
65.77
67.00
16.46
Hienuchy
HRN
RN-50
70.8M
79.18
87.53
81.47
83.49
6.29
76.31
82,38
80.15
80,90
9.54
HTT
VIT-S
21.7M
74.06
80.94
74.88
76.90
10.50
72.04
73.31
70.03
12.45
TransTIP
VIT-S
21.7M
74.35
83.00
76.65
78.76
8.35
68.62
77.31
72.31
73.97
13:04
(ALCCATT)
VIT-S
83.15
84.73
5.26
Our Gaint over SOTA
45.94
+3.29
+3.77
+3.61
+3.10
46.36
+5.51
+3.00
+3.83
+4.28
Configuration
Eatity-13 -1
(13-130)
Entity-30
(30-120)
backhone
Aparams
FPA
coarse
fine
wAP
TICE
FPA
course
fine
wAP
TICE
Flat-VT
VIT-S
65.0M
64.22
76.28
76.06
76.08
21.33
66.93
76.28
74.35
74.77
18.75
Flat
Flat. Dat-ViT+ + HiF
VIT-S
65.0M
65.20
76.47
74.91
75.05
15.68
68.77
76.47
73,92
74.43
11.08
Flat CAST
VIT-S
78.5M
78.63
87.80
83,72
84,09
10.65
82.67
87.89
83,15
84.73
5.26
Fla-CASTT + HIE
VIT-S
78.5M
79.52
87.80
83:40
83,80
6.83
83,70
87.89
84.30
85.02
4.20
FGN
RN-50
24.8M
74.23
85.35
78.00
78.67
9.43
68.52
77.47
73.18
74.04
13.62
Hierarchy
HRN
RN-50
70.8M
81.43
90.00
84.48
84.98
6.34
79.85
86.57
83,35
83.99
8.38
Hier-ViT
VIT.S
21.7M
74.63
86.95
75.39
77.70
5.19
73.01
81.38
74.10
74.76
11.61
TrandIP
VIT-S
21.7M
73.45
86.28
76.23
77.14
8.80
72.00
80.78
75.63
76.66
12.20
Ours (H-CAST)
VII-S
26.2M
85,68
93.42
86.15
87.60
1.69
84,83
90.23
85,45
85,88
2,57
Our Goins over SOTA
+4.25
+3.42
+1.67
+2.62
+4.65
+4.98
+3.66
+2.10
+1.89 1.89
+5.81
Note that flat models train separate networks per level, incurring higher memory and training costs. H- CAST outperforms them, demonstrating its efficiency and effectiveness for hierarchical classification. Similar results are observed Aircraft and CUB (Appendix E.5), with additional evaluation on the larger iNaturalist 2021-Mini dataset in Appendix E.4.
4.4 VISUALIZATIONS OF STRUCTURED VISUAL PARSING
Well-structured visual parsing is crucial for accurate hierarchical classification. H-CAST improves hierarchical classification by ensuring structured visual parsing, where fine details pro- gressively merge into meaningful objects coarser levels. Fig. 6 visualizes this process the
7

---

Published as a conference paper at ICLR 2025
(a) Full-Path Correct Predictions
(b) Full-Path Incorrect Predictions
Figure 6: Well-structured visual parsing enables accurate hierarchical classification, while frag- mented segmentation correlates with misclssiffcation. We compare fine-to-coarse segmentation for fully correct (a) and incorrect (b) predictions on the Entity-30 dataset to show how visual parsing quality affects classification. For correct predictions (a), fine details merge into coherent objects at coarser levels, preserving structure. For example, in the correctly predicted shoe image, the shoe and ankle form a unified segment, maintaining object integrity. In contrast, for incorreet predictions (b), segmentation is fragmented and inconsistent. The misclassified shoe image shows scattered segments blending with the background, making it difficult to recognize the overall shape. Likewise, the beer glass lose key features due unstable grouping. These results highlight that accurate hierarchical classification relies on structured visual parsing.
BREEDS Entity-30 dataset. In full-path correct predictions (Fig. 6, Left), fine-level details are effectively grouped into coherent objects. For example, in correctly predicted shoe image, the shoe and ankle merge into single segment, maintaining object integrity. Similarly, fine details in the earlier segments merge into unified face, eyes, and nose at the coarse level, demonstrating structured feature grouping. Conversely, in full-path incorrect predictions (Fig. Right), segmentation is fragmented and inconsistent. The misclassified shoe image shows scattered segments blending with the background, while even a simple-shaped object like a beer glass is broken into disconnected parts, losing its structural coherence. This highlights that structured visual parsing is key to accurate classification, while fragmented segmentation leads to errors.
the dog example,
4.5 EFFECT OF VISUAL GROUNDING ON HIERARCHICAL CLASSIFICATION
H-CAST Ensures Consistent and Class-Relevant Feature Learning. We compare nearest neigh- bors of class token features at fine and coarse levels for queries correctly classified at both levels on Entity-30 (Fig. 7). If model truly learns discriminative class features, its nearest neighbors should remain consistent across levels. However, Hicr-VIT often retrieves visually similar but semantically incorrect images, while H-CAST consistently selects class-relevant neighbors at both levels. For example, Hier- VIT retrieves person in black uniform the closest fine-level neighbor for Italian Greyhound, likely due to clothing color similarity (top left). Similarly, for a curly-haired Bedlington Terrier, selects a fur coat (bottom left), suggesting it relies on superficial textures rather than class-specific features. In contrast, H-CAST consistently retrieves the correct class, selecting a visually similar Italian Greyhound and Bedlington Terrier at both levels, ensuring more stable and meaningful feature learning. These results show that Hier- ViT lacks visual consistency across levels, while H-CAST maintains class-relevant and hierarchically aligned retrievals.
4.6 ABLATION ANALYSIS OF ARCHITECTURE DESIGN AND LOSS FUNCTION IN H-CAST
Fine-to-Coarse Coarse-to-Fime learning. Our model adopts a Fine-to- Coarse (F-cC) learning strategy, first learning fine labels in the lower block and progressively integrating coarser labels. This contrasts with prior methods, which typically learn coarse features first (Zhu & Bain, 2017; Yan et al., 2015; Wang et al., 2023b). To evaluate its effectiveness, we compare F-+ with two baselines: Coarse-to-Fine (C-F)), which follows a conventional hierarchy by learning coarse labels first, and Fine-Coarse Merging (C+F), which combines fine and coarse features across blocks. For fairness, we exclude Tree-path KL Divergence loss in these comparisons. Table (FrVC-Airrraft) shows that C-+F yields the lowest fine-gra fine-grained accuracy, while C+F slightly improves accuracy but adds significant parameter overhead. In contrast, F-+C balances simplicity and strong performance.
8

---

Published as a conference paper at ICLR 2025
Hier- ViT
H-CAST
Query
Fine
Coarse
Fine
Coarse
Figure 7: Consistent visual grounding enhances feature learning in hierarchical classification. We compare nearest neighbors of class token features at fine and coarse levels for queries correctly classified by H-CAST both levels (Entity-30). Rows 1,2) Hier-ViT retrieves person in black uniform or black hair at the fine level. This raises doubts about whether the model truly learned the dog's features. Row 3) The query bird has gray wings with yellow body. H-CAST retrieves gray- winged bird at the fine level and yellow-bodied bird at the coarse level, capturing fine-level detail and coarse-level semantics. Hier-Vit ViT lacks this distinction. Row 4) The query shows two llamas on green field. Hier- VIT retrieves two oxen in a similar setting, resulting in fine-level misclassification. H-CAST retrieves true llama images: one showing both white and brown llamas at the fine level, and typical Ilama at the coarse level. This shows that H-CAST attends to class-relevant features while maintaining hierarchical consistency. Green image borders indicate same-class retrievals; red image borders indicate mismatches.
making it an effective choice for hierarchical classification. Additionally, attention visualizations (Appendix E.2) reveal that lower blocks focus on fine details, while upper blocks capture broader structures, demonstrating that our design effectively guides attention across hierarchy levels.
Ablation Studies on the Proposed Losses. We conduct two ablation studies to evaluate our loss functions. First, we assess the individual contributions of Hierarchical Opatial-consistens loss LHs
Table 2: Ablation studies of learning design and loss functions on Aircraft data.
Direction
C-+F
C+F
F-+C
Loss abl.
LrK
Tas
Both
Consis.
Flat.
BCE
KI. Div.
FPA
82.01
81.76
82.66
FPA
82.48 82.66
83.72
FPA
82.87
82.18
83.72
maker
93.16
93.52
94.27
maker
94.30
94.27
94.96
maker
94.63
94.21
94.96
family
89.92
90.31
90.19
family
90.37
90.19 91.39
family
90.94
90.13
91.39
model
84.10
84.58
84.40
model
84.04
84.40 85.33
model
84.97
84.88
85.33
wAP
87.50
87.93
87.91
wAP
87.80 87.91 88.90
wAP 88.51 88.11
88.90
design achieves best performance.
(a)
"Coarse + Fine'
(b) Utilizing both losses yields best performance.
(c) KL Divergence loss outper- forms alternative losses.

---

Published as a conference paper at ICLR 2025
CAST
H-CAST
GT
CAST
H-CAST
GT
Living- 17
Entity-30
Emtit-13 13
Region BREEDS with ground truth segmentation
miOU Living 17
Non Living 26
Fintity 30
Entily
H-CAST
32.83
22.12
22.08
16.19
Figure 8: H-CAST improves segmentation by leveraging hierarchical taxonomy. We visualize segmentation results on BREEDS dataset and measure the region mIOU of fine-level objects for samples with segmentation ground truth (GT) from ImageNet-S (Gao et al., 2022). In the visualized images, we can observe that H-CAST better captures the overall shape in more coherent manner compared to CAST. In quantitative evaluation, H-CAST outperforms CAST in most datasets despite using coarse-level supervision for the last-level segments whereas CAST employs fine-level supervi- sion. surprising find that the taxonomy hierarchy can help part-to-whole segmentation.
and Tree-path KL Divergence loss LTK on Aireraft. Table 2b shows that both losses significantly enhance performance, with their combination achieving the best accuracy and consistency. Next, examine the effect of different loss functions by replacing KI. Divergence loss with Binary Cross Entropy (BCE) and Flat Consistency loss. BCE directly substitutes the KI. divergence component, while Flat Consistency loss, inspired by a bottom-up approach, infers coarse labels from fine predic- tions using BCE. As shown in Table 2c, KL Divergence loss achieves the highest FPA, demonstrating superior accuracy and consistency. Additional results on Living- 17 are in Appendix E.3.
4.7 ADDITIONAL BENEFITS OF HIERARCHICAL CLASSIFICATION FOR SEGMENTATION
Hierarchical semantic recognition enhances segmentation. Although our primary focus is hier- archical recognition, we investigate whether incorporating hierarchical label information can also improve segmentation. Fig. 8 provide a qualitative and quantitative comparison between H-CAST and CAST. H-CAST, which uses varying granularity supervision for segments, outperforms CAST, which employs fine-grained level supervision, on most datasets such as Living- 17, Non-Living- and Entity-13. The visualized results show that H-CAST better captures the overall shape in more coherent manner compared to CAST. These findings demonstrate that utilizing hierarchical taxonomy benefits not only recognition but also segmentation. Details of the evaluation method and more visualization comparison with CAST are included in the Appendix E.7 and Fig. 11.
5
SUMMARY
We tackle inconsistent predictions hicrarchical classification by introducing consistent visual grounding, leveraging varying granularity segments align coarse and fine-grained classifiers. Unlike existing methods that soly rely on external semantic constraints, our approach leverages varying granularity segments guide hierarchical classifiers, ensuring they focus on coherent and relevant regions across all levels. This leads to significant improvements across benchmarks, setting new standard for robust hierarchical classification.
ACKNOWLEDGEMENT
This project supported, in part, by NSF 2215542. NSF 2313151, Berkeley Al Research grant with Google, and Bosch gift funds to Yu at UC Berkeley and the University of Michigan, with partial compute support from NAIRR Pilot (CIS240431, CIS250430).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A Lord. Making better mistakes: Leveraging class hierarchies with deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
Clexens-Alexander Brust and Joachim Denzler. Integrating domain knowledge: using hierarchies to improve deep classifiers. In Asian conference on pattern recognition. Springer, 2019.
Dongliang Chang, Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song, and Jun Guo. Your" flamingo" is my" bird": fine-grained, not. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Jingzhou Chen, Peng Wang, Jian Liu, and Yuntao Qian. Label relation graphs enhanced hierarchical residual network for hierarchical multi-granule granularity classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2020.
Jia Deng, Alexander C Berg, Kai Li, and Li Fei-Fei. What does classifying more than 10,000 image categories tell us? In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part V 11. Springer, 2010.
Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings. Part 1 13, 2014.
Shanghua Gao, Zhong- Yu Li, Ming-Hsuan Yang, Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. TPAMI, 2022.
Ashima Garg, Depanshu Sani, and Saket Anand. Learning hierarchy aware features for reducing mistake severity. In European Conference on Computer Vision, 2022.
Vivien Sainte Fare Garnot and Loic Landrieu. Leveraging class hierarchies with metric-guided prototype learning. arXiv preprint arXiv:2007.03304 2020.
Ju He, Jieneng Chen, Ming-Xian Xian Lin, Qihang Yu, and Alan L Yuille. Compositor: Bottom-up cluster- ing and compositing for robust part and object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, 2023.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, 2018
Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang, Xiao Zhang, and Liang-Chite Chich Chen. Segsort: Segmentation by discriminative sorting of segments. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2019.
Kanishk Jain, Shyamgopal Karthik, and Vineet Gandhi. Test-time amendment with a coarse classifier for fine-grained classification. In Thirth-seventh Conference on Neural Information Processing Systems, 2023.
Juan Jiang, Jingmin Yang, Wenjie Zhang, and Hongbin Zhang. Hierarchical multi- gratiiiranularity classification based on bidirectional knowledge transfer. Multimedia Systems, 2024.
Shyamgopal Karthik, Ameya Prabhu, Puneet K. Dokania, and Vineet Gandhi. No cost likelihood manipulation at test time for making better mistakes in deep networks. In International Conference Learning Representations, 2021.
11

---

Published as a conference paper at ICLR 2025
Tsung-Wol Wei Ke, Jyh-Jing Hwang, Yunhui Guo, Xudong Wang, and Stella X Yu. Unsupervised hierarchical semantic segmentation with multiview cosegmentation and clustering transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Tsung- Wei Ke, Sangwoo Mo, and Stella X. Yu. Learning hierarchical image segmentation for recog- nition and by recognition. In The Twelfth International Conference on Learning Representations, 2024.
Liulei Li, Tianfei Zhou, Wenguan Wang, Jianwu Li, and Yi Yang. Deep hierarchical semantic segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2022.
Liulei Li, Wenguan Wang, and Yi Yang. Logicseg: Parsing visual semantics with neural logic learning and reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kumpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculeseu. Open-vocabulay semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Yang Liu, Lei Zhou, Pengcheng Zhang, Xiao Bai, Lin Gu, Xiaohan Yu, Jun Zhou, and Edwin R Hancock. Where to focus: Investigating hierarchical attention relationship for fine-grained visual classification. In European Conference on Computer Vision. Springer, 2022.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 1995.
Yassine Ouali, Céline Hudelot, and Myriam Tami. Autoregressive unsupervised image segmentation. In Computer Vision-EC 2020: 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings, Part VII 16. Springer, 2020.
Lu Qi, Jason Kuen, Weidong Guo, Jiuxiang Gu, Zhe Lin, Bo Du, Yu Xu, and Ming- Hsuan Yang. Aims: All-inclusive multi-level segmentation for anything. Advances in Neural Information Processing Systems, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. (BREEDS): Benchmarks for subpopu- lation shift. In International Conference on Learning Representations, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In Proceedings of the IEEE international conference on computer vision, 2017.
Carlos N Silla and Alex A Freitas. A survey of hierarchical classification across different application domains. Data mining and knowledge discovery. 2011.
Rishubh Singh, Pranav Gupta, Pradeep Shenoy, and Ravikiran Sarvadevabhatla. Float: Factorized learning of object attributes for improved multi-object multi-part scene parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1445-1455, 2022.
Samuel Stevens, Jiaman Wu, Matthew Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger- Wolf, Wei-Lun Chao, and Yu Su. Bioclip: A vision foundation model for the tree of life. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
12

---

Published as a conference paper at ICLR 2025
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Jack Valmadre. Hierarchical classification at multiple operating points, Advances in Neural Informa- tion Processing Systems, 2022.
Michael Van den Bergh, Xavier Boix. Gemma Roig, Benjamin De Capitani, and Luc Van Gool. Seeds: Superpixels extracted via energy-driven sampling. In Computer EsionnEECV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VII 12, 2012.
Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021.
Rui Wang, Cong Zou, Weizhong Zhang, Zixuan Zhu, and Lihua Jing. Consistency- -wware feature learning for hierarchical fine-grained visual classification. In Proceedings of the 31st ACM International Conference on Multimedia, 2023a.
Wenhao Wang, Yifan Sun, Wei Li, and Yi Yang. TransHP: Image classification with hierarchical prompting. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b.
Xudong Wang, Shufan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Dar- rell. Hierarchical open- vvcabuocayylary universal image segmentation. Advances in Neural Information Processing Systems, 2024.
Jonatas Wehrmann, Ricardo Cerri, and Rodrigo Barros. Hierarchical multi-label classification networks. In International conference on machine learning. PMLR, 2018.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report
California Institute of Technology, 2010.
Tz-Ying Wu, Pedro Morgado, Pei Wang, Chih-Hui Ho, and Nuno Vasconcelos. Solving long-tailed recognition with deep realistic taxonomic classifier. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part VIII 16, 2020.
Zhenlin Xu, Yi Zhu, Siqi Deng, Abhay Mittal, Yanbei Chen, Manchen Wang. Paolo Favaro, Joseph Tighe, and Davide Modolo. Benchmarking zero-shot recognition with vision-language models: Challenges on granularity and specificity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, 2024.
Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition. In Proceedings of the IEEE international conference on computer vision, 2015.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, 2019.
Siqi Zeng, Remi Tachet des Combes, and Han Zhao. Learning structured representations by em- bedding class hierarchy, In The Eleventh International Conference on Learning Representations, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412. 2017.
Shichuan Zhang, Sunyi Zheng, Zhongyi Shui, and Lin Yang. HIs-fgve: Hierarchical label semantics enhanced fine- -raigrr visual classification. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.
13

---

Published as a conference paper at ICLR 2025
Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ramaiah. Use all the labels: A hierarchical multi- label contrastive learning framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Xinqi Zhu and Michael Bain. B-cnn: branch convolutional neural network for hierarchical classifica- tion. Xiv preprini htt::/709.00000 2017.
14

---

Published as a conference paper at ICLR 2025
Appendix
A QUANTITATIVE EVIDENCE FOR CONSISTENT VISUAL GROUNDING
To quantitatively validate our observation that inconsistent predictions often occur when coarse and fine-grained classifiers focus on different regions in Figure 3, we analyze the Grad-CAM (Selvaraju et al., 2017) heatmaps of these classifiers. Specifically, compute two metrics: the overlap score and the correlation score.
The overlap
quantifies the degree to which the regions activated by the two classifiers coincide. For each heatmap, we define a significant region as the set pixels where activation values exceed a threshold. Specifically, the overlap count (O) measures the number of overlapping pixels between heatmaps A and B, where both values exceed a threshold (r = 0.001). is defined as:
0 Σ (Ma(..)) A MB(i.j)),
(4)
S.
where (1(1,)) j) and (i, are binary masks indicating significant regions in A and B, respectively. These masks are defined as:
Ma(1,)) - {1 if A(i,)) MB(1,)) = =(0 if B(i,)) otherwise,
(5)
otherwise.
The correlation score measures the linear relationship between the activation values of the lapping regions in the two heatmaps. Let A₃ and Be the values the overlapping regions, then correlation score is computed as:
Ei (Ax PA)(Bx - -MB)
R EA=1(A - pa)2 Σh_ (Br {BB
(6)
where n is the number overlapping pixels, MA is the mean of (As), and μıB is the mean of (BR)-
Higher overlap and correlation scores indicate stronger agreement between the regions attended to by the two classifiers. Conversely, lower scores highlight a lack of alignment in their focus.
Interestingly, empirical results from the FGN model (Chang et al., 2021) the Entity-30 dataset show that when both classifiers make correct predictions, the overlap and correlation scores are significantly higher. In contrast, incorrect predictions correspond to notably lower scores, shown Table 3. These findings support our hypothesis that aligning the focus of coarse- and fine-graine ane-grained classifiers enhances both prediction accuracy and consistency.
Table Overlap and correlation scores between coarse and fine-grained Grad-CAM heatmaps. This shows that correct predictions correspond to higher overlap and correlation between coarse and fine-grained classifiers, highlighting the importance of aligning classifier focus for accuracy and consistency.
Fine-grained
Fine-grained
Overlap
Correlation
True
False
True
False
True 0.51 0.20 0.25 + 0.13
Coarse True 0.70 0.26 0.02 + 0.40
Coarse
0.336 = 08.37 + 0.19
False 0.30 + 0.42 0.35 0.41
(a) Overlap Scores
(b) Correlation Scores
15

---

Published as a conference paper at ICLR 2025
B ADDITIONAL RELATED WORK
Hierarchical classification can be divided into single-level prediction and full-taxonomy predic- tion based on the output structure. Single-level prediction is further categorized into fine-grained classification and local classifier approaches.
1) Single-level prediction:
1-1) The fine-grained classification (bottom-up) approach focuses on predicting fine- tineggrained classes (e.g.. Icaf nodes) by leveraging taxonomy (Deng al., 2014; Zhang al., 2022: Zeng et al., 2022). It is often referred to as bottom-up method because higher-level coarse classes can be inferred from the predicted fine-grained classes. Various methods have been proposed to effectively use hierarchical information. For example, hierarchical cross-entropy (HXE) loss (Bertinetto et al., 2020) reweights cross-entropy terms along the hierarchy tree based on class depth. Inspired by transformer prompting techniques, TransHP (Wang et al., 2023b) introduced coarse-class prompt tokens to improve fine-grae finegrained classification accuracy. Recently, BIOCLIP (Stevens et al., 2024), trained on large-scale Tree of Life data, achieved superior few-shot and zero-shot performance using a CLIP (Radford et al., 2021) contrastive objective on text combining fine-grained and higher-level classes. One of the actively studied topics is minimizing "mistake severity" (e.g., the tree distance between incorrect predictions and the ground truth) (Bertinetto et al., 2020; Karthik et al., 2021; Garg al., 2022).
However, while effective on clear and detailed images, this approach struggles in real-world scenarios where fine-grained predictions are challenging (e.g.. birds flying high altitude), leading to incorrect predictions at higher levels. To address this, we propose a model that predicts across the entire taxonomy, which we believe provides greater robustness in practical applications.
1-2) The local classifier (top-down) approach leverages local information, such as higher-level class predictions, to make predictions at the next level. This design allows predictions at arbitrary nodes by stopping the inference process when a certain decision threshold is met, leading to more reliable predictions at higher levels (Deng et al., 2010; Wu et al., 2020; Brust & Denzler, 2019). As trade-off (Valmadre, 2022). While a single model is commonly used, HiE (Jain et al., 2023) adjusts fine-level predictions post-hoc using coarse predictions from independently trained classifiers. However, a disadvantage of this top-down approach is the propagation of errors from higher-level predictions to lower levels.
result, these methods emphasize metrics such as the correctness-ppecfiiie
2) The full-taxonomy prediction (global classifier) approach aims to predict the entire taxonomy at once, unlike prior approaches. Most popular and effective methods uses a shared backbone with separate branches for each level (Zhu & Bain, 2017; Wehrmann et al., 2018; Chang et al., 2021; Liu et al., 2022; Chen et al., 2022; Jiang et al., 2024; Zhang et al., 2024). The key difference lies in how the hierarchical relationships are modeled. For instance, in FGN (Chang al., 2021), finer features are concatenated to predict coarse labels, whereas in HRN (Chen et al., 2022), coarse features are added to finer features through residual connections. A critical issue in this approach is maintaining consistency with the taxonomy in the predicted labels. To address this, Wang et al. (2023a) proposed a consistency-awar awarre method by adjusting prediction scores through coarse-to- fine deduction and fine-to-caarse induction. However, we observed that using separate branches can lead to inconsistency. as each branch processes the image independently. To address this, we propose a model based on consistent visual grounding. To the best of our knowledge, no prior work has utilized visual segments to resolve inconsistency in hierarchical classification.
Hierarchical Semantic Segmentation aims to group and classify each pixel according class hierarchy (Li ct al., 2022; Singh et al., 2022; Li et al., 2023; He et al., 2023; Wang et al., 2024; Qi al., 2024), with pixel grouping varying based on the taxonomy used. However, these works require pixel-level annotations, which are not available in hierarchical classification. In addition, while these method focus on precise pixel-level grouping, our work leverage unsupervised segments of varying granularities within the image for hierarchical clasification.
Semantic Segmentation aims to group pixels without pixel-level annotations or using only class labels (Hwang et al., 2019; Ouali et al.. 2020; Ke et al., 2022; 2024). These works employ hierarchical grouping to achieve meaningful segmentation without pixel-level labels. Here, "hicrarchical" refers to part-to-whole visual grouping, where smaller units (c.g.. a person's 's face or arm) are grouped into larger regions (e.g., the whole body). Based on our intuition
Unsupevised/Wentt weaiiiseddwaall---sseeeeeee
16

---

Published as a conference paper at ICLR 2025
that fine- fine-grained classifiers need more detailed information, while coarse classifiers focus broader groupings, our approach leverages these varying types of visual grouping. To implement this, we adopt the recently proposed CAST (Ke et al., 2024), whose graph pooling naturally supports consistent visual grouping. Notably, our work introduces the novel insight that part-to-whole segmentation can align with taxonomy hierarchies (e.g., finer segments for fine-grained labels, coarser segments for coarse labels), a connection not previously explored.
C BENCHMARK DATSET
Table 4: Benchmark Datasets.
Dutasets
L-17
NL-26
E-13
E-30
CUB
Aircraft
iNat21-Mini
Levels
2
3
classes
17-34
26-52
13-130
30-120
13-38-200
30-70-100
273-1.103-10.00
# Train images
44.2K
65.7K
167K
IS4K
5,994
6,667
S0nK
# Test images
1.7K
2.6K
6.5K
6K
5,794
3,333
100K
D HYPERPARAMETERS FOR TRAINING.
For a fair comparison, we use ViT-Small and CAST-Small models of corresponding sizes. As in CAST, train both ViT and CAST using DeiT framework (Touvron al., 2021), and segmentation granularity is set to 64, 32, 16, 8 after 3, 3. 3, encoder blocks, respectively. Our training progresses from fine to coarse levels, with each segment corresponding accordingly. The initial number of superpixels is set to 196, and all data is trained with a batch size of 256. Following the literature (Chen al., 2022), use ImageNet pre-trained models for the Aircraft, CUB, and iNat datasets. For the ImageNet subset BREEDS dataset, we train the models from scratch. We show hyper-parametce settings Table
Table 5: Hyper-parameters for training H-CAST and VIT on FGVC-Aircraft, CUB-200-2011, BREEDS, and iNaturalist datasets. We follow mostly the same set up as CAST (Ke et al., 2024).
Parameter
Aircraft
CUB, BREEDS, iNaturalist
batch size crop_size learning rate
256
256
224
224
le
5 5--4
weight.decay momentum
0.05
0.05
0.9
total_epochs
100
100
warmup.epochs
5
5
warmup varmup_lcarningraat
1e-4
le-6
optimizer
Adam
Adam
learning rate catecp
Cosine decay RandAug(9, 0.5)
Cosine decay
augmentation (Cubuk et al., 2020)
RandAug(9, 0.5)
label. smoothing (Szegedy al., 2016)
0.1
0.1
mixup (Zhang al., 2017)
0.8
0.8
cutmix (Yun et al., 2019)
1.0
1.0
(weight for TK loss)
0.5
0.5
VIT-S: Tokens
[196]
CAST-S: # Tokens
[196] x3, [64] x3, [32] xa. [16) x2
17

---

Published as a conference paper at ICLR 2025
E ADDITIONAL EXPERIMENTS
E COMPARISON BETWEEN FPA AND TICE.
FPA evaluates both accuracy and consistency, while TICE focuses solely on consistency. Achieving high FPA is the primary goal in hierarchical classification. The distinction between FPA and TICE is shown in Table 6.
Table 6: FPA considers both correctness and consistency. While TICE (Wang et al., 2023a) measures only consistency, FPA marks predictions positive only when they are both correct and consistent.
TICE
X
X
X
FPA
/
X
X
X
X
E.2
VISUALIZATION OF ATTENTION MAP
To validate our claim that the model guides classifiers toward consistent visual grounding, we visualize attention maps from H-CAST in Figure The visualizations demonstrate that we progress from lower to upper blocks, the model increasingly attends to similar regions. In the lower blocks, attention more detailed and localized, while in the upper blocks, attention expands to cover broader regions, including those highlighted by the lower blocks. These patterns align with our intended design for visual grounding in hierarchical classification.
Image
Level
Level 3
Level
Figure 9: Visualizations of Attention maps from H-CAST. We align the attention weights with superpixels and average them across all heads. Darker red areas represent regions with higher attention weights. At the lower level (level 2), the attention is more focused on specific regions, such as the snake's head and parts of its body, emphasizing these as critical features for anne-graineddee classification (e.g., "Hypsiglena torquata"). In contrast, at the upper level (level 4), the attention expands to encompass the entire body of the snake, suggesting a shift towards a more holistic understanding of the object for coarse label (e.g., "snake"). This progression from localized to broader attention illustrates how H-CAST hierarchically integrates information across layers, supporting consistent visual grounding for hierarchical classification.
18

---

Published as a conference paper at ICLR 2025
E.3 ADDITIONAL EXPERIMENTS FOR LOSs ABLATION
Similar to the results on the Aircraft dataset, the Living-17 dataset also shows consistent performance trends, with our proposed loss achieving strong results (Table 7. Table 8). Interestingly, for TICE, which measures only semantic consistency, the TK loss alone (TTable 7) and the BCE or Flat Con- sisteney loss achieved better performance (Table 8). However, when considering both accuracy and consistency (i.c., FPA), our proposed loss delivered the best overall performance.
Table 7: Utilizing both losses yields best performance on Living-17.
Table 8: KL Div. lass shows best perfor- mance Living-17.
Las
Ltk
FPA
Coarse
Fine
wAP
TICE
Sem. Consis.
FPA Coarse
Fine
wAP
TICE
x
84.00
90.71
84,30
86.43
1.71
Flat Cons.
82.82
88.88
83.53
85.31
2.51
/ 84.21
90.24
84.59
86.78
2.59
BCE
83.65
89.76
84.00
85.92
1.76
/
85.12
90.82
85.24
87.10
3.19
KL Div.
85.12
90.82
85.24
87.10
3.19
E.4
EVALUATION ON THE LARGE-SCALE INATURALIST DATASETS.
We present the results of our experiments on the large-scale dataset, iNaturalist21 1mmiii (Van Horn et al., 2021) and NNtturalist-201 (Horn et al., 2018). First, iNaturalist21 1mmii contains 10,000 classes, 500,000 training samples, and 100,000 test samples, organized within an 8-level hierarchy. For our experiments, we focused on 3-level hierarchy consisting of name, family, and order. This selection was motivated by the need for models with practical and meaningful granularity for real-world applications.
The number of classes at each hierarchical level is as follows: Kingdom (3), Supercategory (11), Phylum (13), Class (51), Order (273), Family (1,103), Genus (4,884), Name (10,000)
We excluded coarse-grained levels such as kingdom (3 classes), Supercategory (11 classes), because their minimal granularity adds little value for classification tasks. Similarly, overly fine-grained levels such as genus (4,884 classes), where many species are represented by only one or two samples, offer limited differentiation from direct name-level classification. Instead, (273 classes), family (1,103 classes), and name (10,000 classes) to ensure that each higher-level class represents a meaningful subset of lower-level classes, allowing for interpretable and consistent predictions.
focused order
The results are shown in Table 9. Compared to ViT baselines that the same ViT-small back- bone, our method achieves a 5.98% improvement over TransHP (Wang et al., 2023b) and a 9.27% % improvement over Hier-ViT in the FPA metric, demonstrating significant performance advantage.
Table 9: Our H-CAST outperforms ViT-hackbone baselines, Hier- ViT and TransHPP on the large- scale NNaturalist2021-Mim dataset, achieving significantly higher accuracy and consistency.
NNauualist2021-Minii (273- 1,103 10,000)
FPA(1)
order(t)
family(t)
name(t)
wAP(t)
TICE())
Hier- ViT
55,65
87.25
77.81
62.71
64.76
26.21
TransHP
58.94
83.49
82.15
68.82
70.46
24.63
Ours (H-CAST)
64.92
89.72
84,00
70.00
71.83
16.00
Our Gains
+5.98
+2.47
+1.18
+1.27
+1.37
+8.63
19

---

Published as a conference paper at ICLR 2025
We further compare our method with single-level approaches that utilize hierarchical labels to enhance fineggriined accuracy (e.g., Guided (Garnot & Landrieu, 2020), HiMulConE (Zhang et al., 2022), and TransHP (Wang al., 2023b)). The comparison is conducted on the large-scale iNaturalist- 2018 dataset, following HP. iNaturalist-201 2011 includes two-level hierarchical annotations with 14 super-categories and 8, 142 species, comprising 437,513 training images and 24,426 validation images, We use the same H- H-CAST-small and the model is trained for 100 epochs, using the same hyper-parameters in Table 5. As shown Table 10, our method achieves strong fine-grained accuracy, demonstrating the effectiveness of consistent visual grounding.
Table 10: H-CAST outperforms methods leveraging hierarchical labels for fine-grained accuracy on the large-scale INaturalist-201s dataset, demonstrating the elfectiveness of visual consistency. The results are reported from TransHP (Wang et al., 2023b).
2Natu (Acc.)
Guided
63.11
HiMulConE
63.46
TransHP 64.21
H-CAST
67.13
E.5 EVALUATION ON CUB-200-2011 AND FGVC-AIRCRAFT DATASETS.
Table 11 and 12 presents results CUB and Aircraft datasets. In our experimental results, first observe a significant performance drop of Hier- ViT compared to Flat-Vir This highlights a common challenge in hierarchical recognition, where training coarse and fine-grained classifiers simultaneously results in performance degradation, as observed in previous ResNet-basee hierarchical recognition models (Chang et al., 2021). Our experiments reveal that this problem also exists in ViT architectures. This indicates that hierarchical recognition is a challenging problem that cannot be solely addressed by providing hierarchy supervision class tokens. On the other hand, our method consistently outperforms most Flat models.
Compared ViT-backbone models, Hie and TransHP, our approach achieves significantly better performance. Specifically, using the FPA metric, which captures both accuracy and consistency across all levels, our model outperforms TransHP by +8.2%% on the Aircraft dataset and +2.6%p on the CUB dataset.
also evaluate BIOCLIP (Stevens et al., 2024), a foundation model for biology, on the CUB dataset, as it focuses bird categories. BIOCLIP operates flat-based hierarchical model, concatenating the entire taxonomy into single text representation. As a result, all higher-level classes are directly determined by the fine-grained species predictions, resulting in TICE Airtonomy-Inconssstii Error) of O. While BIOCLIP achieves strong performance, its reliance on fine-grained predictions to define coarse classes introduces limitations in accurately predicting higher-level classes.
As Vision Transformer backbone models, when the training dataset is small, such as Aircraft and CUB with around 6K images, HRN, ResNet-based models, demonstrates better performance. However, HRN's method is highly sensitive to batch size, with a significant drop in performance observed when increasing the batch size from 64, This sensitivity makes it less suitable for training on large-scale datasets.
20

---

Published as a conference paper at ICLR 2025
Table 11: Ours consistently shows the best performance on CUB-200-001.. H-CAST outperforms ViT-backbone models, Hier-ViT and TransHP, by over 6.3 and 2.6 percentage points, respectively. Additionally, it achieves 3.2 percentage point gain in the FPA metric over the ResNet-based HRN while using significantly fewer parameters. (A higher metric indicates better performance, except for TICE.) Flat models require training three separate models.
Configuration
CUB-200-201
(13 38
-200)
backbone
#params
input
batch size
FPA
order
family
species
WAP
TICE
Flat
FIat-VIT
VIT-S
65.1M
224
256
82,30
98.50
94.84
84.78
87.01
5.76
Flat-CAST
VIT-S
78.5M
224
256
81,50
98.38
94.82
83.78
86.21
6.14
RN-50
24.8M
224
76.08
97.05
91.44
79.29
82.05
7.73
Hierarchy
HRN
RN-50
94.5M
448²
64
80.07
98.17
93.75
83.
85.52
6,51
HRN
RN-50
94.5M
448
8
84.15
98.58
95.39
86.13
88,18
4.62
BIOCLIP (zeroshot)
ViT-B
149.6M
78.18
78.18
78.18
78.18
78.18
0.0
Hier-Vit VIT
VIT-S
21.7M
224
256
77.03
98.40
92.94
79.43
82.46
8.72
TransHP
VIT-S
21.7M
224
128
80.70
96.70
94.15
84.59
86.66
7.16
Ours (H-CAST)
VIT-S
26.2M
224
83.28
98.65
95.12
84.86
87.13
4.12
Our Gains over SOTA
-0.87
+0.07
-0.27
-1.27
-1.05
+0.50
Table 12: Evaluation on FGVC-Aireraft. On the smaller Aircraft dataset, ResNet-based models such as FGN and HRN show good performance. However, our H-CAST achieves better results in the consistency metric (TICH) and performs comparably in the FPA metric. Notably, H-CAST outperforms ViT-backbone models, Hier-Vii and TransHP, by over and percentage points, respectively, in the FPA metric.
Configuration
FGVC-Aireraft (30
100)
backbone
#params
input
batch sizel
FPA
maker
family
model
wAP
TICE
Flat-VIT
ViT-S
65.1M
224
256
76.99
94.27
91.93
80.14
86.39
10.98
Flat-CAST
VIT-S
78.5M
224
256
78.22
92.95
88.93
82.39
86.26
10.77
FGN
RN-50
24.8M
224
128
85.48
92.44
90.88
88.39
89.87
7.50
Hicrarchy
HRN
RN-50
94.5M
448"
64
83.56
94.93
92.68
86.59
89.97
7.26
HRN
RN-50
94.5M
4482
8
91.39
97.15
95.65
92.32
94.21
3.36
Hier-Vit
VIT-S
21.7M
224
256
72.10
92.35
86.26
75.94
82.01
15.75
TransHP
VIT-S
21.7M
224a
128
75.49
90.16
87.46
81.46
84.86
13.95
Ours (H-CAST)
VIT-S
26.2M
224
256
83.72
94.96
91.39
85.33
88.90
5.01
Our Gains over SOTA
-7.67
-2.18
-6.99
-5.31
-1.65
E.6 ADDITIONAL VISUALIZATIONS OF SEGMENTS
We visualize additional examples of feature grouping from fine coarse for full-path correct and incorrect predictions on the Entity-30 dataset in Figure 10. For full-path correct predictions (all levels correct), visual details effectively grouped to identify larger objects coarser levels. In contrast, for full-path incorrect predictions (all levels incorrect), segments fail to recognize the object.
21

---

Published as a conference paper at ICLR 2025
Fine to coarse segments
Fine to coarse segments
a) Full- Path Correct Predictions
b) Full-Path Incorrect Predictions
Figure 10: Additional examples of the differences in visual grouping between cases where predictions at all levels are correct and where they are not. Correct predictions show better clustering, while incorrect predictions often exhibit fractured misaligned groupings.
E.7 COMPARISON OF IMAGE SEGMENTATION WITH CAST
To quantitatively evaluate the segmentation results in Figure 8, we use the ImageNet segmentation dataset, ImageNet-S (Gao al., 2022). obtain the ground-truth segmentation data for BREEDS dataset. The number of samples in the BREEDS validation data for which ground-truth segmentation data can be obtained from ImageNet-S is 381 for Living-17. 510 for Non-Living-2e 1,336 for Entity-30, and 1,463 for Entity-13. 13. To calculate the region mIOU for fine-level objects, we use the last-level segments (8-way) for segmentation. Following CAST, we name the 8-way segmentations using OvSEG (Liang et al., 2023).
Also, we further visualize the segmentation results on Entity-30 in Figure 11, and show that additional taxonomy information improves segmentation. For example in the first 'bird' image, H-CAST is able to segment meaningful parts such as the face, belly, and a branch, with less fractured compared to the CAST. Thus. H-CAST delivers an improvement in segmentation with the benefits of hierarchy.
22

---

Published as a conference paper at ICLR 2025
image
CAST: Fine to coarse segments
H-CAST: Fine to coarse segments
Figure 11: Additional visual results segmentation show that H-CAST with additional taxonomy information improves segmentation. H-CAST successfully segments meaningful parts with fewer fractures compared to CAST.
23


Reviews:
[{"review_id": "6rORmTsuOr", "reviewer": "Reviewer_d3Ey", "summary": "The paper proposes H-CAST architecture for hierarchical classification tasks. The architecture builds on top of prior CAST work: superpixels are fed into a ViT network, where periodic graph pooling operation aggregates the tokens of high similarity. This produces a fine-to-course hierarchy of features. Linear layers at each level of the hierarchy are used as classification heads. The paper also presents tree-path KL loss, where the entire path in the hierarchical class tree is matched. The method shows strong performance over baselines in several benchmark datasets.", "questions": "Please see questions listed alongside weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) The model shows better results than prior works.\n\n(2) The examples given in the introduction help to explain and illustrate the reasoning behind the hierarchical focus.\n\n(3) The ablations confirm that the added additional loss contributes to the performance.", "weaknesses": "(1) In L61, the difference in available labelling is presented as one of the motivations for hierarchical classification. However, the presented method assumes that all levels of the hierarchy are available. Can the technique work if the finest levels of supervision are not available?\n\n(2) Similarly, given the availability of the finest-level label, the other course levels in the tree are implied, so perhaps a more appropriate flat baseline would be a ViT that only predicts finest-level classes (and thus parent nodes by simple aggregation). It would also provide a more appropriate comparison in terms of architecture.\n\n(3) Given the relatively \"small\" sizes of the datasets (Tab 1.) by modern standards and some occasional closeness to the flat baselines in the scores (Tab 2.) Has there been any significant variability observed in the results? Would it be possible to include some measure of variance for some key results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QBFOYQTsgu", "reviewer": "Reviewer_khL4", "summary": "This work proposes H-CAST, a model built upon CAST for hierarchical image classification, by addressing both visual consistency and semantic consistency across predictions at different hierarchical levels. To achieve this, hierarchical supervision at different network layers and a tree loss are introduced. Experiments on three datasets verify the proposed method can achieve better performance than the baseline method.", "questions": "Please see the above.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation for visual consistency is sound.\n2. This work introduces new metrics for hierarchical classification, which can measure the coherence of hierarchical predictions.\n3. The proposed method achieves SOTA on all datasets.", "weaknesses": "1. The key point of this paper, which is making classifiers at different levels attend to consistent visual cues, lacks support from quantitative results or theory. For example, if wrongly classified cases are all associated with incorrect CAMs? What is the transfer rate after adopting the proposed model? Qualitative comparisons are subjective and lack statistical significance. In addition, Grad-CAM is an approximation and does not truly explain how the network operates.\n2. This work relies heavily on CAST. Though the authors propose the concept of visual consistency, its implementation is directly borrowed from CAST. Consequently, the primary contributions of this work are merely the hierarchical supervision loss and tree KL loss, neither of which can ensure visual consistency. Additionally, the fine-to-coarse training strategy is also derived from CAST. The tree-path KL loss is trivial.\n3. The tree KL loss defines the ground truth distribution according to the number of hierarchical levels (i.e., 1/L). It would be beneficial to address whether the proposed solution can effectively manage larger hierarchies, particularly those with depths of up to 10 or 20.\n4. While the clustering module would potentially guide the model to attend to spatially-coherent regions, it is unclear why the model would “ensure that each hierarchical classifier focuses on the same corresponding regions”. In fact, the model still has the opportunity to find shortcuts (attending to different regions in different levels as Fig. 2) and meanwhile deliver correct classification results. In addition, I expect visualizations of Grad-CAM results in different hierarchical levels from the proposed model. \n5. The experiments are somewhat questionable. Firstly, the latest hierarchical competitor, HRN, published in CVPR'22, is relatively outdated. Secondly, the experimental results of HRN differ from those reported in the original publication. Thirdly, there are several competitors [a-b] (might be more, not carefully checked), released before two months prior to the DDL, that outperform this work.\n6. Results in Fig. 5 do not totally make sense to me. Examples in a) and b) are not equally recognizable, i.e., all examples in b) are much harder to distinguish/group than those in a). As a result, it is hard to confirm whether poor clustering of pixels in b) is the cause or the effect of incorrect predictions. One way for improvement is to examine for similar hard-level images in a) whether correct predictions are achieved along with better clustering results. \n7. According to [c], existing work has explored various loss functions for hierarchical classification. It would be useful to compare the effectiveness of tree KL loss against these alternatives. Of course, the analysis should also be provided.\n8. The comparison to hierarchical approaches is unfair. While FGN and HRN use ResNet-50 as the backbone， this work adopts ViT-S.\n9. More top-leading hierarchical classification work should be included in the comparison.\n10. The datasets used for evaluation are small. larger datasets with complex hierarchy (ie.g., ImageNet-1K, iNaturalist) should be evaluated to better assess effectiveness.\n11. How about the training and inference speed of the proposed method? Given the incorporation of superpixels and segmentation in addition to classification, it is necessary to provide a comparison of resource costs, including both time and memory usage.\n12. The literature review is far from complete. In addition to [a, b, c], numerous efforts in hierarchical scene parsing is totally missing; see  related work section in [d, e]. As a top-conference paper, a comprehensive literature review is a basic requirement. I believe the reference part should be greatly extended. \n13. In fact, the hierarchical loss function in [d] is superior to the proposed Tree-PATH KL loss, in that it guarantees hierarchy-aware coherent predictions while the proposed loss cannot. A strict quantitative comparison of the two loss functions should be provided.\n14. Minor issues include: a) the usage of the terms \"interpretability\" (L447) and \"explainability.\" b) citation format. c) vector images. d) missing period after Eq. 2. e) the presentation of Eq. 3.\n15. A Conclusion section should be added to properly conclude the work and offer insights of downside of impact of the work. \n[a] HLS-FGVC: Hierarchical Label Semantics Enhanced Fine-Grained Visual Classification.\n[b] Hierarchical multi-granularity classification based on bidirectional knowledge transfer.\n[c] Hierarchical classification at multiple operating points. NeurIPS 2022\n[d] Deep Hierarchical Semantic Segmentation. CVPR 2022\n[e] LogicSeg: Parsing Visual Semantics with Neural Logic Learning and Reasoning, ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vtvl67eTxj", "reviewer": "Reviewer_YARY", "summary": "This article deals with hierarchical image classification using neural networks. This classification is actually composed of several classifications performed at different levels of precision (coarse to fine). It is important that the classes detected for an image at different levels match the logical organization of the labels. Labels are organized in a tree structure, where each level of precision is a level of the tree. Therefore, there must be a direct path between the different classes identified in an image for the classification to be correct. \n\nSo we understand that in hierarchical classification, there are different levels at which to judge the quality of the classification. At each level, there's the accuracy with respect to the expected label, and there's the logical connection between classifications at different levels.\n\nA simple solution would be to use a single encoder to represent the image and different classification heads for each hierarchical level. The article explains that this is a bad solution because it puts the different hierarchical levels in competition with each other because they require different levels of representation. The current state of the art divides the architecture into independent branches, each of which generates a representation for a different hierarchical level. The authors have found that this separation has led to a non-causality in the classification of the different levels (decisions are independent), which could lead to inconsistencies on the relationship between classifications of different levels. \n\n\nThe article focuses on CAST, a variant of the transformer network that does not slice the image into patches but into superpixels and integrates a hierarchical structure by including superpixel merging. The architecture thus goes from the fine to the coarse level. Thus, the authors have found that this architecture is adapted to the problem of hierarchical classification by adopting a fine-to-coarse classification logic. Therefore, they have adopted the same architecture, but instead of classifying the different levels in parallel like the state-of-the-art, they classify them sequentially. Thus, the super pixel embedding and the class token pass through this architecture, and in the course of this processing, the class token is classified several times. The classifier heads follow each other and classify at increasingly coarse levels. \n\nThe authors also propose to combine two losses, one which is the cross entropy of the independent classification at each level, and one which is the concatenation of the probability of all classes (renormalized) where several classes are expected (one for each level). Therefore, the losses promote a local and a global level of good classification. \n\nIn the experimental section, the authors show that their sequential classification allows a good improvement in the collection of metrics (local and structural) compared to architectures that process the task in parallel. In addition, parallel architectures are larger because they have to divide the network into as many branches as there are hierarchical levels. The authors show that a smaller network can outperform a larger one if it is designed to match the structure of the problem to be solved. \n\nThe authors also added an ablation study to investigate the advantage of fine-to-coarse over coarse-to-fine, and the influence of both local and structural loss. The authors also show that hierarchical training improves segmentation results.", "questions": "The following paragraphs are meant to be an open reflection. I'm not an expert in hierarchical classification, so I may have missed some key parts of the problem, modeling, and overall reflection. \n\nI think fine-to-coarse works well here because it fits the network architecture, not because it's a better choice overall. Indeed, if the hierarchical structure of the labels is a tree, one could simply put all the effort into classifying the fine level and infer the higher levels by going up the tree, since each class has only one parent. Of course, this would mean that a single error on the fine level would invalidate all other classifications. \n\nOverall, I think the coarse-to-fine direction makes more sense, since it's all about descending a tree and thus reducing the search field of the lower layers. I just don't think this architecture is suited for that. However, it may be possible to design a sequential architecture in coarse-to-fine logic, for example a U-net. We could imagine a U-net or an encoder-decoder using the CAST structure with a symmetric decoder. It would then be necessary to find a way to divide the super-pixels. The classification will be perform sequentially by going up the decoder\n\nWhether it's one way or the other (but rather the other), I think that the sequential classifications should directly influence each other (like in a Markov chain or a transition table), and not just as different normalizations of the classification probabilities.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and organized. The authors have made a good effort to put the problem in the context of the state of the art. The problematic behavior of the state-of-the-art architecture is well illustrated in Figures 1 and 2. In general, the illustrations and explanations allow a non-expert to understand the specificity of hierarchical classification, the state-of-the-art choices, and the resulting problems. I also appreciated the care taken in explaining the metrics to understand what they represent and how they complement each other. \n\nThe solution proposed by the authors makes sense, as they have found that CAST's architecture is very well suited to solve this problem. The explanations and schematics of the architecture are clear, I just had to go to the CAST article to understand how super-pixel pooling is done (which is not critical to understanding the article). The two losses also make sense, as the authors realized that the problem of the state of the art comes from an independent resolution at each level of the hierarchy, and added a loss that forces a (more) globally correct classification. The experimental part (Figure 4, TICE metric) shows that their model makes fewer structural errors (compared to the tree structure), even if it is smaller.\n\nTable 2 and the different ablations clearly show the impact of the improvements made. The last part, which shows that hierarchical learning can improve segmentation, is an interesting addition, which may lead us to think of an opening towards hierarchical segmentation (for which super-pixel-based embedding seems to be well suited).\n\nOn a more personal note, at a time when the tendency is to build gigantic, high-consumption networks, I find it appreciable to see methods demonstrating that designing a solution specifically for a problem can increase its efficiency.", "weaknesses": "In part 4.4, I'm not sure how to interpret the result. We can assume that the result is a failure if the superpixel segmentation doesn't make sense. But to me, this just shows that superpixel segmentation is not necessarily adapted to a semantic problem, because it's done at the color level, and semantic in images is not necessarily associated with color.  \n\nIn Figures 1 and 4, I find that the space between class names (like \"Chimpanzeeferret\" in Figure 1) and their position/alignment in the tree can be difficult to read. In two-word classes, you might want to break a line, because at first sight the second part of the name seems to be another class link to another node.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H0Q88Fppqp", "reviewer": "Reviewer_ZNGF", "summary": "This work aims to tackle hierarchical image classification. It is motivated by a hierarchical image segmentation work, which also conducts image classification. This work makes an extension on this basis and proposes a tree KL loss to deliver semantic consistency predictions regarding hierarchy. Evaluation on three datasets verifies the effectiveness over the baseline, which focuses on segmentation rather than classification.", "questions": "Why KL divergence is better than CE? Both experiments and the theoretical explanation should be provided.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The overall presentation is easy to follow. There is no difficulty in understanding the work.\n2. The improvement on the baseline is considerable.", "weaknesses": "1. The technical contribution is not enough. Specifically, the proposed method contains visual consistency and semantic consistency. However, the major design and implementation of visual consistency are directly borrowed from CAST [1]. The claim in L260-264 is not convincing. Simply adding supervision in each decoding level can not be considered a vital contribution compared to directly using the overall architecture of CAST.\n\n[1]. Learning hierarchical image segmentation for recognition and by recognition. ICLR 2024\n\n2. The review only contains hierarchical image classification. However, pixel-level hierarchical classification (i.e., hierarchical image segmentation [2,3,4]) can also provide insights for this work. In fact, the visual consistency mentioned in this work is conducting segmentation on the image, and the baseline method (CAST) is also a hierarchical image segmentation work. Therefore, a literature review on hierarchical image segmentation should be included.\n\n[2] AIMS: All-Inclusive Multi-Level Segmentation for Anything. NeurIPS 2023.\n\n[3] Hierarchical Open-vocabulary Universal Image Segmentation. NeurIPS 2023\n\n[4] LOGICSEG: parsing visual semantics with neural logic learning and reasoning. ICCV 2023. \n\n\n3. Concatenate labels from all levels to create a distribution has already been explored in [3,4]. The difference is [3,4] using cross-entropy loss, while this work uses KL divergence. \n\n4. The motivation for using KL divergence instead of cross-entropy is unclear. Since CE loss contains both the KL term to minimize the differences between distribution and a punishment term to minimize the uncertainty. Why KL divergence is better than CE? Both experiments and the theoretical explanation should be provided.\n\n5. The evaluation of the proposed method is limited. \n\n      i) The comparison to hierarchical counterparts only contains out-of-date methods published before 2022 and the baseline (focusing on segmentation rather than classification). The top-leading solutions (e.g., [5]) are all ignored.\n\n      ii) The label hierarchy is shallow, with only up to 3 hierarchical levels.\n\n[5]. TransHP: Image Classification with Hierarchical Prompting. NeurIPS 2023\n\n6. The claims in 4.6 are similar to the insights provided in [1].\n\nOverall, the technical contribution, evaluation, and insights provided by this work are all limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9BrczVjaSf", "reviewer": "Reviewer_uxN6", "summary": "The paper explores hierarchical image classification (HIC) task, which is old but worthy to explore. I have experienced many bad reviews in this domain, for instance, “the proposed method relies on coarse labels and therefore not useful in the real world”. However, I think they are the misunderstanding of this area and I want to give some new one.", "questions": "Please specific what is the difference between your work and TransHP?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "NA", "weaknesses": "Overlap with former work? I noticed that an important reference, TransHP: Image Classification with Hierarchical Prompting (NeurIPS 2023), is not cited in your paper. This work may be directly relevant, as it appears to share similarities with your proposed method, specifically in the use of different ViT blocks for different levels of hierarchy. \n\nLimited novelty. Your proposed approach introduces elements such as Superpixel and Graph pooling. While these are effective, both are well-established techniques in computer vision. A more detailed explanation of how these additions provide novel contributions within the hierarchical framework would clarify the unique aspects of your work.\n\nLimited evaluation. ImageNet, as a large-scale hierarchical dataset, might provide a stronger test of your method's capabilities compared to the smaller datasets currently used. \n\nFormatting problem: Please ensure the readability of all components of the paper. For instance, the font size in the tables is small, which may make it difficult for readers to check the data presented", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)