Hierarchical Prototype Learning for Semantic Segmentation
Authors: Seoha Lim
Jinmyeong Kim
Jieun Kim
Sung-Bae Cho
Venue: ICLR 2026 Poster (2026)
Keywords: Semantic Segmentation
Contrastive Learning
Prototypical Networks
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Conventional semantic segmentation methods often fail to distinguish fine-grained parts within the same object because of missing links between part-level cues and object-level semantics.
Inspired by how humans recognize objects, which involves first identifying them as a whole and then distinguishing their parts, we propose a hierarchical prototype-based segmentation method called Hierarchical Prototype Segmentation (HiPoSeg). This builds a structured prototype space that captures both abstract object-level representations and detailed part-level features, enabling consistent alignment between levels. HiPoSeg leverages a hierarchical contrastive learning strategy to structure semantic representations across levels, encouraging both intra-level discrimination and cross-level consistency.
Experiments on standard benchmarks such as Cityscapes, ADE20K, Mapillary Vistas 2.0, and PASCAL-Part-108 demonstrate that HiPoSeg produces consistent performance improvement with an average gain of +3.07\%p mIoU without any additional inference cost.

Published as a conference paper at ICLR 2026
HIERARCHICAL PROTOTYPE LEARNING FOR SEMANTIC SEGMENTATION
Scoha Lim'; Jinmyeong Kim"; Jieun Kim', Sung-Bae Choè
'Department of Artificial Intelligence, Yonsei University *Department of Computer Science, Yonsei University {secha815, jmkim.,, lilly y 9928, sbcho @yoneeii.......ooo
ABSTRACT
Conventional semantic segmentation methods often fail distinguish fine-grained parts within the same object because of missing links between part-level cues and object-level semantics. Inspired by how humans recognize objects, which in- volves first identifying them as a whole and then distinguishing their parts, we propose a hierarchical prototype-based segmentation method called Hierarchical Prototype Segmentation (HiPoSeg). This builds structured prototype space that captures both abstract object-level representations and detailed part-level features, enabling consistent alignment between levels. HiPoSeg leverages hierarchical contrastive learning strategy to structure semantic representations across levels, encouraging both intra-level discrimination and cross-level consistency. Exper- iments on standard benchmarks such as Cityscapes, ADE20K, Mapillary Vistas 2.0, and PASCAL-Part- demonstrate that HiPoSeg produces consistent perfor- mance improvement with an average gain of +3.07%p mloU without any addi- tional inference cost.
background
is cow's tow's leg.
cow
cow torso
horse
Model
horse head
Prediction
horse
horse muzzle
That's a horse.
horse
horse's 's leg!
neck
Human
Figure 1: Comparison of semantic segmentation between human and computational model.
INTRODUCTION
Semantic segmentation, fundamental task in computer vision, requires assigning semantic la- to every pixel an image (Xie et al., 2021 et al. 2022 2023 Fu et al. 2019 Zhao et al.) 2019 Sung et al. 2024 Zhou et al. 2022). Although deep learning has significantly advanced this field (Kim & Cho 2022), most current approaches still treat it as flat classification problem. where each pixel is assigned directly to predefined class label without considering any structural relation- ships among classes (Michieli et al. 2020 Chen et al., 2020 Huang et al. 2019 Chen et al.] 2018 result in semantic inconsistencies. especially when visually similar parts belong to different classes. contrast, the human visual system approaches recognition quite differently: operates hierarchically. For example, in Fig. [ when we observe an
Zhao al. 2018). This lack of structure
"Equal contribution.

---

Published as a conference paper at ICLR 2026
object, we first recognize its broader category (e.g., horse) and then identify its finer components (e.g.. horse's 's leg). This top-down process allows for faster and more accurate recognition by reduc- ing the scope of plausible interpretations: once we know it as horse, we naturally expect its parts to be horse-related. Rather than using such hierarchical priors, conventional segmentation models treat all classes independent, limiting their ability reason contextually semantically. This limitation is especially problematic in fine-grained segmentation tasks or scenarios involving a large number of visually similar categories. Without hierarchical guidance, models often fail disam- biguate small object parts or rare classes, resulting in degraded performance and a lack of structural coherence.
To overcome these limitations, we propose HiPoSeg. novel segmentation method inspired by hu- man visual perception, which is built on two key mechanisms. 1) Hierarchical prototype learning: Instead of learning pixel-to-class mappings in isolation, we organize class representations hierarchi- cally and learn prototypes at each semantic level. This allows the model to perform recognition in top-down fashion: first estimating coarse-level categories and then refining to fine-grained sub- categories conditioned on the higher-level predictions. This structure encodes the dependency of categories and promotes consistency within semantic groups. 2) Multi-level class alignment con- straint: To ensure coherence between the predicted levels, we introduce a constraint that enforces alignment between hierarchical layers. Once a high-level category is predicted, lower-level predic- tions are guided to remain within its semantic subtree. This significantly reduces misclassification among semantically unrelated classes and improves part-level accuracy. By integrating these two mechanisms, HiPoSeg mimics the hicrarchical nature of human perception, offering structured alternative to flat segmentation. This not only improves prediction accuracy, but also enhances se- mantic consistency, especially in complex scenes with fine-grained parts and large category spaces.
In sum, this paper has the following contributions.
Hierarchical prototype learning: We maintain higher- and lower-level class prototypes aligned with the label tree and train embeddings from multiple feature stages to be orga- nized around these prototypes. This encourages parts of the same object to share global context while stably differentiating fine- fine-grained representations, thereby achieving seman- tie consistency across the high-low hierarchy.
Multi-level class alignment constraint: We introduce hierarchy-aware alignment constraint that suppresses probability leakage and representation drift toward non- subcategories when a high-level category is activated. This improves fine-grained stability and reduces misclassification in boundary and confusing regions.
Training-only, plug-and-play efficiency: The proposed method is minimally attached to the output end of existing segmentation models as an auxiliary training component. It is used only during training and removed inference, introducing additional parameters, computation, or latency at the test time, and it integrates easily with various bonkbones and decoders.
RELATED WORK
2.1 HUMAN VISUAL COGNITION IN DEEP LEARNING
Human visual cognition is often characterized as a hierarchical process that quickly anchors coarse (global) category and then progressively refines it into parts (Bar 2003 Bar et al. 2006 Bar 2004). This top-down mechanism narrow the candidate space early and regularizes fine decisions with global context, improving efficiency and robustness (Bar 2003). Similar ideas have been exploited deep scene understanding, for example, establishing global context and then refining local details, or using coarse map to guide boundary correction (Ji et al.) 2020 Meletis & Dubbelman 2018). However, many previous methods implicitly assume a flat label space at the level network design or loss construction, and thus do not explicitly model hierarchical constraints.
We operationalize these cognitive insights via a hierarchical prototype representation that is used only during training. We first stabilize global semantics at the high level and then separate fine categories at the low level in a contrastive manner, while enforcing an alignment constraint to main- tain consistency across levels (Li et al. 2022). In practice, implement the coarse to fine process
2

---

Published as a conference paper at ICLR 2026
through combination of prototype memory. contrastive objectives, and margin-based alignment, thus achieving both object-level context and part-level discrimination (Bar 2009). Importantly, the proposed method is training-only (zero inference overhead), making it plug-and -ug-and- addition to standard backbones and decoders.
2.2 HIERARCHICAL SEMANTIC SEGMENTATION
long line of work has sought exploit hierarchy in semantic segmentation, either by structurally integrating global and local context or by reporting performance across high-level/low-leee label groups. For example, OCRNet (Yuan al., 2020) strengthens object-region representations to en- code global context, while RegionSeg (Hu et al. (0211) leverages regional partitioning and relations to improve mask quality. More direct treatments of hierarchy introduce information flow or con- straints between label levels (e.g., HSSN (Li et al. 2022) and LogicSeg (Li et al., (023)). They highlight the need for hierarchical reasoning, but combine signals primarily at the probability/logit level or handle hierarchy as fixed auxiliary term, falling short of structuring the class representation itself.
We lift hierarchy from mere loss add-on to a design principle of the representation space. Con- cretely, we maintain class-wise prototype memories at both the high- and low-levels and de- fine/update each prototype the average of normalized features with the same label (Snell al.] 2017) (Sec. B.T). We then apply hierarchical contrastive learning (Sec. (32)): (a) feature-prototype and prototype-prottyyp objectives the high level, (b) their counterparts the low level, and (c) a high low-level margin alignment that preserves the topology across levels. This design induces hierarchically organized embedding space without relying on hand-crafted fusion rules.
2.3 CONTRASTIVE LEARNING FOR SEMANTIC SEGMENTATION
Contrastive learning has been widely adopted to strengthen pixel/region representations in segmen- tation (Chen et al. 2020; Moon & Cho 2025). Representative methods such as ContrastSeg (Wang et al., 2021) construct positive/negative pairs at the pixel region level to widen decision mar- gins, while RegionSeg (Hu et al., 2021) performs region-level contrast to enhance mid-level fea- tures. Prototype-basee variants stabilize training by anchoring features to class-centre represen- tatives (Zhou et al., 2022 Kee et al. 2024), and recent work further accelerates convergence via context-aware sampling and hybrid objectives (Sung et al. 2024).
However, most contrastive methods assume a flat label space and, therefore, overlook hierarchical re- lations when defining positives and negatives (Tang al. 2023). We address this limitation through a hierarchical prototype bank combined with hierarchy-aware margin alignment. Concretely, (i) the high- and low-feature contrasts pull the embeddings toward the correct prototype each level; (ii) the contrasts between the high- and low-level classes enforce clear inter-class separation within each level; and (ii) an alignment constraint restricts the high- and low-level distances to (0, margin) while ensuring that distances among the high-level classes remain above the margin. This design naturally induces a coarse-to- to-nne curriculum: the high-level space first narrows down candidate classes, and then the low-level space performs fine-grained discrimination. The alignment term is particularly important, as it prevents gradient interference and prototype drift that can arise when multi-level contrastive losses are naively combined, while maintaining the training- only, zero-overhead prop- erty at inference.
3 PROPOSED METHOD
As illustrated in Fig 2 new method proposed for hierarchical class-wise prototype learning for semantic segmentation. Before learning the prototype, Sec[3.1 describes the construction of the hierarchical prototype space, and Sec elaborates details of the prototype learning procedure, where prototype-based hicrarchical contrastive learning is employed.
3.1 HIBRARCHICAL PROTOTYPE SPACE CONSTRUCTION
In this section, we define a prototype space that reflects the hierarchical label structure and describe how to update it stably during training. Let the dataset be D {(Xx, Yx)) k=)) and wa denote the
3

---

Published as a conference paper at ICLR 2026
Buffers
High.lovel
Prototype Construction
High-level Learning
4
A
Output
A4
A
Loo--ioval
Embedding?
Hierarchical Prototype Space
Low-level Leemning
Hierarchical Alignment
Hierarchical Prototypes Update
#(H >a 1 < d(ti.() [f) s
Decoder
! = m (1-m)) m)-0
d (
A2
Encoder
B
Multi scale Concateration
Attract
High-level Protatypes
Repal
Low level Prototypes
Align
Image: /
Pixel Embeddings
Prototype-based Hierarchical Contrastive Learning
Figure 2: Overview of HiPoSeg, composed of (1) prototype construction: pixel features are grouped by low-level labels and their corresponding high-level mappings to initialize both low- and high-level prototypes, (2) high-level learning: pixel features are pulled toward their high-level prototypes that are enforced to be mutually separated, (3) low-level learning: after high-level convergence, low-level discrimination is refined using the same pull-push mechanism, and (4) hierarchical alignment: each low-level prototype is constrained to remain close to its parent high-level prototype while preserving separation between different high-level groups
set of pixel indexes the image Xg. Given an encoder-decoder segmentation model fo (backbone decoder), the pixel embeddings are
Zk a E E wk
fo(Xx).
Assume two-level hierarchy of labels (high-level and low-level). Let Y2 (v n E w/ } be the low-level labels of X2. where yd the label corresponding to z We obtain the high-level labels by a fixed mapping π y + as u m
Y { 1/4
E w
For prototype construction and contrastive learning, a projection head 96 Rd - Rd fol- lowed by fz-normalization:
g0(2g)
2
Rdd
[g0
We maintain two memory buffers for prototypes in each hierarchy: a high-level buffer M and low-level buffer M
S [h' lieyæ,
M 11/1 leya,
(4)
where (²)(1) = EL m()) = i is the set of low-level parts of the high-level class i. We compute batch-wise (or running) prototype estimates using normalized embeddings 2
Σ Σ M = 2
Σ
z
n²
14
(5)
ΣHEE +
ΣΣ +
nEWL
k
nEw
where [F] is the indicator function and prototypes are then ey-normalized:
O is small constant for numerical stability.
The
1
14
h
13
(6)
I| 11
4

---

Published as a conference paper at ICLR 2026
3.2 PROTOTYPE-BASHR HIERARCHICAL CONTRASTIVE LEARNING
This section describes (i) contrastive learning at the high-level, (ii) contrastive learning at the low- level, (iii) alignment constraint that couples the two levels, and (iv) prototype memory updates and training schedule. All similarities are derived from the Euclidean metric. Let ö(a, b) = I|a bll2 be the Euclidean distance and s(a, 6) - b)2 its negative (used inside softmax).
High-level Contrastive Learning. Motivated by coarse-to-fine recognition process, we first learn broad semantics by contrasting pixel features against high-level prototypes and also separating high-level prototypes from each other. For a pixel =g² with high-level ground-truth y" E y let the positive prototype be ply. h and negatives {py h "'' The proture contrastive loss is
1 [
exp(s(2r) p/T))/T) R. Eyey" exp(s(20) p.y )/T)
log
where 7 > O is the temperature and [W**| is the number of valid pixels in the batch. Currently, we discourage high-level prototypes from collapsing via repulsive regularizer.
Luk D 1)
Σ LAKY cfth
exp(s(h', h //)),
(8)
with temperature K > O. Minimizing Lyn + Cub pulls the features towards their correct high-level prototypes, pushes them away from the incorrect ones, and separates the prototypes themselves.
Low-level Contrastive Learning. After establishing high-level semantics, we learn fine-grained semantics by contrasting features and low-level prototypes, together with a separation term among low-level prototypes. For pixel zg with low-level ground-truth y² E )/4, thel loss of fea- ture-prototype is
exp(s(ē1) pzy)//))
W+ Σ log k,n
(9)
Evere exp(s(28) p1))T))
and the prototype prototype repulsion is
Lu = y 11D41111 -1)
Σ exp(s(13, )/c).
(10)
j.Ey itk
These terms make features close to their correct low-level prototypes and keep distinct low-level prototypes well separated.
High-low Alignment Constraint. Using only the above losses may cause confusion between high-level and low-level semantics. We therefore align low-level prototypes with their high-level prototypes while enforcing inter-high separation. Let 1 be a low-level prototype of the high-level
class ht We constrain low-to-high proximity via
Lalign.i
Σ
ma 8(1f, h²) - -01),
(1I)
D
(())):m()=
and keep different ligh-levels well separated via
Calign out
1
Σ i igk
ma 8(h², h k))
(12)
1 1 - 1)
The total alignment loss
Calign Calign.i + B Calign
(13)
with margins 0 < o 02 (we use -0.25, =2= and weights 0, 0.
The total training loss is a weighted sum of all components:
L \ Lee Xa Lyh + Xg Luh + da Li As Lu Ag Latign.
(14)
5

---

Published as a conference paper at ICLR 2026
Prototype Memory Update and Training Schedule. After computing batch-wise prototypes, the memory bank is updated by momentum. Let ht and i3 be the newly estimated (normalized) prototypes from the current batch, and h², denote the entries stored in the memory:
h + mhh & + - m) m)h
1j m + (1 m)
mlj
(15)
with momentum m (0, 1) (we use m=0.9).
To stabilize the representation and follow top-down curriculum, we employ phased schedule: (i) for the first 7.5% of total iterations, learn feature representations only (no prototype learning); (ii) after 7.5%, enable high-level prototype learning; (iii) after 22.5%, enable low-level prototype learning as well; and (iv) after 37.5%, additionally activate hierarchical alignment losses. This schedule first consolidates coarse semantics and then refines fine- fie-grained semantics under explicit high-low alignment.
4 EXPERIMENTAL RESULTS
4.1 EXPERIMENTAL SETUP
Datasets. We validate our method on four benchmark datasets, such Cityscapes (Cordis et al., 2016), ADE20K (Zhou et al. 20166), Mapillary Vistas 2.0 (Neuhold et (0077). and PASCAL-Part. 108 (Michieli et al.] 22020), using their hierarchical label versions. Dataset statistics and hierarchy definitions are followed by (Li al. 2023).
Cityscapes. Urban street-scene dataset (2.975/500/1,522 train/val/test). Two-label levelss: 7 coarse classes and 19 fine classes.
ADE20K. Dcily-scene dataset (20,210/2,000/3,000 train/val/test). Three-label levels: top-level, 14 mid-level, and 150 fine-grained classes.
Mapillary Vistas 2.0. Urban street-scene dataset 000/2,00000550000 train/val/test). Three-label levels: 4 top-level, 16 mid-level, and 124 fine-grained classes.
PASCAL-Part-108. Object part parsing dataset (4,998/5,105 train/test). Two-label levels: 21 coarse classes and 108 part-level classes.
Although the original hierarchies are defined with either two or three label levels depending on the dataset, our method does not rely on the absolute number of levels but only requires the availability of higher-level concept. Therefore, for consistency, we unify all datasets into a two-level hierarchy our experiments; specifically, for ADE20K and Mapillary Vistas 2.0, we use the mid-level as the higher-level (coarse) concept and the fine-grained labels as the lower-level classes.
We adopt the standard mean Intersection-over-Uuue hierarchy- evaluation, we report mIoU at each label level where larger & denotes higher (coarser) level in the hierarchy. We train DeepLabV3+ (Chen et al. 2018) with ResNet-101 back- bone using SGD (Ir=le-2, momentum 0.9, weight decay le-4). For Cityscapes and Mapillary Vistas 2.0, use 512 x 1024 crops, batch size 8, and train for 80K iterations. For ADE20K and PASCAL- Part- 108, we use 512 x 512 crops, batch size 16, and train for 60K iterations.
(mloU) for semantic segmentation. For
4.2 QUANTITATIVE EVALUATION
Cityscapes. Tab. shows that integrating our method into DecpLabV3+ improves mIoU by +10.49%p over the baseline and achieves an average gain +3.07%p compared to competing meth- ods. In particular, HiPoSeg surpasses hierarchical-semantie approaches such as HSSN and Log- icSeg by +1.02%p and +0.84%p, respectively. In addition, it records the highest mloU (84.04%) among all methods, outperforming both CNN-based designs (e.g., PSPNet and PSANet) and recent prototype-drived approaches (e.g., ProtoSeg and ContextSeg). Notably, while many competitors ex- ploit stronger backbones such as HRNet- 448, HiPoSeg achieves superior results with ResNet- 101, highlighting its efficiency and robustness. These results validate the effectiveness of hierarchical prototype contrastive leaming Cityscapes.

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative results on the Cityscapes
Table 3: Quantitative results on the Mapillary
val set.
Vistas 2.0 val set.
Method
Backhone
mioU (9)
mloU (%)
PSPNet
Zhao et al.
2017
ReaNet-101
80,91
Seamless
Porzi ot al.
2019
ResNet-101
38.17
PSANet
Thao et al
2018
ResNet-100
80.96
HMSANet Wane
2020
HRNg--W48
39.53
PAN
LLet
2018
ResNet101
81.12
OCRNet Yuan
HRNet-W48
38.26
Acfael
Zhung et al
2019
ResNet-101
81.60
Maskformer
Cheng et
ResNei-101
DANet
ct 2019
ResNet-101
81.52
HSSN [L
ResNet-101
CCNet
Hunng
m
2019
ResNet-100
81.08
LogieSeg
2023
ResNet-101
40.72
OCRNet
Yuun et al. 2022
ReaNet-101
82.33
Deep Chen 2018
ResNet-101
31.65
ContrastSgg
Wang
202
ResNet-101
HiPoSeg
ResNet-101
41.42 (49.77)
Son
Wang ot al
2021
HRNet-W48
RegionSeg
Table 4: Quantitative results on the PASCAL-
RegionSeg
HRNet-W488
Multi-scale
Pissas
2022
ResNet-101
79.00
Part-108 test set.
Multi-scale Pissus et
2022
IRRNet-W4S
79.59
ProtoSey Zhou
2123
HRNNet W48
81.10
Backhone
(%)
HSSN Li 2022
ResNet-101
83,02
LogieSeg
( ct a. X22
ResNet-101
83.20
(hellhamer al.
ResNet-101
Contextradl
Sung et al.
2024
IIRNet- W448
82.20
SogNet
Badrinaray anan
2015
ResNet-101
BSANe Zhao
2019
ResNet-101
47.36
DeepLabV3+ Chen
2018
ResNet-101
73.55
GMNet
Michicli ot 2020
ResNet-101
47.21
HiPoSeg
ResNet-101
84.04 (+10.49)
FLOAT
Singh ct 7022
ResNet-101
48.08
HSSN
e al.
2023
ResNet-101
48.32
Table 2: val set.
Quantitative results on the ADE20K
LogieSeg
Lretal 2123
ResNet-101
DecpLabV3+
Chen et al. 2018
ResNet-101
46,90
HiPoSeg
Method
Backbone
mloU (%)
Table 5: Ablation study of loss components on the Cityscapes val set.
OCRNet
Yiaa
2020
HRNat-WaR
44,92
RegionSeg
et
2021
ResNet-101
46.85
Multi-scll
Pissas
22
ResNet-101
45.60
Multi scale
Pissas
2123
IRNett W48
47.40
Loe
Lsh
LAd.
L1t
Lu
Lation mloU (%)
ProtnSeg
Zhou al. 2025
HRNet-WaR
43.00
HSSN
Li al. 2122
ResNet-101
47.69
73.55
LogicSeg
Li
2023
ResNet-101
48.46
d
80.96
Contextrasi
Sung
2024
HRNeet-Wal
43.42
81.04
DeepLabV3+
Chen
2018
ResNet-101
44.48
d
d
79.16
HiPoSeg
ResNet-101
(+4.51)
d
84.04
Table 6: Ablation study on hierarchical prototypes (mIoU %).
High-level class
Low-level class
Cityscupes
ADE20K
Mapillary
PASCAL-Part-10
73.55
44.48
31.65
46.90
81.04
46.92
37.21
47.02
80.96
46.87
38.85
47.59
84.04
41.42
ADE20K. In Tab. 2 our method improves DeepLabV3+ by +4.51%p mloU and outperforms alter- natives by an average of +2.78%p. Compared to HSSN and LogicSeg, HiPoSeg achieves gains of +1.02%p and +0.84%p. respectively. Achieving the highest accuracy (48. (88.99%) among all methods, HiPoSeg consistently establishes new state-of-the-ar results under fair backbone settings. Impor- tantly, robustness further demonstrated by outperforming several methods that rely on stronger backbones, confirming both the efficiency and scalability of the proposed method.
Mapillary Vistas 2.0. Tab. shows that HiPoSeg yields an improvement of +9.77%p over the base- line and an average advantage of +2.01 %p over competitors. Compared to HSSN and LogicSeg, the gains are +1 +1.26%p and +0. 40%70%p respectively. HiPoSeg reaches the highest accuracy (41.429 11.4%)) us- ing ResNet- 101, surpassing matching methods. This demonstrates the scalability and robustness of hierarchical prototype learning on diverse large-scale street-scene datasets, where fine- grained distinctions are particularly challenging.

---

Published as a conference paper at ICLR 2026
Table 7: High-level IoU comparison between baseline and HiPoSeg on the Cityscapes val set.
Table 8: Low-level IoU comparison between baseline and HiPoSeg on the Cityscapes val set.
high class
basclinc (%)
HiPoSeg (%)
gap (%pp)
low class
baseline (%)
HiPoSeg (%)
gap (%p)
Blat
96.54
96.90
+0.36
road
98.10
98.44
+0.34
construction
89.75
90.88
+1.13
sidewalk
85.15
86.87
+1.72
object
72.06
74,31
+2.25
buildingg
91.76
93.41
+1.65
nature
90.53
91.48
+0,95
wall
40.06
52.54
+12.48
sky
95.02
95.48
+0.46
human
80.25
82.61
+2.36
fence
55.33
62.49
+7.16
vehicle
87.06
93.49
+6,43
pole
66.21
70.85
+4,64
traffic light
68.63
76.48
+7.85
95.0
91.31
93.37
traffic sign
78.30
81.79
+3.49
22.16
HSSN
Logiteg HiluSeg
vegetation
92.09
92.96
+0.89
90.0
terrain
57.44
63.05
+5.61
87.5
301430.17 (19-77)
sky
94.52
95.48
+0.96
85.0
11:0.08 030
83 02 83.20
person
81.01
85.19
+4.18
80.9
rider
57.83
67.84
+10.01
car
92,88
96.14
+3.26
77.5
truck
54.15
84.05
+29.90
mioo
mloi
69.93
89.83
+19,90
train
23.88
83.47
+59.59
Figure 3: Comparison of m and mloU on Cityscapes (mloU' for the low level, mloU² for the high level).
motorcycle
58.76
73.02
+14.26
bicycle
75.91
80.92
+5.01
PASCAL-Part-108. From Tab, 4 HiPoSeg improves the baseline by +2.43%p mloU and achieves an average margin of +4.41%p over other approaches. Compared to HSSN and LogicSeg, the method ensures improvements (49.33%). Given the fine- fine-grained, imbalanced nature of this dataset, these results underline the ability of HiPoSeg to handle rare and small object parts. By enforcing hierarchical alignment, the method reduces inconsistencies between object-level and part-level predictions, yielding more herent parsing.
+1.01%p and +0.87%p, respectively, reaching the best accuracy
4.3 ABLATION ANALYSIS
Tab. shows the effeet of each loss term. Using only cross-entropy (73.55%) leaves sibling confu- sions and weak boundaries. High-level contrast (80.96%) structures the parent space and prunes fine-level candidates, while low-level contrast (81.04%) sharpens boundaries and small objects. Combining both without alignment (79.16%) hurts performance due to gradient interference and high-low mismatch. With the proposed alignment (84.04%), the hierarchy preserved and learning proceeds from coarse to fine: parents stabilize first, then fine classes converge.
Tab. aebalueess hierarchical prototypes. Using only parents only children improves the baseline by regularizing global semantics or refining local details, respectively. Using both yields the best mloU across Cityscapes, ADE20K, Mapillary, and PASCAL-Part-008 (84.04%, 48.99%, 41.42%. and 49.33%), confirming their complementarity: parent prototypes constrain the candidate space, while child prototypes drive fine-grained separation.
Fig. reports the results of Cityscapes comparing the baseline and HiPoSeg mIoU (low-level labels) and mloU² (high-level labels). HiPoSeg exceeds competing methods in both metrics and exhibits a smaller gap between them (9.77%p) than other approaches, suggesting that hierarchical prototype learning mitigates cross-level inconsistencies. Tab. 7 [nd visualize per-class IoU at high and low levels, respectively; HiPoSeg consistently exceeds the bascline both, indicating strong predictive accuracy throughout the hierarchy. All baseline models are 33.++.
8

---

Published as a conference paper at ICLR 2026
road sdnwalk
pole
traffic light
traffic skinn
vegutation
sky
person
rider
motarcyclo
bicycle
Figure 4: Baseline (left) and our method (right) 1-SNE visualization of the feature space on the Cityscapes val set. Colors for each class are indicated in the legend the right, and similar color tones denote classes belonging to the same higher-level concept (e.g., 'road' and 'sidewalk' fall under the higher-level category 'flat').
Figure 5: Qualitative results dense small objects and occlusion. Our method clearly segments a bus occluded by a fence or cars, and accurately predicts a traffic light located among vegetation.
4.4 QUALITATIVE EVALUATION
Fig. 4 visualizes the feature space on the Cityscapes val set using t-SNE for the baseline (left) and HiPoSeg (right). Compared to the baseline, which shows heavier overlap among categories and less distinct separation even between semantically related classes, HiPoSeg produces a more structured embedding space: classes belonging the same higher-level concept form ters, while different higher-level groups become more clearly separated. This qualitatively supports that HiPoSeg injects hierarchy into the representation space itself, learning clearer coarse-to-fine semantic organization rather than relying on post-hoc correction at the output stage.
coherent clus-
In addition, Fig. 5 provides qualitative results in challenging cases with dense small objects and severe occlusion. For instance, our method cleanly segments bus heavily occluded by fence or cars, and accurately predicts small traffic light located among vegetation. These examples suggest that HiPoSeg improves not only boundary delineation but also small-object recognition and robustness under severe occlusion.
O presents qualitative comparisons between the baseline and the proposed HiPoSeg on three challenging benchmarks. HiPoSeg delivers cleaner segmentations than the baseline on all datasets, consistently improving boundary sharpness, small-object recognition, and semantic consistency. For instance, in the left Cityscapes example, the baseline mixes the bus with 'building' whereas HiPoSeg correctly segments it as 'bus', indicating that hierarchical label learning effectively reduces confu- sion. Similarly, in the right ADE20K example, the baseline confuses the sea with the sky, while HiPoSeg correctly identifies the sea, demonstrating consistent recognition across parent-child cate-

---

Published as a conference paper at ICLR 2026
baseline
HiPoSeg
(a) Cityscapes
baseline
HiPoSeg
(b) ADE20K
baseline
HiP PPoeee
(c) Mapillary Vistas 2.0
Figure 6: Qualitative case studies on three benchmarks: (a) Cityscapes, (b) ADE20K, and (c) Map- illary Vistas 2.0.
gories. Overall, these qualitative results show that HiPoSeg reliably enhances boundary delineation, small object recognition, and semantic consistency across diverse datasets, validating the effective- ness of hierarchical prototype learning for fine-grained segmentation.
5 CONCLUDING REMARKS
We have proposed prototype- based hierarchical method of HiPoSeg for semantic segmentation. Unlike flat classification approaches, organizes class representations into hierarchical prototype space and enforces cross-level consistency through contrastive learning, enabling a coarse-to- recognition process. Extensive experiments on semantic segmentation benchmarks demonstrate that contrastive methods,
it consistently outperforms strong baselines and previous hierarchy-awer with an average gain +3.07%p mloU, without incurring additional inference cost. Beyond accu- racy, improves semantic consistency across label hierarchies, reducing confusion between unre- lated classes, and enhancing fine-grained part segmentation.
In future work, will extend HiPoSeg to open-vocabulary and long-tailed segmentation, where structured prototypes can mitigate data imbalance, and to multi molti-modal perception tasks such as vasion-l segmentation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by the Yonsei Fellow Program funded by Lee Youn Jae, and IITP grant funded by the Korea government (MSIT) (No. R6-2020-11201661, Artificial Intelligence Graduate School Program (Yonsei University); No. RS-2022-11220133 Developing Sustainable Collabora- tive Multi-modal Lifelong Learning Framework).
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. SegNet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence (TPAMI), 39:2481-2495, 2015. URL https:://aii.orr/100..
TPAMI. 2016 2644615
Moshe Bar. A cortical mechanism for triggering top-down facilitation in visual object recognition.
Journal of Cognitive Neuroscience, 15:600-609, 2003. URL /
Moshe Bar. Visual objects in context. Nature Reviews Neuroscience, 5:617-629, 2004. URL
https: //tp::://oii.oo oro.ioorg1100 10..:..11330mmmmmm
Moshe Bar. The proactive brain: Memory for predictions. Philosophical Transactions of the Royal
Society B: Biological Sciences, 364:1235-1243, 2009, URL https: /ttp::/::///.i...//0/0/0 10.99/001/11
rstb. 2008. 0310
Moshe Bar, Karim S. Kassam, Avniel Singh Ghuman, Jasmine Boshyan, Anita M. Schmid, An- ders M. Dale, Matti S Hämäläinen, Ksenija Marinković, Daniel L. Schacter, Bruce R. Rosen, and Eric Halgren. Top-down facilitation of visual recognition. Proceedings of the National Academy of Sciences of the United States of America, 103(2):449-454, 2006. URL https: //doi.org//// www.://///....////..... 10.3///..aa.s....... pnas. 050 7062103
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 801-818, 2018. URL https:
//doi.or//10..100//991--
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey E. Hinton. simple frame- work for contrastive learning of visual representations. In Proceedings the International Conference on Machine Learming (ICML), volume 119, pp. 1597-1607, 2020. URL https:
//proceedings mlr.
ptwoeeddnns..mxx.rres/ssll.. wwwpppeesssssiee chen20j..... html
Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In Advances Neural Information Processing Sys- tems (NeurIPS), volume 30, pp. 17864-17875, 2021. URL https:://d/l wttp:/o........... 485507
arXiv. .0210..067788
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 3213-3223, 2016. URL pretps://don ott.10.11/1119.1 CVPR. 2VPR.2016.35)
Jun Fu, Jing Liu, Haijie Tian, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3146-3154, 2019, URL http::://doi..rgg//// 1109///RRR 2019 9. 00326
Hanzhe Hu, Jinshi Cui, and Liwei Wang. Region-aware contrastive learning for semantic segmenta- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16291-16300 2021. http::://oo..ooo....../////////.
/ooi
2021 01598

---

Published as a conference paper at ICLR 2026
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang. Yunchao Wei, Humphrey Shi, and Wenyu Liu. CCNet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 603-612, 2019. URL
ht.pps:/
1ttp///...11/TCCC 201 1.009.000 00069
Ruyi Ji, Dawei Du, Libo Zhang, Longyin Wen, Y. Wu, Chen Zhao, Feiyue Huang, and Siwei Lyu. Learning semantic neural tree for human parsing. In Proceedings of the European Conference Computer Vision (ECCV), volume 12358, pp. 205-221 2020. URL RtLhtttp: /ttapp://doi.oo http///o.ogg.....oo
Robin Inho Kee, Dahyun Nam, Seok-Jun Buu, and Sung-Bae Cho. Disentangled prototypical con- volutional network for few-shot learning in in-vehicle noise classification. IEEE Access, 12: 2024.
httpa::///doooooooo org.100..11... wttp:///o..110//C.3.... 222.://CEESSS.22...... 3397842
Ji- Yoon Kim and Sung-Bat Bung-Bae Cho. Ensemble of diverse deep neural networks with pseudo-labels for repayment prediction in social lending. Science Progress, 105, 2022. URL. httpss///oo.
Hanchao Li, Pengfei Xiong, Jie An, and Lingxue Wang. Pyramid attention network for seman- ott..o00.1...../...11.....
tic segmentation. a80iv: 10000 2018. URL
Liulei Li, Tianfei Zhou, Wenguan Wang, Jianwu Li, and Yi Yang. Deep hierarchical semantic seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR). pp. 1246-1257, 2022. URL /
2022.0013
Liulei Li, Wenguan Wang. and Yi Yang. LogieSeg: Parsing visual semantics with neural logic learning and reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4122 4133, 2023. URL
2023. 00381
Panagiotis Meletis and Gijs Dubbelman. Training of convolutional networks on multiple hetero- geneous datasets for street scene semantic segmentation. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), pp. 1045-1050, 2018. URL
2018. 8500398
Umberto Michieli, Edoardo Borsato, Luca Rossi, and Pietro Zanuttigh. GMNet: Graph matching network for large scale part semantic segmentation in the wild. In Proceedings of the European Conference on Computer Vision (ECCV), volume 12353, pp. 397-414, 2020. URL https:
1007/978 3- 030 58598- 2_
Hyung-Jun Moon and Sung-Bae Cho. Continual learning by contrastive learning of regular- ized classes in multivariate gaussian distributions. International journal of neural systems, pp. 2550025, 2025.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulò, and Peter Kontschieder. The Mapillary Vistas dataset for semantic understanding of street scenes. In Proceedings the IEEE In- lernational Conference on Computer Vision (ICCV), pp. 4990-4999, 2017. URL https:
[ídoi.org/ll 11.00.1..00 1109/ ICCV. 2017.534
Theodoros Pissas, Claudio S. Ravasio, Lyndon Da Cruz, and Christos Bergeles. Multi-scale and cross-scale contrastive learning for semantic segmentation. In Proceedings of the Eu- ropean Conference Computer Vision (ECCV), volume 13689, pp. 413 429, 2022. URL
fettps://doi.org//21 100///00...1//989188/ 3---3--333 11--031--88 1111111988-22200 24
Lorenzo Porzi, Samuel Rota Bulò, Aleksander Colovic, and Peter Kontschieder. Seamless scene segmentation. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8277-8286, 2019. URL http:///...../...11111/////// 201 9. 00847
12

---

Published as a conference paper at ICLR 2026
Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp- 3431-3440, 2015. URL /ttp::://oo...o///0...
2015.
7298965
Rishu Singh, Pranav Gupta, Pradeep Shenoy, and Ravi Kiran Sarvadevabhatla. FLOAT: Factorized learning of object attributes for improved multi-object multi-part scene parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1445-1455,
2022. URL https: / hwt. org/ 10. 1109/ CVRRC2 688 2022. 00150
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017. URL https:
//doi. ooggioogg/// 0. 48 48550/arXiv 703.000000000 05175
Chan-Yong Sung, Wanhee Kim, Jungho An, Wooju Lee, Hyungtae Lim, and Hyun Myung. Contex- trast: Contextual contrastive learning for semantic segmentation. In Proceedings the the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3732-3742, 2024. URL
22W.CV222223332221222... 203430035888 00358
Maofeng Tang, Konstantinos Georgiou, Hairong Qi, Cody Champion, and Mare Bosch. Semantic segmentation in acrial imagery using multi-level contrastive learning with local consistency. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3798-3801 2023. URL htt.hhttp:///tt ort..o../1111.1...... htt/://./...11///////666333333 2023. 2023.0037
Wenguan Wang, Hailong Zhu, Jifeng Dai, Yanwei Pang, Jianbing Shen, and Ling Shao. Hierarchical human parsing with typed part-relation reasoning. Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pp. 8929-8939, 2020. URL https:
//doi.org/10.. 11tp:1111////pp200010 2020. 00895
Wenguan Wang, Tianfci Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Ex- ploring cross-image pixel contrast for semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 7303-7313, 2021. URL https:
7/doi. otg/10..r110..0
2021. 00721
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, José Manuel Álvarez, and Ping Luo. SegFormer: Simple and efficient design for semantic segmentation with transformers. In Ad- vances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 12077-12090, 2021.
URL httphhttss/eeee /ttp:::://ooiiooo www.oooooooooo/. 4tt.:/1....558000 wwt.:/440550000000000 15203
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic seg- mentation. In Proceedings of the European Conference Computer Vision (ECCV), volume 12351, pp. 173-190, 2020. htthhtts:/// http://o....0//1..00000////71--1 0-00779978-300305553------
Fan Zhang, Yanqin Chen, Zhihang Li, Zhibin Hong, Jingtuo Liu, Feifei Ma, Junyu Han, and Errui Ding. ACFNet: Attentional class feature network for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6798-6807, 2019. URL
https: /ww. wwrii.oog/111 11t.1/00..11100000 wtt.://.............
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2881 2889- 2017. URL httpp:://cc....... .rreei..rr/// 10. 1119//VPR..../ 2017 660
Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. PSANet: Point-wise spatial attention network for scene parsing. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 267-283, 2018. WtRLhttps:/// :/dtts http:///ooooooo...////
1007/970 7978 10778-3-030--200
17
Yifan Zhao, Jia Li, Yu Zhang. and Yonghong Tian. Multi-class part parsing with joint boundary- semantic awareness. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 9177-9186, 2019. URL https: /tts:///co..ooo http://ttto....oo.//oo///o.o.ett http:///0...0//0.1111///////// 2019. 00927
13

---

Published as a conference paper at ICLR 2026
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. International Journal of Computer Vision, 127:302-321, 2016. URL httppp///oo...crrr.......///111.
Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmen- tation: A prototype view. In Proceedings of the IEEE/CVF Conference on Computer Vision and 1109/
Pattern Recognition (CVPR), pp. 2582-2593, 2022. URL
CVPR52688, 202.2226888022200222022
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A USE OF LLMS
In this work, we used an LLM (ChatGPT) solely general- purpose assistive tool. Specifically, it was used to polish the writing, rephrase sentences for more natural and clearer expression, and check grammatical consistency. In addition, it was used to generate the Human and Model illustrations in Fig. I Beyond these uses, the LLM was not involved in research idcation, the design of the proposed method, the execution of experiments, or the analysis of results; all such technical contributions were conceived and carried out entirely by the authors.
B EXTENDED EXPERIMENTAL RESULTS
Backbone generalization (Table 9) Table summarizes the results on multiple backbones and benchmarks. Our method yields consistent improvements on both KRRNet-W48 and SegFormer across the evaluated datasets, supporting troduce any backbone-specific hyperparameters for HRNet-Waad or SegFormer, and we observe no performance degradation any evaluated configuration. Notably, the gains remain present even on strong modern architecture such SegFormer, suggesting that our hierarchical prototype space complements feature formation rather than relying on backbone- or -eccofe--ppecffic output- stage heuristics. Overall, these results strengthen claim that the proposed method is safe, architecture taniitccturr-agossiic plug-in module whose advantages persist across both CNN- and Transformer- based segmentation frameworks.
behavior. Importantly, we do not
σ1 controls the low-high semantic proximity, while dg regu- lates the separation among high-level prototypes. As shown in Tob inadequate proximity (too small r) or insufficient separation (too small 02) leads to noticeable performance drops. Overall, σ1 =1=0.25 and d2=1.00 provide the best balance between semantic compactness and discrimination, yielding the highest mloU.
Effect of 01 and 02 (Table 10
Momentum coefficient m (Table 11). We evaluate how the momentum used the prototype- memory update affects segmentation performance. As reported in Table [1] moderate momentum (m=0.90) achieves the best trade-off between stability and adaptability, resulting in the highest mloU among the tested values.
Staged vs. joint training (Table 12 timization setup in Table 12 Activating hierarchical constraints after an initial feature stabilization phase mitigates early prototype drift and leads to consistently better convergence, which translates into higher mloU. These results confirm that the chosen hyperparameters and the staged activation mechanism are important for stable hierarchical prototype learning and for achieving performance improvements.
We compare sequential hierarchical learning with a joint op-
Hierarchical IoU analysis on Mapillary Vistas 2.0. We provide hierarchical breakdown of the segmentation performance on Mapillary Vistas 2.0 val. As shown in Table [3 and Table [4] HiPoSeg substantially improves the recognition of high-level semantic groups, yielding large gains on several major categories (c.g.. construction, object, and marking at the top level, and vehicle, flat, and sorkknggdiscreet the high level). These results indicate that the proposed hierarchical prototype space effectively mitigates broad semantic confusions that are common in large-scale street-scene taxonomies.
Fine-grained improvements at the low level. Table 15 further reports the class-wise IoU at the low level. Beyond the gains at higher abstraction levels, HiPoSeg also improves numerous fine- grained classes, including several structure- and marking-relatee categories as well as small or thin objects (e.g., various traffic-relteed classes). While small number of categories show marginal regressions, the overall trend remains strongly positive, suggesting that the hierarchical constraints primarily enhance discrimination without sacrificing fine-detail segmentation.
15

---

Published as a conference paper at ICLR 2026
Table 9: Backbone generalization results across datasets.
Backbone
Cityscapes
PASCAL-Part-008
ADE20K
Mapillary
ResNet-101
73.55
46.90
44.48
31.65
HiPoSeg
84.04
49.33
48.99
41.42
HRNet- HiPoSeg
80.85
43.77
38.59
30.17
81.39
43.98
38.95
43.48
SegFormer 80.11
51.32
43.69
44.29
HiPoSeg
80.63
51.59
44.29
45.04
Table 10: Effect of 01 and d2.
Table 11:
Momentum coeffi
Table 12: Staged vs. joint train- ing.
cient m.
01
02
mIoU (%)
0.10
0.50
82.30
mloU (5)
training joint
mloU (%)
0.10
1.00
82.37
81.34
82.47
0.25
0.25
0.50
1.00
81.53
84.04
0.90
0.99
84.04
83.28
sequential
84.04
Figure/Jproviee qualitative comparisons on Cityscapes, ADE20K, and Map- illary Vistas 2.0. Compared to the baseline, HliPSSg produces clearer object boundaries and more coherent regions, improves segmentation of small or visually ambiguous objects, and reduces con- fusions among visually similar classes. These observations consistent with the quantitative im- provements reported in Table 13 Table [5] supporting that our method improves both semantic grouping (high-level correctness) and fine-grained delineation (low-level accuracy).
Qualitative results.
16

---

Published as a conference paper at ICLR 2026
high-tovel
baseline (%)
HiPoScg (8)
(Pp)
barrier
34.07
65.98
+31.91
flat
39.39
89.74
+50.55
top-level
buseline (%)
HiPoSeg (5)
gap (%p)
siructure
57.95
40.1
71.30
87.01
+29.06
+31.19
construction
40.14
91.16
+51.22
rider
33.76
54.85
+21.09
bumman
36.30
71.92
+35.62
marking continious
36.46
57.94
+21.48
marking
20.12
69.02
+48. -48.90
murking disccrete
21.97
68.01
+46.04
nature
63.50
96.06
+32.56
nature
63.50
96.06
+32.56
object
21.54
36.38
+14.84
object
30.46
78.02
+47.56
sign
14.70
49.39
+34.69
void
40.50
62.19
+21.69
support
45.96
52.59
+6.63
traflic light
27.64
67.07
+39.43
traffic sign
33.61
67.04
+33.43
Table 13: Top-level IoU comparison on Mapil- lary Vistas 2.0 val.
vehicle
38.58
90.46
+51.88
void
40.50
62.19
+21.69
Table 14: High-level IoU comparison on Map-
illary Vistas 2.0 val.
low-level
buseline (%)
HiiPoSe (%)
gap (Rp)
low-level
baseline
HliPoSg (9) gop (Sip)
Concrete
69.79
70.32
62.28
Curb
58.80
61.09
+2.29
Other Barrier
0.00
0.00
0.00
Guard Rail
63,70
64.85
+1.15
Road Side
9.61
6.36
33.25
Median
Temporary
Lane Separator
17.52
19.44
+1.92
Bike Lane
39.64
41.52
+1.88
Wall
50.21
$1.28
+1.07
Curh Cut
20.64
22.18
+1.54
Crosswalk Plain
31.94
33.34
+1.40
Parking
19.24
15.61
-3.63
Driveway
19.12
20.13
+1.01
Pedestrian Area
29,50
49.77
+20.27
Parking Aisle
0.00
3,44
+3.44
Road
87.84
Rail Track
53.63
$4.12
+0.49
Service Lane
47.32
45.58
-1.74
Sidewalk
65.29
69.18
+3.89
Traffic Island
38,54
36.72
-1.2
Bridge
Building
Tunnel
12.99
65.62
+52.63
Person Group
7.21
8.60
+1.39
Individual
68.72
71.62
+2.90
Motorcyclist
46.21
53,89
+7.68
Bicyclist
46.52
Lane Marking Anow (Leff)
Lane Marking Dashed Line
43.01
46.76
+3.75
Lane Marking Crosswalk
71.74
74.11
+2.37
Marking Straight Line
60.18
62.62
+2.44
Lane Marking Hatched (Diagonal)
48.84
50.86
+2.02
Lane Marking Arrow (Straight)
24.89
24.74
-0.15
Lane Marking Stop Line
45.65
50.05
+4.00
Lane Marking Hatched (Chevron)
37.13
35.83
-1.30
Lane Marking Symbol (Bicyele)
44.88
49.79
+491
Marking
34.92
Marking-
Mountain
45.05
$0.99
+5.94
Sky
97.85
97.86
+0.01
Sand
0.00
0.01
+0.01
Terrain
65.28
65.06
40.22
73.35
Vegetation
Water
78.67
64.65
-14.02
Banner
28.66
31.56
+2.90
Bench
39.21
38.42
-0.79
Bike Rack
0.00
0.52
+0.52
Fire Hydrant
9.28
51.28
+42.00
Catch Basin
39.78
+0.18
Junction Box
36.77
44.92
+8.15
Manhole
$1.55
$1.97
+0.42
Signago Advertisement
45.34
47.38
Signage
Signuge Information
0.52
1,68
+1.16
Signage Store
39,59
39.05
0.54
Street Light
47.31
48.53
+1.22
Pole
42.06
46.85
+4.79
Pole Group
9.10
Sign
Utility Pole
45.65
$1.58
+5.93
Trafic Cone
56.61
61.60
+4.99
Traffic Light Pedlestriass
40.36
47.35
+6.99
Traficc Light
General (Horizantal)
47.63
50.71
+3.08
Traffic Light General (Upright)
64.12
+3.67
Traffic Light
Direction (Back)
41.74
Traffic Sign Amhiguous
0.00
1.82
+1.82
Traffic Sign
Direction (Front)
69.94
71.07
+1.13
Traffic Sign (Front)
59.28
62.52
+3.24
Traffic Sign Parking
17.55
34,20
+16.65
Traflic Sign Temporary (Freat)
0.00
10.72
+10.72
Trush Cán
41.74
42.68
+0.94
Bicycle
$3.57
$8.20
+4.63
Boat
1.71
20.46
+18.75
Bus
Motorcycle
53.16
57.91
+4.75
On Rails
28.84
46.50
+17.66
Other Vehicle
17.13
18.22
+1.09
Truck
71.55
73,96
+2.41
Vehicle Group
Dynamic
Ego Vehicle
89.18
92.33
+3.15
Ground
30.56
33.29
+2.73
Static
18.74
20.23
+1.49
Unlabeled
0.37
0.55
+0.18
Table 15: Low-level loU comparison between baseline and HiPoSeg the Mapillary Vistas 2.0
val valle set.
17

---

Published as a conference paper at ICLR 2026
baseline
HiPoSeg
(a) Cityscapes
baseline
HiPoSeg
(b) ADE20K
baseline
HiPoSeg
(c) Mapillary Vistas 2.0
Figure 7: Compared to the baseline, HiPoSeg yields clearer boundaries, improved small-object seg- mentation, and fewer confusions among visually similar classes.
18


Reviews:
[{"review_id": "6KcpIBYT0F", "reviewer": "Reviewer_hVxW", "summary": "Existing semantic segmentation methods often treat the task as flat classification, ignoring structural relationships between classes and failing to replicate human “whole-to-part” hierarchical recognition, leading to poor fine-grained part segmentation. To address this, the paper proposes HiPoSeg, a hierarchical prototype-based method that constructs a structured prototype space capturing object-level and part-level features, uses hierarchical contrastive learning for intra-level discrimination and cross-level consistency, and only acts during training (no inference overhead). Experiments on Cityscapes, ADE20K, Mapillary Vistas 2.0, and PASCAL-Part-108 show an average +3.07%p mIoU gain over baselines and state-of-the-art methods. Its core contributions include: 1) hierarchical prototype learning for structured class representation; 2) multi-level alignment constraints to suppress semantic leakage; 3) a plug-and-play, training-only design with zero inference cost.", "questions": "1. Experimental Design:\nCould you provide results of HiPoSeg on alternative backbones (e.g., HRNet-W48, SegFormer-B5) to verify its generalization? If performance drops on certain backbones, what is the root cause?\nCould you add a hyperparameter sensitivity study (varying σ₁, σ₂, m, and temperature τ/κ) and explain how you selected the optimal values?\nCould you supplement qualitative results on challenging scenes (dense small objects, severe occlusion) and analyze error cases (e.g., which low-level classes still have low IoU)?\n2. Motivation and Comparison:\nHow does HiPoSeg’s hierarchical prototype design specifically outperform 2024 SOTA methods like Contextrast (which uses contextual contrastive learning)? Could you add a head-to-head comparison with these methods on the same benchmarks?\nPrior work (e.g., ProtoSeg) also uses prototypes—why is combining prototypes with hierarchy more effective than flat prototype learning? Could you provide a controlled experiment comparing HiPoSeg with a flat prototype baseline?\n3. Theoretical and Mechanistic Analysis:\nCould you provide t-SNE or UMAP visualizations of the prototype space to show how high/low-level prototypes of related classes (e.g., “car”/“truck” in high-level, “horse head”/“horse leg” in low-level) are structured?\nFor rare low-level classes (e.g., “train” in Cityscapes, which has a baseline IoU of 23.88%), how does HiPoSeg’s prototype memory bank address data sparsity? Could you analyze the prototype update process for these classes?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Unlike prior hierarchical segmentation methods (e.g., HSSN, LogicSeg) that handle hierarchy as a fixed auxiliary term or fuse signals at the probability level, HiPoSeg explicitly constructs a hierarchical prototype space and uses contrastive learning to structure class representations—this design principle is a meaningful departure from existing work.\n2. The experimental design is relatively systematic: it validates performance on 4 diverse benchmarks (urban scenes, daily scenes, part parsing), and ablation experiments (Table 5-6) confirm the necessity of high-level/low-level contrastive losses and alignment constraints, ensuring the method’s core components are evidence-based.\n3. The method description is detailed: Section 3 clearly defines prototype construction (Eqs. 1-4), hierarchical contrastive learning (high/low-level losses), and alignment constraints (Eqs. for 𝒱ₐₗᵢ₉ₙ), making the technical implementation reproducible in principle.\n4. The “zero inference overhead” design is practically valuable—HiPoSeg acts as an auxiliary training component, avoiding the common trade-off between performance gain and deployment cost in segmentation research, which is useful for real-world applications (e.g., autonomous driving).", "weaknesses": "1. Experimental Completeness: No hyperparameter sensitivity analysis: The paper sets σ₁=0.25, σ₂=1, m=0.9 without justifying why these values are optimal or how performance changes if they vary (e.g., σ₁=0.1 vs. 0.5).\nLack of alternative backbone validation: All experiments use ResNet-101, but no results on mainstream backbones like HRNet-W48 or SegFormer are provided, making it unclear if HiPoSeg’s gains generalize beyond ResNet.\n2. Insufficient qualitative challenging cases: The qualitative results (Figure 3/4) only show simple scenes (e.g., clear buses in Cityscapes) and omit scenarios with dense small objects (e.g., crowded pedestrians) or severe occlusion—critical for verifying fine-grained segmentation ability.\n3. Motivation and Context:The related work section does not contrast HiPoSeg with 2024 SOTA methods (e.g., Contextrast’s contextual contrastive learning) in depth, only listing them in tables. It fails to explain why HiPoSeg’s hierarchical prototype design outperforms these latest methods, weakening the motivation’s urgency.\n4. No failure case analysis: The paper only reports successful segmentation results but does not analyze when HiPoSeg fails (e.g., which classes/regions still have high error rates) or why, limiting insights into the method’s limitations.\n5. Theoretical Depth:The paper does not provide quantitative analysis of prototype space (e.g., t-SNE visualizations of high/low-level prototypes or feature similarity metrics between related classes like “truck” and “bus”). It only claims “structured representation” without empirical evidence of how prototypes reduce cross-class confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9FQylnfcrS", "reviewer": "Reviewer_5rqn", "summary": "This paper introduce HiPoSeg, a approach to semantic segmentation that leverages hierarchical prototype learning to improve segmentation performance, particularly for fine-grained tasks. The key idea is to model both high-level categories and their subcategories through hierarchical prototypes, allowing the model to capture both coarse and fine-grained semantic structures.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper introduces hierarchical prototype learning for semantic segmentation, combining prototype-based learning with a hierarchical structure, which improves the model’s ability to differentiate fine-grained subcategories.\n* The problem is clearly defined, and the paper is well-structured,  presenting experimental results in an understandable manner.\n* HiPoSeg improves segmentation results, and its scalability and flexibility suggest broad potential for various applications.", "weaknesses": "* This paper uses DeepLabv3, a classic but somewhat outdated baseline. To validate the generalizability of this method, the authors should consider using a wider variety of CNNs and Transformer-based baselines. Also, this can demonstrate the author's claim of \"plug-and-play efficiency\".\n\n* The paper mentions the novelty of hierarchical prototype learning, but it lacks a discussion with other similar methods, such as other prototype-based semantic segmentation methods or hierarchical network approaches.\n\n* It would be useful to clearly distinguish the fundamental differences between this approach and existing hierarchical methods, for example, does the prototype learning mechanism in HiPoSeg offer better generalization ability? Does it form a clearer semantic structure in the feature space?  Does HiPoSeg more effectively leverage logical constraints or hierarchical consistency?\n\n* It may be better to add visual analysis (e.g., t-SNE or prototype space structure plots) to demonstrate how the prototype space is organized and whether the hierarchical structure is reflected in the learned representations.\n\n* While the paper introduces hierarchical prototype learning for handling fine-grained segmentation tasks, there has not been sufficient exploration of its performance on fine-grained subcategories and its advantages in these tasks.\n\n\nI would be happy to revise my score if the author addresses these points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BRGfrv80xt", "reviewer": "Reviewer_Bxs7", "summary": "* The paper introduces Hierarchical Prototype Segmentation (HiPoSeg), a new framework for semantic segmentation that models visual recognition as a coarse-to-fine process, reflecting how humans perceive objects.\n* HiPoSeg builds a hierarchical prototype space that captures both high-level object semantics and fine-grained part semantics, linking them through a hierarchical contrastive learning objective. To maintain consistency across these levels, the model applies a multi-level alignment constraint, reducing misclassifications between unrelated categories.\n* A key strength of HiPoSeg is that it functions as a training-only, plug-and-play module, meaning it adds no extra computational cost during inference while still enhancing semantic consistency and mean Intersection over Union (mIoU).\n* Tested on four major benchmarks, Cityscapes, ADE20K, Mapillary Vistas 2.0, and PASCAL-Part-108, HiPoSeg consistently outperforms strong baselines such as HSSN, LogicSeg, and ProtoSeg, achieving notable improvements across all datasets.", "questions": "Can the hierarchical prototype space be learned when the label hierarchy is not predefined?\nWhat is the rationale behind the curriculum schedule for enabling hierarchical components, and how sensitive is the model to these thresholds?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper clearly formalizes hierarchical semantics by organizing prototypes in a top-down hierarchy, addressing the structural limitations of flat segmentation.\n* The proposed component is used only during training, requiring no additional computation or parameters at inference, which is practically appealing.\n* Cross-level semantic consistency: Through an alignment constraint, HiPoSeg maintains strong coherence between high-level and low-level features, minimizing semantic drift and enhancing the precision of fine-grained segmentation.\n* Comprehensive validation: The paper provides extensive quantitative and qualitative analyses across multiple benchmark datasets, clearly demonstrating the robustness and generality of the method.\n* HiPoSeg achieves higher mIoU than SoTA models (HSSN, LogicSeg, ProtoSeg, ContextSeg), even with a weaker backbone (ResNet-101), underscoring its efficiency and scalability.", "weaknesses": "* While the hierarchical prototype framework is well-motivated, it conceptually overlaps with prior hierarchical reasoning models such as HSSN (CVPR 2022) and LogicSeg (ICCV 2023). The improvement may be seen as incremental rather than fundamentally new.\n* The experiments focus primarily on two-level hierarchies. The extension to deeper hierarchies (e.g., three-level settings in ADE20K and Mapillary) is only lightly discussed, leaving generalization uncertain.\n* Although the paper presents loss-wise ablations, it lacks an analysis of hyperparameter sensitivity.\n* The qualitative comparisons rely heavily on visual inspection of boundary sharpness and small-object recognition, which are somewhat subjective and lack quantitative support.\n* Stability under multi-term losses: The hierarchical contrastive loss combines multiple objectives (Lfh, Lfl, Lalign), which may cause gradient interference in large-scale training unless carefully balanced.\n* HiPoSeg assumes a fixed and known label tree, making it less applicable to datasets without clear hierarchical semantics or dynamically evolving taxonomies.\n* Although mentioned in the conclusion, the paper does not empirically validate HiPoSeg in open-vocabulary or long-tailed scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HIJgN9PtVb", "reviewer": "Reviewer_LYNi", "summary": "Traditional semantic segmentation methods often overlook semantic relationships across levels. To address this issue, the article proposes a Hierarchical Prototype Learning (HiPoSeg) method. By introducing high- and low-level prototype contrastive learning and high- and low-level alignment constraints, it effectively combines coarse categories and fine-grained part semantics, improving segmentation accuracy and consistency. Experiments across multiple datasets validate the proposed HiPoSeg method's ability for fine-grained segmentation and semantic consistency.", "questions": "- Some table captions are above the tables, while others are below. It is recommended to review and adjust the table layout for greater clarity.\n\n- In the first paragraph of Section 3, \"...prototype space in 3.1. Second, in 3.2...\" should be revised to \"...prototype space in Sec. 3.1. Second, in Sec. 3.2...“.\n\n- There is an unnumbered equation between Equation 1 and Equation 2.\n\n- Table 7 should be a figure, not a table.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper effectively integrates coarse- and fine-grained semantics via hierarchical contrastive learning, thereby improving segmentation accuracy. \n\nIt enhances semantic consistency across different levels by employing alignment constraints between high- and low-level features.", "weaknesses": "- In Section 3.2, regarding the setting of some hyperparameters (e.g., \"σ1=0.25, σ2=1\" and \"m=0.9\"), providing an analysis of how the choice and adjustment of these hyperparameters affect performance would help enhance the reproducibility and understanding of the method. Additionally, the paper introduces a step-by-step activation mechanism at different stages (from feature representation to prototype learning, and then to alignment loss). It is recommended to provide more experimental or theoretical support to clarify why this staged training arrangement effectively facilitates hierarchical learning.\n\n- The explanation in Section 3 could benefit from more step-by-step details, especially regarding prototype construction. Although Figure 2 provides a good overview of the overall structure, adding more detailed annotations and explanations would help readers better understand each step.\n\n- The latest relevant methods are missing from the comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)