Diving Segmentation Model into Pixels
Authors: Chen Gan
Zihao Yin
Kelei He
Yang Gao
Junfeng Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Semantic Segmentation
Pixel Learning
Representation Learning
Contrastive Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: More distinguishable and consistent pixel features for each category will benefit the semantic segmentation under various settings.
Existing efforts to mine better pixel-level features attempt to explicitly model the categorical distribution, which fails to achieve optimal due to the significant pixel feature variance.
Moreover, prior research endeavors have scarcely delved into the thorough analysis and meticulous handling of pixel-level variance, leaving semantic segmentation at a coarse granularity.
In this work, we analyze the causes of pixel-level variance and introduce the concept of $\textbf{pixel learning}$ to concentrate on the tailored learning process of pixels to handle the pixel-level variance, enhancing the per-pixel recognition capability of segmentation models.
Under the context of the pixel learning scheme, each image is viewed as a distribution of pixels, and pixel learning aims to pursue consistent pixel representation inside an image, continuously align pixels from different images (distributions), and eventually achieve consistent pixel representation for each category, even cross-domains.
We proposed a pure pixel-level learning framework, namely PiXL, which consists of a pixel partition module to divide pixels into sub-domains, a prototype generation, a selection module to prepare targets for subsequent alignment, and a pixel alignment module to guarantee pixel feature consistency intra-/inter-images, and inter-domains.
Extensive evaluations of multiple learning paradigms, including unsupervised domain adaptation and semi-/fully-supervised segmentation, show that PiXL outperforms state-of-the-art performances, especially when annotated images are scarce.
Visualization of the embedding space further demonstrates that pixel learning attains a superior representation of pixel features.
The code is available at https://github.com/ChenGan-JS/PiXL.

Published conference paper at ICLR 2024
DIVING SEGMENTATION MODEL INTO PIXELS
Chen Gan Zihao Yin Kelei He 3,, Yang Gao Junfenn Zhang"
1 Medical School of Nanjing University
2 State Key Laboratory for Novel Software Technology, Nanjing University
3 National Institute of Healthcare Data Science, Nanjing University
{ [chengan, zihao. . yin) e smai in ijj.. www.:nww..u....o.ommm edu.cn, {hk1, gaoy, jez edu.
ABSTRACT
More distinguishable and consistent pixel features for each category will ben- efit the semantic segmentation under various settings. Existing efforts mine better pixel-level features attempt to explicitly model the categorical distribution, which fails to achieve optimal due to the significant pixel feature variance. More- over, prior research endeavors have scarcely delved into the thorough analysis and meticulous handling of pixel-level variance, leaving semantic segmentation at a coarse granularity. In this work, we analyze the causes of pixel-level variance and introduce the concept of pixel learning concentrate on the tailored learning process of pixels to handle the pixel-level variance, enhancing the per- pixel recog- nition capability of segmentation models. Under the context of the pixel learning scheme, each image is viewed as distribution of pixels, and pixel learning aims pursue consistent pixel representation inside an image, continuously align pixels from different images (distributions), and eventually achieve consistent pixel rep- resentation for each category, even cross-domains. We proposed pure pixel-level learning framework, namely PiXL, which consists of a pixel partition module to divide pixels into sub-domains, prototype generation, selection module to pre- pare targets for subsequent alignment. and a pixel aligmment module to guarantee pixel feature consistency atra-/inter-images and inter domaiss Extensive evalu- ations of multiple learning paradigms, including unsupervised domain adaptation and moni//fully-supervvi ahe-art performances, especially when annotated images are scarce. Visualization of the embedding space further demonstrates that pixel learning attains superior representation of pixel features. The code available here
segmentation, show that PiXL outperforms state-of-
Level-Leve Variancee
b. Pixel Learning
Insas foo Dortais
Ima mort Uomots
Liw Darsk
diviioation of wel tenumsinonningg
PER Aerheee
0000
Glaani Laad Bace
For Drance
phod buture
Find Pontao Car
trageteee lomingg
Pheilevming
Piul-Lavel Varienco Different Settings
festin
(heesfot
Fully Supervied Seginestaionn
Semi-Sppervted Segeomtaiion
Unagerrieed Dooui Acaypatioo
Figure 1: a. Causes of pixel-level variance. b. Highlights pixel learning, c. Summary of pixel- level variance in different settings.
Corresponding author

---

Published a conference paper at ICLR 2024
1 INTRODUCTION
Semantic segmentation is challenging as requires categorical pixel-level annotations precisely and consistently under different background knowledge contexts, e.g., organs or lesions in medical image analysis (He et al. 2022 Liu et al. 2022), and city scenes in autonomous driving (Tsai et al. 2018) Hoyer et al. (222a). As dense prediction task, the pixel features within the same category across different images are expected to conform to an implicit global distribution. Thus, the pixel features extracted from one several images reveal partial of the global distribution, which can be denoted as local distribution, and exhibits bias from global distribution. Existing rescarch mainly focused on directly modeling or representing the global distribution using prototypes (Zhang et al. 2021 Liu
eet al. 20200 Yang et al., (220)) or speculated prior (e.g., the Gaussian distribution (Xie et al. (023 ). However, the complexity of the global distribution exceeds the representation capacity of prototypes or primary prior distribution. The notable pixel-level variance caused by the factors summarized in Fig. also aggravates the problem. The exquisite design of multiple prototypes or sophisticated prior distribution for specific scenarios also makes them infeasible and biased. We observe that successful segmentation models can construct well global distribution for categorical features during training, but pixel features within local distributions may deviate. Therefore, per-pixel alignment on deviated features helps mitigate pixel-level variance and approach global consistency.
In this work, we raise the concept of pixel learning, wherein each image regarded as a distribu- tion of pixels. Obviously, pixel features are varied in the same distribution (image), among different distributions, and different domains. The unreliable pscudo labels in unsupervised domain adapta- tion or supersupe settings also exacerbate the variance, as summarized in Fig. I Therefore, we propose the pixel learning scheme to address the intra- and inter-distributiio (image) variance and achieve consistent global distribution for each category eventually. Following pixel learning, we introduce a novel segmentation framework PiXL, built on a pure pixel-based learning strategy. Specifically, pixel features from several images form the local distribution at each learning step. For each class, only a subset of pixel features within the local distribution conforms to the global distri- bution, while the others do not, which are denoted as joint pixels and drift pixels, respectively. We first propose the Pixel-Level Sub-Domain Partition (PSP) module to discern those pixels based on their entropy. The multi-resolution prototypes are derived from the joint pixels serve alignment targets for drift pixels. They undergo categorical selection within the Adaptive Prototype Generation (APG) module to ensure their semantic richness. Compared with previous prototype-based methods, prototypes embed rich local distribution information besides global distribution information, de- noted as local prototypes. Hence, the drift pixels in each local distribution undergo asymmetric pushing toward the local prototypes in the Drift Pixels Asymmetric Contrast (DPA) module to har- monize pixels intra- and inter-distribution, ultimately approximating the global distribution.
Extensive experiments of multiple learning settings, including unsupervised domain adaptation (UDA), and semi-/fully-supervvieed segmentation, validate the performance of PiXI.. In the UDA setting, PiXL outperforms the state-of-the-art methods on GTA5 and SYNTHIA + Cityscapes, re- spectively. In the semi-supervised setting on the Cityscapes dataset, PiXL produces consistent and competitive performance, especially on rare labels. When only 3.3% images are labeled, PiXL out- performs the SOTA by large margin. The competitive results in the fully supervised setting on the Cityscapes dataset also show the power of pure pixel learning. The visualization of the pixel feature distribution illustrates that PiXL produces intra-class compact and inter-class separable pixel fcatures for each category.
Our contributions are as follows:
We introduce the pixel learning scheme by treating each image as local distribution of pixels. It then can comprehensively analyze and meticulously handle the pixel-level variance to enhance the per-pixel recognition capability of the segmentation model and integrate various segmentation tasks under unified framework.
We propose the PiXL framework, which segregates pixels within a given local distribution into sub-domains: joint pixels and drift pixels. Then, the PiXL employs an asymmetric alignment ap- proach to align drift pixels with the joint pixels, effectively addressing pixel-level variance in a divide-and-conuuee manner.

---

Published conference paper at ICLR 2024
Extensive experiments confirm PiXL's performance, especially demonstrating promising results label-scarce settings. The t-SNE visualization of the pixel embedding space further illustrates that PiXI. attains enhanced intra-class compactness and inter-class distinctiveness in pixel features.
RELATED WORK
Semantic segmentation using pixel features. Semantic segmentation requires per-pixel classi-
fication, while previous image-level (Wang et al. 2020 [Tsai al. 2018) methods were limited to rough pixel recognition. Recent efforts strive to advance semantic segmentation to achieve a
more refined level of granularity. In unsupervised or self- superviie learning, Xie et al. (2021c) pre pre-training task. In semi-supervised learning. Alonso ct al. (202T) aligned the per-pixel feature to high-quality memory bank pixels. In weakly-supervee
adopted pixel-level contrastive learning
learning. [Ahn & Kwak] (2018): : Du et al. (2022) extracted the pixel-level feature under supervi- sion from CAMs (Zhou al. 2016). Han et al. 2023) enforced pixel- and class-wiss consistency constraints across various perspectives of 3D medical images. In fully supervised learning. Wang ct al. (2021b) explored forming contrastive pairs with the cross-image pixels. In [Ma ct al. (2023).
image was considered disorganized points, and clustering was employed to extract features. enhancing the model's interpretability. conducted contrastive learning on pixels from unified different forms of pixel contrastive learning and added Gaussian distribution prior in SePiCo.
unsupervised domain adaptation, /Vayyat et al. (2022)
images in CLUDA. Xie et al. (2023)
However, few of these prior methods conducted an ex exhaustive analysis of pixel-Ievel variance or de- vised learning strategies at a per-pixel level. Our PiXL, on the other hand, delves into pixel-level variance and tailors pixel-learning scheme address it.
Domain adaptation. Most studies in domain adaptation, including UDA, have predominantly emphasized addressing inter-domain variance (Zhang et al., [202;; Vu et al. 2019). However, the significance of intra-domain variance and even intra-distribution variance is equally noteworthy but has been rarely explored. |Pan et al. (2020) proposed IntraDA explicitly conduct intra-domain adaptation. CCai et al.] (2022) extended IntraDA to an iterative adaptation manner. Yan et al. (202T) proposed PixIntraDA to conduct intra-domain alignment at the pixel level. Whereas the lack of comprehensive investigation of variance the pixel level hampers addressing the issue. To bridge Besides, we approach the pixel- level
the gap, we analyze the causes of pixel-level variance in Fig. [ variance based on the pixel learning scheme in divide-and-come cnd-que manner to align drift pixels in local distribution to joint pixels following global distribution.
Contrastive learning. Contrastive learning is adopted either as a pretext task for pre-training (He) et al.
2020 Wu et al. 2018 Zhha et al. (221]) or a plug-and-play module in a model (Xie et al. Vayyat al. (202)) to refine feature representation. Utilizing the widely employed InfoNCE loss (Oord et al. 2018). features tend to approach positive samples while moving away from negative ones. In early works (Zhang el al. 2022 Chen el al. 2020), contrastive learning was used in image- level learning scenarios. Recently, ly, some works attempted to apply contrastive learning region (Xie) al. (202Ta) or pixel (Xie al. 2023 Wang al. (2021) level. Xie et al. (202 introduced patch- level contrastive learning by adding patch-level InfoNCE loss. [Wang et al. (2022) applied contrastive learning at the pixel level with carefully selected contrastive pairs. Vayyat et al. (2022) introduced an explicit pixel-to-pixe contrastive manner UDA. Nevertheless, these previous works lacked further consideration in contrastive mechanism design and appropriate positive sample selection, which could potentially cause adverse effects and misleading signals. Contrarily, PiXL develops an asymmetric contrast mechanism inspired by Yu et al.) (2022) and collaborates with PSP and APG modules to guarantee the reliability of positive pixels and accurate alignment at the pixel level.
2023
3 METHOD
3.1 PIXEL LEARNING
We first introduce the pixel learning scheme, where each image X is construed as a local distribution of pixels sampled from the global pixel distribution of each category according to equation I
(X, Y) = (13,,))) I,... N: Øj 1,... CC
(1)

---

Published conference paper at ICLR 2024
j represents the pixel index, and signifies an image's total number of pixels. For pixel features of each class k; let Gi denote the global distribution and 9k indicate the local distribution. Conse- quently, feature maps F with their corresponding labels Y are considered as a distribution of labeled pixel features, as depicted in equation 2 where N denotes the number of pixel features in F. While pixel features in 9% are anticipated to follow Gk. this is not always true.
(F,Y)= (ff 1j =1,---.- N; 11 -)= = Ak:
(2)
k=1
where = ((f. 13 k). fy. V, are pixel feature and label.
For the sake of narrative, image also referred to distribution in this work. The pixel learning scheme contributes semantic segmentation in the following two aspects:
Pixel learning elevates the granularity variance analysis to the pixel level, concentrating on intra- and inter-distribution, pixel-level variance, especially emphasizing the intra-image pixel variance.
Pixel learning serves unified scheme in semantic segmentation, entailing continuous intra- and interddistribution pixel-level domain adaptation, i.e., the ongoing alignment of pixel features from two sampled distributions could ultimately address tasks like UDA, semi- and sully-supervised semantic segmentation.
3.2 OVERVIEW OF PIXL FRAMEWORK
Based upon principles of pixel learning, we construct pixel learning framework called PiXL, con- sisting of four modules, i.e., i). Multiple Resolution Feature Extraction, ii). Pixel Level Sub- Domaain Partition (PSP), iii). Adaptive Prototype Generation (APG), iv). Drift Pixels Alignment (DPA).
At each training step PiXI., the learning process proceeds as follows:
Two images are sampled to construct a local pixel feature distribution g by obtaining the features of pixels from the extracted feature map F and the corresponding label Y. We employ multi- resolution input strategy to incorporate more comprehensive semantic information.
The PSP module partitions the pixel features within the local distribution into two categories: f (f..... fN,) } representing joint pixel features assumed to follow the global feature distribution of each class, and F [f,...,,,N, fx,) denoting drift pixel features considered to deviate from those distributions.
The APG module generates multi-resolution prototypes for each category using the joint pixel features and adaptively selects the semantic richness prototype the alignment target for each category.
The drift pixels are aligned to the categorical global pixel feature distribution by pulling them towards prototypes of the same class while simultaneously pushing them away from other prototypes the DPA module.
3.3 MULTIPLE RESOLUTION FEATURE EXTRACTION
We employ a multiple-resolution imput strategy to guarantee the semantic richness in features and the model's robustness varied resolutions of images, inspired by Hoyer al. (2022b). Specifi- cally, PiXI. cuts out a high-resolution part X from image while resizing to the same size of X to form a low-resolution image X', providing abundant details and sufficient context in- formation respectively. The extracted feature maps FL and F# from X and H collaborated
with corresponding labels y and " constitute the local distribution pixel features given by equation 3
g g= u {$/ 1, NN)U((( 1, =1, Nn
3.4 PIXEL LEVEL SUB- DOMAIN PARTITION
PiXL employs entropy as the criteria to segregate the pixel features in into joint pixels and drift pixels. The entropy h of pixel feature is determined by its predicted probability distribution. To

---

Published conference paper at ICLR 2024
a. Multiple Resolution Feature Extraction Module
Class Peak.
Foat.
LR Pioto.
Dhit
Pusk
Joua Roal Aoymmnctia
Coop box
d. Adaptive Prototype Generation Module (APG)
crop
Class
Jein
X
X
X
X
Feat. (LR)
maraingg
Joiet Pinels
pecameless
grreess pesteegee
backbooe
Class & Cass
Class Class
1R Phota.
Num
18 Pixd
Blace Touds
P
pL
Proto
Mam
HBl Pood
Adapeis Phoceegpe Selection
Finds medlan
Piecle voa
Placls na. Plach
Entropy Partnoe
Entopyhhsss! Partatin
VS
Pinvices e s Pieelews Pierk,
Deifi Fixd
Joint Pixel
Drit Finel Ber
Joent Pixel
Fest
e. Drift Pixels Alignment Module (DPA)
Irih Pooe Feat Callettrr
Joint Show Bear Collector
b. Level Domain Partition Module (PSP)
Entropy-based Partition
Povil
Posel Heat..
ang head
waily
Padl
Fanory
Pisel Entrogy
Cuantli
Figure overview of PiXL framework. At each training step, two images are sampled and fed into the subsequent modules. Multiple Resolution Feature Extraction Module extracts features from low- and high-resolution images. Pixel-Level Sub-Domain Partition Module arranges pixels from each image into joint pixels and drift pixels based on entropy. The Adaptive Prototype Generation Module selects prototypes from different resolutions to balance context and details. Drift Pixels Alignment Module aligns drift pixels to local prototypes.
pursue a suitable partition of pixel features, low-resolution and high-resolution pixel features are concatenated and then sorted according their per-pixet entropy. Subsequently, PiXL determines the threshold using the 7) quantile the sorted pixel entropy to conduct partition according to equation 4 It should be noted that pixel partition is performed separately for each distribution to ensure fairness, considering the entropy gap across images.
1= <e), F -5,1% 2 20,
I. N Nu.
(4)
3.5 DRIFT PIXELS ALIGNMENT
Local prototypes. conduct intra- and distribistiibttinn alignment on pixels from two sampled distributions. local prototypes for each category are calculated on joint pixels. Given joint pixels from distribution i and distribution j, local prototype for category & is given by:
(K)
Σ f. where f. f(()) p(M).
p())
A(
(5)

---

Published conference paper at ICLR 2024
The (! and (( refer to feature of joint pixels belongs to category from distribution and distribution j respectively.
Drift pixels asymmetric contrast. Contrastive learning is employed to align pixel features intra- and inter-distribution. We design asymmetric alignment strategy push drift pixels local prototypes p and suppress the misleading signals from drift pixels. Specifically, given a drift pixel fi affiliated category the anchor. The positive sample is the local prototype ((k) while other local prototypes { I, C,j * } are negative samples. Thus, for the single drift pixel fi the asymmetric contrast loss given as follows:
(Fop!
L. LIMLNCE (fr.s,,1)) 100gl E FeB.
log
e(Foptu)/t)
(Feple/r)
E plien
where Sa (p)))),1))))) 11, - 1p ,.. of ki), T temperature parameter.
(6)
PiXL stops the gradient accumulation of prototypes in the DPA module to realize the asymmetric contrast. The asymmetric alignment loss address all drift pixels from two distributions is given by equation?? 7
Lpm Fi Bj)
Σ f.
DAA
PUP
3.6 ADAPTIVE PROTOTYPE GENERATION
Considering the semantic variation in the features extracted from images of different resolutions, the local prototypes calculated from them should share complementary information. For common classes, like sky, road, and building, context information gets compromised in high-resolution fea- tures F", On the contrary, the low-resolution features F Flackk sufficient details. Thus, propose an adaptive local prototype selection module to choose the most meaningful pixels for each cate- gory to form local prototypes. As shown in Fig. 2 the number of pixels emhedded in each prototype is also quantified simultaneously during the generation of multi-resolution prototypes, which is re- (k) ferred as (p((), (() 1, C). We sort these low-resolution prototypes Pi, according their corresponding number pixels (a(0) and compute the median ((()) denoted as m. For
a class k the adaptive selection of the prototype is given by equation g
(x) 2 m, (K)
PL
(k)
< m and N
< (R)
PL
PH otherwise.
3.7 LOSS
addition to Lppa loss, cross-entropy (CE) loss Lin and Thing-Clat ImageNet Feature Distance loss Lin following Hoyer et al. (2022a b): Vayyat et al. (2022) are also employed. The complete loss is summarized in equation where App is the weight of Ltp Please refer to the Appx. A.3 for details of Liz and LiD.
Lpxi. = Lis + AEDdTp - Lppa
(9)
4 EXPERIMENTS
4.1 DATASET
GTA5. A large-scale synthetic dataset contains 24,966 annotated images. SYNTHIA. A collection of generated urban images including 9,400 images. Cityscapes. A real-world street scene dataset contains 2,975 training images and 500 validation images.

---

Published conference paper at ICLR 2024
TASK SETTING
Fully supervised learning. use the training images from Cityscapes train our PiXT, and evaluate that on the corresponding validation part. Semi-supervised learning. We randomly select 1/4, 1/8, and 1/30 labeled images from the Cityscapes training set while utilizing the left images as unlabeled images to train PiXI.. Performance reported on validation part. Unsupervised domain adaptation. We evaluate our PiXI on GTAS-<Ciyscapes and SYNTHIA -Cityscapes. mloU is utilized to evaluate the model's performance in these tasks, which is calculated by the average of intersection over the union each category.
4.3 IMPLEMENTATION DETAILS
Network architecture. The HRDA model (Hoyer al. 2022b) is adopted the baseline, which consists of a MřT-B5 encoder (Xie et al.) (0221) and feature fusion decoder from DAFormer (Hoyer et al. 2022a). We further validate the effectiveness and plug-and-play property of PiXL on DeepLabV3+ (Chen et al. 2018) backbone, referring Appx. A.2 for more details.
Training. We follow the Vayyat et al. (022) training strategies and parameters, i.e. a batch size of 2, the optimizer is A with learning rate of 6 10 for the encoder and 6 x 10 for the decoder with linear warmup policy, DACS data augmentation, rare class sampling strategy on labeled training data, the self-training method for unlabeled training data. In UDA and SSL settings, employ the Exponential Moving Average (EMA) updated teacher model to generate the pscudo labels for unlabeled images, as shown Fig. Moreover, we set the training epoch to 60000, set the to 0.2 initially, and decrease linearly until 0.001 during training. The model is trained single Tesla V100 with 32 GB memory. More details are provided in Appx. $ A.4
Performance varies with labeled image proportions.
60
Table Comparison with previous methods in the remi-supervised setting on Cityscapes.
5000
0.122%
miuu
0.00%%
Mithoch
Irege Siee
1/06 (64)
1/6132)
(/4/44)
PRL PIBN Degpatio BeCe Buchaaaata
Class Mil (Truuur Syal
<13 1001
$1.00
61:25
(1.65
60
112-1104
790--191
11.00
15311
0.500
0.95
9,48
g175
250
930-231
Proportion of labeld Images
312-312
PALL (HBBAA)
312-312
1132
18.91
Figure 3: Performance on different proportions of labeled images.
Caiblii
Labeled Image
Libbbeked Imagee
POST Saadem
EMA
CH: 1938 22 1,87
CH 1722.25
D6 2/42
POOL
CLUDA
Figure 5: Pseudo labels generation in PiXT.
Figure 4: t-SNE of pixel features CLUDA and PiXI..

---

Published conference paper at ICLR 2024
Table 2: Comparison with previous methods in UDA setting on GTAS, SYNTHIA + Cityscapes.
Methods
Ruad S. Walk Build. Wall Fance Polu Tel Tilhight Sign Viget. Turrain Sky Person Rider Car Track Bux Train M.Bika Bike mled
GTAS
Cityscaps
AdaptSee at
2014
86.5
25.9
7938
22.1
200
23.6
35.1
21.5
51.5
25.9
75.9
57.5
26,2
76,5
29,8
32.1
7.2
29.5
31.5
41.4
CRST 2on
201
91.8
53.5
80.5
32.7
21.0
34.0
26.9
30.4
53.9
34.2
80.
53.1
34.0
82.7
30.
15.9
16.0
25.9
42.8
45.
DACS Teamenaen
2021
89.9
39.7
87.
30.7
39.5
38.5
46.4
52.8
88.0
44.0
888
67.2
35.8
54.5
45.7
50.2
0.0
27.3
34.0
52.1
CarlA Wann
94.7
63.1
876
30.7
406
40.
473
51.5
82.6
47.0
89.
66.7
35.9
96.2
48.9
57.5
39.
56.0
566
RAPATLI
94.4
61.0
88.0
36.5
199
1g.
46.1
55.3
82.8
46
894
58.8
40.0
96.2
60.
59.0
45.1
542
57.4
ProDA
87.8
56.0
79.7
46.3
448
456
53.5
53.5
58.6
45.2
82.1
70.7
39.2
58.8
45.5
99.4
10
48.9
564
57.5
DAFormeri
95.7
70.2
894
53.5
48.
896
553
59.4
58.9
47.9
92.5
72.2
44.7
94.5
65.1
55.9
61.
68.
SePiC
96.9
76.7
89.7
55.5
49.5
53.
600
64.5
90.3
50.3
908
74.5
44.
91.3
710
79.5
63.6
61.0
65.
703
HRDA (oweraa
14.4
63.9
09.3
19.0
841
CLUDA
78.0
910
60.3
85.3
36.3
68.3
71.3
91.3
51
78,4
37.9
36.5
7500
69.7
24.1
PXX..
97.0
77.6
91.1
59.9
$6.1
57.3
64.5
68.1
91.5
51.8
94.8
M0.5
57.3
94.6
83.8
SKL7
78.0
65.6
67.8
75.0
SYNTHIA
Cliyscapos
AdaptScd Tvi ct
A018
79.2
57.2
78.8
0.9
10.5
7X.
80.5
53.5
19.6
67.0
29.5
21.5
31.3
57.2
CBST 2og
2013
68.0
29.9
76.3
10.5
1.4
35.9
22.8
29.5
77.5
78.3
04.5
25.3
$1.6
23.5
18.5
29.8
42.6
DACS Tepinaac
2021
80.6
25.1
81.9
21.5
2,9
37.2
22.7
24.3
53.7
90.
67.5
38.3
$2,9
38.9
28.5
47.5
483
CorlA Wrin e
2020
93.3
61.6
85.3
19.5
5.1
37.8
36.6
42.8
$4.9
93.4
00.7
41.8
85.6
38.4
32.6
53.9
55.0
BAPAILU La
20::0
91.7
53.8
83.9
22.4
0.5
34.9
30.5
42.8
56.5
88.2
66.0
34.1
56.6
51.3
29.4
50.5
53.3
ProDA Zhanr ni
D121
87.8
45.7
84-6
37.1
06
44.0
54.6
17.0
58.1
84.
74.2
24.3
88.2
51.1
40.5
45.
33.
DAFormer Tiins
20024
84.0
40.7
8844
41.5
6.5
50.0
55.0
54.5
$6.0
898
73.2
48.2
$7.2
53.1
53.9
61.7
AM
SePICaAA
20:3
87.0
52.6
88.5
40.5
10.6
49.8
57.0
55.4
56.5
86.2
75.4
52.7
92,4
78.9
53.0
62.6
64.3
HRDA Aiverra Miaah
85.2
47.7
88.8
49.5
4.8
57.3
65.7
60.9
85.3
92.9
70.4
62.8
86.0
64.7
61.9
649
658
CLUID
87.7
46.9
90.2
49.0
79
99.5
66.
58.5
58.3
94.6
90.1
57.1
90.5
68.1
65.5
65.8
668
PIXL
29.9
5388
00.1
528
55.
40.9
63.5
S8.4
94.6
50.5
55.9 90.8
68.1
67.0
62.9
67.1
Table 3: Comparison with previous methods in the fully supervised setting on Cityscapes.
Table 4: Component ablation of PiXL.
Methods
Image Size
Cityscapes
baseline PIXL PIXI
74.7
+0.9
HKNetV2 Wang all
2020
512x 1024
81.10
75.0
+1.2
HRVIT-bI
ContrastiveSeg
Wane
ar 2021b
512x 1024
82.40
HRVIT-b2
1024x 1024
Table 5: Quantile ablation of PiXI..
SegFormer
Xue al
20215
1024xx 1024
84.00
Mask2Former
512x 1024
84,50
0.4
ScMask
mloU
74,8
75.0
75.0
74.9
74.1
512x552
Image
HRDA
CLUDA
PiXL (Ours)
Ground Truth
Figure 6: Qualitative comparison with bascline and SOTA methods in UDA setting on GTA5 +
Cityscapes. PiXL performs better boundary pixels and internal pixels.
4.4 RESULTS
Unsupervised domain adaptation. In the UDA setting, our PiXL outperforms the baseline model HRDA and achieves commensurate performance with the SOTA. In GTAS - Cityscapes and SYN- THIA - Cityscapes, our model PiXL surpasses the baseline by margin of 1.2% and 1.3% respec- tively in Tab. 2 Compared with the SOTA model CLUDD,, method outperforms by margin

---

Published a conference paper at ICLR 2024
of 0.6% and 0.3%. Specifically, in GTA5 -> Cityscapes, PiXL obviously improves the performance on rare classes, like pole, motorbike, elc. Moreover, the performance improvement is also consistent in SYNTHIA - Cityscapes. In the common class pixels. PiXL also achieves comparable results or surpasses the previous works, like vegetation, road, sidewalk, etc. Fig. C provides more details, depicting that PiXL performs better in internal and boundary pixels. Experiments conducted in the UDA setting confirm the effectiveness of PiXL, implying that the pixel learning scheme produces more consistent pixel features and mitigates the domain gap at the pixel level.
Semi-supervised semantic segmentation. We evaluate PiXL on DeepLabV3+ (Chen et al. 2020) and HRDA (Hoyer et al. 20226) to verity its plug-and-play property. PiXL performs compettively with the SOTA models summarized in Tab. especially in 3.3% images labeled setting, where PiXL (DeepLabV++), PiXL (HRDA) surpass SOTA by 5.72% and 6.83% respectively. Fig. B further confirms PiXL maintains its excellent and robust performance as the labeled image ratio varies. The qualitative analyses are available in Appx. $ Experiments on semi-supervised seg- mentation demonstrate PiXL's ability to thrive in label-efficient scenarios, which further highlights the strengths of pixel learning.
Fully supervised semantic segmentation. As depicted in Tab. 3 PiXL surpasses HRNetV2 and HRViT-bl. which concentrate multi-resolution feature mining. PiXL exhibits competitive perfor- mance compared with the SOTA methods trained on higher resolution images, such as 768 768, 512 x 1024, and even 1024 x 1024. Notably, PiXL achieves this level of performance while being trained on 512 x 512 images, Although images of lower resolution and smaller fields of view may potentially deteriorate pixel feature representation, which exacerbates the bias of the local distribu- tion, the pixel learning scheme effectively mitigates these influences to some extent and continues deliver competitive performance. The qualitative analyses are provided in Appx. A.5
Ablation study. We conduct abundant ablation of PiXL in GTAS - Cityscapes task. The ablation on components, including the DPA and APG modules, are summarized in Tab. 4 Compared with the baseline model HRDA, our drift pixels alignment module improves the performance by 0.9%, which verifies that the asymmetric contrast mechanism effectively addresses pixel feature variance and ultimately improves the model's per-pixel recognition capability. Adopting the adaptive selection in the APG module further improves the performance of PiXL to 75.0%, which outperforms the baseline by margin of 1.2%. This underscores that the adaptive selection contributes to achieving a balance between details and contextual information encapsulated in the local prototypes. making them better alignment targets for drift pixels. Despite the ablation study of presented in Tab. 3 demonstrating robust improvement on the baseline performance 73.8%. still affects the model sightly. Considering the partition threshold determined by -1 quantile, a larger 1) means more pixels are drift pixels. The improvement is marginal when is 0.8, confirming that the drift pixels are indeed small portion and tailored adjustment in PiXL refines each category's 's pixel features.
Visualization. We employ t-SNE visualize the distribution of pixel features in the embedding space and compare the pixel features of PiXL and the SOTA method CLUDA. Fig. iillusraats that PiXL refines pixel feature representation with better inter-class separability and intra-class compact- ness. PiXI's performance quantitative metrics Calinski-Harabasz Index (CH) Index (DB), i.e. 1936.22 and 1.87 while that CLUDA is 1722.23 and 2.42, further confirming the above conclusion, as larger CH and a smaller DB indicate better inter-class separability and intra-class compactness. Both visualization and quantitative analysis substantiate the improvement pixel features achieved by PiXI..
5 CONCLUSTON
In this work, we propose novel pixel learning scheme to dive semantic segmentation models into the pixel level by treating an image as distribution of pixels. This advocates addressing pixel-level variance to enhance the segmentation model's per-pixel recognition capability. We proposed PiXL, a pure pixel-level learning framework that executes pixel-level intra- and inter-distribution (image) alignment within the context of pixel learning. Extensive quantitative and qualitative experiments in various settings of semantic segmentation confirm the effectiveness of PiXL, demonstrating the feasibility and superiority of the pixel learning scheme, which deserves further exploration.

---

Published conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
To guarantee the reproducibility and completeness of this paper, devote the following efforts:
We illustrate the model pipeline in Fig. 2 and Fig. with detailed explanations provided in section
3.2 to facilitate understanding.
We provide abundant details regarding the model implementation and hyper-parameters choice in
section 4.3
Appx.
A.2 and Appx. |A.4
We comprchensively explain the loss functions in our work in Appx. A.3
The code will available upon acceptance.
ACKNOWLEDGMENTS
This work is partially supported by the National Natural Science Foundation of China (grant no. 62106101), and the Natural Science Foundation of Jiangsu Province (grant no. BK20210180). This work is also partially supported by the Al & Al for Science Project of Nanjing University.
REFERENCES
Jiwoon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image- level supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4981 4981-4990, 2018.
Inigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano, and Ana C Murillo. semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 8219-8228,
2021.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. Proceedings of the IEÉE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Yuxiang Cai, Yingchun Yang, Yongheng Shang, Zhenqian Chen, Zhengwei Shen, and Jianwei Yin. Iterdanet: Iterative intra-domain adaptation for semantic segmentation of remote sensing images. IEEE Transactions Geoscience and Remote Sensing, 60: 2022.
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplah: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE transactions pattern analysis and machine intelligence, 40(4): 934-848, 2017.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmen- tation with cross pseudo supervision. In Proceedings of the IEEF/CVV Conference Computer Vision and Pattern Recognition, pp. 2613-26222 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1290-1299, 2022.
Ye Du, Zehua Fu, Qingjic Liu, and Yunhong Wang. Weakly supervised semantic segmentation by pixel-to- prototype pixel-to-prototype contrast. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4320-4329, 2022.
10

---

Published conference paper at ICLR 2024
Jiaqi Gu, Hyoukjun Kwon, Dilin Wang. Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai. Vikas Chandra, and David Pan. Multi-scale high-resolution vision transformer for semantic segmen- tation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12193-12103, 2022.
Meng Han, Xiangde Luo, Wenjun Liao, Shichuan Zhang, Shaoting Zhang, and Guotai Wang. Scribble- 3d multiple abdominal organ segmentation via triple-branch multi-dilated network with pixel- and class-wist wise consistency, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Kelei He, Chen Gan, Zhuoyuan Li, Islem Rekik, Zihao Yin, Wen Ji, Yang Gao, Qian Wang, Junfeng Zhang. and Dinggang Shen. Transformers in medical image analysis: A review. Intelligent Medicine, 2022.
Lukas Hoyer, Dengxin Dai, Yuhua Chen, Adrian Koring, Suman Saha, and Luc Van Gool. Three ways improve semantic segmentation with self-supervised depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11130-11 11140, 2021.
Lukas Hoyer, Dengxin Dai, and Luc Van Gool. Daformer: Improving network architectures and training strategies for adaaiiee semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9924-9935, 2022a.
Lukas Hoyer, Dengxin Dai, and Luc Van Gool. Hrda: Context-aware righ-resolution domain- adaptive semantic segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX, pp. 372-39 -92. Springer. 2022b.
Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised seman- tic segmentation via adaptive equalization learning. Advances in Neural Information Processing Systems, 34:22106 22118. 2021.
Jitesh Jain, Anukriti Singh. Nikita Orlov, Zilong Huang, Jiachen Li, Steven Walton, and Humphrey Shi. Semask: Semantically masked transformers for semantic segmentation. arXiv preprint 12.11:2112.12782. 2021.
Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson WH Lau. Guided collaborative training for pixel-wise supervied learning. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pp- 429-445. Springer, 2020.
Shikun Liu, Shuaifeng Zhi, Edward Johns, and Andrew J Davison. Bootstrapping semantic segmen- tation with regional contrast. arXiv preprint arXiv:2 04465, 202
Xiaoming Liu, Quan Yuan, Yaozong Gao, Kelei Hc, Shuo Wang, Xiao Tang. Jinshan Tang, and Dinggang Shen. Weakly supervised segmentation of covid19 infection with scribble annotation on et images. Pattern recognition, 122: 108341, 2022.
Yahao Liu, Jinhong Deng, Xinchen Gao, Wen Li, and Lixin Duan. Bapa-net: Boundary adapta- tion and prototype alignment for cross-domain semantic segmentation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 9801-8811. 202
Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He. Part-aware prototype network for few-shot semantic segmentation. In Computer Vision-Ei ECCV 2020: 16th European Conference, Glasgow, UK, Augusi 23-28, 2020, Proceedings, Part IX 16. pp. 142-158. Springer, 2020.
Xu Ma. Yuqian Zhou, Huan Wang, Can Qin, Bin Sun, Chang Liu, and Yun Fu. Image as set points. In The Eleventh International Conference on Learning Representations, 2023. URL https:/ https::/ppeeeevvie net. forum? d=awnoo2j66

---

Published conference paper at ICLR 2024
Viktor Olsson, Wilhelm Tranheden, Juliano Pinto, and Lennart Svensson. Classmix: Segmentation-
based data augmentation for semi-supervised learning. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 1369-1378, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprini 1rXiv:1807.03748, 2018.
Yassine Quali, Céline Hudelot, and Myriam Tami, Semi-supervised semantic segmentation with cross-consistency training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12674- 12684, 2020.
Fei Pan, Inkyu Shin, Francois Rameau, Seakju Lee, and In So Kweon. Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3764-3773, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances neural information processing systems, 30, 2017.
Wilhelm Tranheden, Viktor Olsson, Juliano Pinto, and Lennart Svensson. Dacs: Domain adapta- tion via cross-domain mixed sampling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1379-1389, 2021.
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Hsuan Yang. and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7472-7481, 2018.
Midhun Vayyat, Jaswin Kasi, Amuraag Bhattacharya, Shuaib Ahmed, and Rahul Tallamraju. Cluda: Contrastive learning in unsupervised domain adaptation for semantic segmentation. arXiv preprint arXiv:2208. 14227, 2022.
Tuan- Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Perez. Advent: Adver- sarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang. Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learn- ing for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 43 (10):3349-336, 2020.
Qin Wang, Dengxin Dai, Lukas Hoyer, Luc Van Gool, and Olga Fink. Domain adaptive semantic segmentation with self-supervised depth estimation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 8515-8525, 2021a.
Wenguan Wang. Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Ex- ploring imase pixel contrast for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7303-7313,200 2021b.
Yuchao Wang. Haochen Wang. Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pscudo-labele In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4248 4448,4257,25 2022.
Zhirong Wu, Yuanjun Xiong. Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 3733-3742, 2018.
Binhui Xie, Shuang Li, Mingjia Li, Chi Harold Liu, Gao Huang, and Guoren Wang. Sepico: Semantic- guided pixel contrast for domain adaptive semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Enze Xic, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8392-8401, 202
12

---

Published conference paper at ICLR 2024
Enze Xie. Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:12077-12090 202
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16684-16693, 2021c.
Zizheng Yan, Xianggang Yu, Yipeng Qin, Yushuang Wu, Xiaoguang Han, and Shuguang Cui. Pixel- level intra-domain adaptation for semantic segmentation. In Proceedings of the 29th ACM Inter- national Conference Multimedia, pp. 404-413, 2021.
2021.
Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, Augusi 23-28, 2020, Proceedings, Part VIII 16, pp. 763-778. Springer, 2020.
Qiying Yu, Jieming Lou, Xianyuan Zhan, Qizhang Li. Wangmeng Zuo, Yang Liu, and Jingjing Liu. Adversarial contrastive learning via asymmetric infonce. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V, pp- 53-69. Springer, 2022.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Harry Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In International Conference Learning Representutions, 2022.
Pan Zhang. Bo Zhang, Ting Zhang, Dong Chen, Yong Wang, and Fang Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12414- 12424. 2021.
Xiangyun Zhao, Raviteja Vemulapalli, Philip Andrew Mansfield, Boqing Gong, Bradley Green, Lior Shapira, and Ying Wu. Contrastive learning for label efficient semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10623-10633, 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza. Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation se- mantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pp. 289-305, 2018.
A APPENDIX
A.1 OVERVIEW
In this appendix, we provide further insights into the PiXL framework, including implementation training and inference procedures $ A.4. and supplementary
details $ A.2 loss functions $ A.3
qualitative analysis A.5
A.2 IMPLEMENTATION DETAILS
We have implemented two variants of the PiXL framework utilizing different backbones and de- coders, as depicted in Tab All the backbones have been pre- trained ImageNet-1K.
Table 6: Variants of PiXI. indicates the inclusion of modifications.
Framework
Backbone
Decoder
PIXI (Deepl.ah334)
ResNer101
DeepLahhV3 Decnder!
PIXI. (HRDA)
DAFommer Dacoder

---

Published conference paper at ICLR 2024
PiXL (HRDA) We adopt the MiT-B5 as the backbone and employ a context- feature fusion decoder proposed by Hoyer et al. (2022a). The MiT-B5 extracts multi-scale features from input images, which are passed through an Atrous Spatial Pyramid Pooling (ASPP) (Chen et al. 2017). This module generates the final feature representation for per-pixel prediction and constitutes the pixel features used for pixel learning. The feature dimension C' of each pixel is 256. The attention- based multi-scale prediction fusion mechanism is also applied in our PiXL following Hoyer et al. (2026))
PiXL (DeepLabV3+)) To validate the aad---ddd-pyy property of PiXL, we have also implemented a CCNN-based variant, which comprises ResNet101 (He ct al., 2016) encoder and DeepLabV3+ head (Chen et (018). In addition to enriching pixel features by combining the low-level features with the fused features, we modify the input of the ASPP head by aligning and stacking features from all four encoder blocks, similar to the approach employed in the DAFormer head.
A.3 LOss FUNCTIONS
According to 3.7 the loss function in PiXL encompasses cross-entropy loss LeE, Thing-Clit Class ImageNet Feature Distance loss L'D and our Drift Pixel Alignment loss D detailed in 3.5
Cross-entropy loss Given feature map F, its cross-entropy loss is given following equation 10 where Pt, denotes the probability pixel feature f. belongs class Y, signifies the label asso- ciated with feature ft-
LCE(F) =
We
Σ K=I,-.,
II.. log Pte
(10)
wy indicates the per-pi per-pixel cross-entropy loss weight, set to 1.0 when the label is available. Otherwise, Wy is determined by the confidence of pseudo labels to relieve noise from them. In the context where F, E RAxwxC is obtained from an unlabeled image X,, we use Y, Bix to denote the pseudo label predicted by the EMA teacher. Additionally, M, E Rax is employed to represent the confidence of each pixel. with each value corresponding to the maximum value within each pixel's probability distribution. For calculating wy for each pixel in Fj, we utilize the formula provided in equation 11 with 5 set to 0.968.
h
(M) >
W
1=
(11)
h X
Since we sample two distributions each training step and employ multi-resolutionn input strategy, the comprehensive cross-entropy loss defined as follows:
L'ce L. + Lee( ( ) AudCa((F" AH.CCE(
(12)
The value dω is set 0.1 in equation 12
Thing-Clase ImageNet Feature Distance loss The Thing-Class ImageNet Feature Distance loss,
referred to as LTD. is employed to leverage the recognition capability of an ImageNet pre-trained backbone for training regularization purposes. following Hoyer et al. (2022a b): Vayyat al. (2022). Let S' represent the ImageNet pre- pre-trained backbone and F" denote the last layer features extracted from F Similarly, I and represent the backbone and its corresponding last layer features from PiXL. The distance between each pixel feature f, f and its corresponding feature f in f calculated as follows:
dy
f
(13)
As illustrated in Hoyer et al. (2022a b): Vayyat al. (202)). the LED is applied to thing classes (Caesar al. 2018). denoted as Crhings A mask generated to distinguish between things stulf, where each element I, indicates whether feature f, E Crhings- Consequently, the Lyn on features from pretrieedde backbone Ff and PiXL backbone F is given follows:
Ex= dg
LiD LFo(F.FF
(14)
14

---

Published conference paper at ICLR 2024
We exclusively apply LDD to labeled images, with a weight of AFD set to 0.005, in order to pre- vent overfitting in the contexts of UDA and semi-supervised segmentation. However, we observe a performance drop in fully supervised segmentation when Lpp is applied. This suggests that the regularization might impede the exploitation of abundant supervision. Consequently, we A 0 in the fully supervised setting.
A.4
TRAINING AND INFERENCE DETAILS
A.4.1 TRAINING
Augmentation: PiXL employs several image augmentation techniques, including random crop- ping and random flipping. Initially, images from GTAS, SYNTHIA. and Cityscapes datasets re- sized 2560 1440, 2560 1520, and 2048 1024, respectively. Subsequently, 1024x 1024 patches are randomly cropped from each image, with random flips applied as well. Unlike previous meth- ods that downsampled the original images, this augmentation strategy preserves more image details cost of contextual information, intensifying the bias of local distribution g. However, PiXI. excels in handling those issues.
Crop in Multi-Resolutie Feature Extraction: In the ule, an augmented 1024 1024 image is simultaneously cropped into a 512x512 high-resolution patch X H and resized into a 512x512 low-resolution patch X During the initial 12000 iterations, random cropping is employed. Afterward, regions with high entropy are selectively cropped, aiding PiXL in focusing the drift pixels.
Feature Extraction mod-
have integrated ClassMix (Olsson et al. 2021 into PiXL to enhance data augmen- tation. Specifically, for two images X, and Xj, we randomly select half of the classes from X,, mix them with Xy. and produce an augmented image X,. Following Tranheden et al. (2021), apply color jitter and Gaussian blur to Xz. Subsequently, both X, and the augmented image X', are input into PiXI., as depicted in Fig. 5
ClassMix:
Pseudo label generation: An EMA-updated version of PiXL employed as the teacher model generate pseudo labels for unlabeled images in UDA and semi-supervised segmentation. We denote the parameters in the teacher model and the back-propagated PiXL at training step as ((() EMA and
( PiX y. both initialized be the same. The EMA update process is executed in accordance with the formula presented equation 16
W P 0 (0)
( EMA
(15)
W
+ (1 \EMA) w t
ABMA
The A step t is illustrated as follows:
A
min(1
8 0.999)
(16)
EMA
Entropy PiXL, we employ Information Entropy (entropy) as the measurement criteria split the pixels into joint and drift pixels. For a pixel feature representation with its predicted probability distribution, its entropy A is given by:
h h=
Σ
p& log
(17)
where C' is the number of classes and pt: is the probability of the pixel belonging to class k.
A.4.2 INFERENCE
We employ sliding window strategy during the inference stage make predictions, similar the baseline HRDA (Hoyer et al., 20226). The sliding window has a size of 1024x 1024, with an overlap of 512 x 512. Each patch within the sliding window is predicted by combining results from the 512 x 512 high-resolution and fow-resolution inputs.
15

---

Published conference paper at ICLR 2024
A.5 QUALITATIVE ANALYSIS
This section qualitatively analyzes segmentation results semi-supervised and supev-se sce- narios to highlight PiXL's effectiveness and robustness. As shown in Fig. 7 our results maintain precision and consistency across major classes like road, sidewalk, and car, well minor classes such as pole, bicycle, and traffic sign, even when the number of labeled images decreases signifi- cantly. This further validates the efficacy of pixel learning, particularly in label-scarce scenarios.
We also examine several failure cases of our PiXI. in Fig. Pixels located boundaries pose chal- lenges for accurate recognition. like fence and traffic sign, or terrain and road, can be error-prone. Additionally, distinguishing pix- els between categories with similar semantic information is also challenging, especially sidewalk and road, as depicted in Fig. Finally, some minor categories with very few pixels in region can be overlooked, such as pole.
instance, distinguishing pixels at the boundaries categories
A.6 FUTURE WORK
Through extensive experimentation, we have substantiated the efficacy of the pixel learning scheme. This approach delves the segmentation model into the pixel level, significantly emphasizing the tai- lored learning process for each pixel. The proposed pixel learning scheme inherently aligns with the per-pixel recognition attributes integral to semantic segmentation. also offers a unified frame- work for diverse semantic segmentation scenarios by accounting for pixel-level variances. To further harness the advantages of pixel learning, the following considerations merit attention:
Pixel learning serves as a plug-and-play framework that seamlessly integrates with diverse back- bones and decoders, enhancing the performance of existing models.
While multi multi-resolution strategies are common in semantic segmentation, they are optional in our scheme. In fully supervised segmentation tasks, approach, which incorporates mulli-resolution strategy, conserves space compared with existing models. However, further optimization opportuni- ties exist.
While the simple choice of demonstrates the model's robustness compared with the baseline. it's important consider varying pixel-level variances across tasks and differences in bias between local and global distributions. Further research on the selection strategy for 1) or alternative pixel feature partitioning methods is valuable.
Image
3.3% Laheled
12.5% Labeled
25.0% Labeled
100.0% Labeled
Ground Truth
Figure More qualitative examples on PIXL (HRDA) on different ratios labeled images. The segmentation results are precise and consistent in both major categories and minor categories.
16

---

Published conference paper at ICLR 2024
Image
3.3% Labeled
12.5% Labeled
25.0% Labeled
100.0% Labeled
Ground Truth
Figure 8: Failure cases of PiXL (HRDA) in the semi-supervised and super-sup setting. The ability to recognize similar pixels lying in boundaries requires further improvement in categories like sidewalk, road, terrain, and fence.
17


Reviews:
[{"review_id": "mLRA7cvZer", "reviewer": "Reviewer_L3L6", "summary": "The paper proposes  a pixel-level learning framework, PiXL, for semantic segmentation. The framework consists of a pixel partition module,  a prototype generation and selection module, and a pixel alignment module. The pixel partition module separate pixels features into joint and   drift pixels based on their entropy. The prototype generation module is to select the most meaningful pixels. The pixel aligment module adopts contrastive learning to align pixel features intra and inter-distribution. The effectiveness of proposed framework and components are experimentally validated on three public datasets: GTA5, SYNTHIA, and Cityscapes.", "questions": "no", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of investigating semantic segmentation from pixel feature distribution perspective is novel.\n2. The proposed pixel learning framework and its components are technically solid and innovative.\n3. The motivation and the underlying principles of designing the framework and each components are clearly presented and explained, so that it is easy to follow the work.\n4. The experiments are extensive and solid.", "weaknesses": "The experiment results of the proposed results are not much better than previous works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7R7gahWBMy", "reviewer": "Reviewer_nhEN", "summary": "This paper takes pixel-level distribution (local distribution) into consideration, and proposed Pixel Level Sub-Domain Partition Module (PSP), Adaptive Prototype Generation Module (APG); Drift Pixels Alignment Module(DPA) modules, the effectiveness of which is proved in ablation study.\nFirst, the PSP module divides all features (with multiscale feature extraction) into Joint pixel features and Drift Pixel features based on the entropy of each segmentation pixels corresponding to each pixel feature. For Joint ones, they use APG to generate local feature prototypes based on their semantic classes, while the Drift ones would be pulled by the prototypes extracted from APG using info NCE loss. \nThe prototypes from APG is the mean value of pixel features which belongs to Joint pixel features in two samples. From which, the paper argues that can get intra-image and inner-image information.\nFinally, the effectiveness of these module is proved in unsupervised domain adaptation, semi-supervised semantic segmentation together with fully-supervised semantic segmentation.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and nicely organized.\n2. Extensive experiments have been conducted and a number of quantitative and qualitative results are shown, demonstrating the effectiveness of the proposed method.", "weaknesses": "1. The method seems incremental. The paper generates class prototype from two image features and push or pull image features based on these prototypes. Their novelty lies in the partition of joint features and drifted features, the selection of high resolution feature prototypes and low resolution prototypes, which seems to be triky.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jzuGWjSZV9", "reviewer": "Reviewer_YSzS", "summary": "In this paper, the authors introduce the pixel learning scheme by treating each image as a local distribution of pixels. The PiXL framework, which segregates pixels within a given local distribution into sub-domains: joint pixels and drift pixels, is proposed. Then, the PiXL employs an asymmetric alignment approach to align drift pixels with the joint pixels, effectively addressing pixel-level variance in a\ndivide-and-conquer manner. Extensive experiments confirm PiXL’s performance, especially demonstrating promising results in\nlabel-scarce settings.", "questions": "The questions are summarised with weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper  proposes a novel pixel learning scheme to dive semantic segmentation models into the pixel level by treating an image as a distribution of pixels. This advocates addressing pixel-level variance to enhance the segmentation model’s per-pixel recognition capability.\nThe strengths are as follows:\n1. This paper proposed PiXL, a pure pixel-level learning framework that executes pixel-level intra- and inter-distribution (image) alignment within the context of pixel learning. \n2. Extensive quantitative and qualitative experiments in various settings of semantic segmentation confirm the effectiveness of PiXL, demonstrating the feasibility and superiority of the pixel learning scheme, which deserves further exploration.\n3. The writing is clear and well-reading.", "weaknesses": "The weakness are as follows:\n1. Some description is not very clear. For example, in equation 4, \"PiXL determines the threshold ...\", how to determine the threshold is not presented. why \" that pixel partitioning is performed separately...considering the entropy gap across images.\"?\n2. \"PiXL employs entropy as the criteria to segregate the pixel features in g into joint pixels and drift\npixels.\" how to compute the entropy?\n3. the paper validate the effectiveness on HRDA model, but if the proposed methods can be applied to general semantic segmentation methods is not verified.\n4. In table 3, the proposed method cannot show state-of-the-art performance compared with other methods. The authors should prove its effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W7zQ69EcJh", "reviewer": "Reviewer_GSD6", "summary": "This paper proposed a  pixel learning framework for semantic segmentation. Intra-image, inter-image, inter-domain pixels variances are considered in this framework. The framework is elaborate, which consists of four components, i.e., Multiple Resolution Feature Extraction, Pixel Level Sub-Domain Partition, Adaptive Prototype Generation, and Drift Pixels Alignment. The motivation of this paper is interesting. The experimental results demonstrate the effectiveness of the proposed method.", "questions": "see the weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Pixel variance is important in semantic segmentation. This paper proposed a new solution. \n\n(2) This framework is flexible, it is quite easily to perform different semantic segmentation tasks.\n\n(3) The performance is good.", "weaknesses": "(1) The authors did not report the results on higher resolution images, is that because too many pixels should be considered?\n\n(2) In semantic segmentation, contextual information is quite important to assign a class label to a pixel.  But this paper discards the context in some extent. Is this reasonable?\n\n(3) Pixel-level contrastive learning is widely used in unsupervised semantic segmentation, both local and global relations are considered. In these methods, global pixel features are usually store in a memory bank.  The differences with these method should be given in detail.\n\n(4) In Table 1, the proposed method performs worse with 1/8 and 1/4 than 1/30. The authors should explain this.\n\n(5) In addition of intra-image, inter-image pixel relations, this method also considers the inter-domain one, but this is not presented in abstract.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)